PostHog Engineer Dylan Martin Details Move from Feature Flags to AI Product Engineering

PostHog engineer Dylan Martin described his transition from leading the feature flags team to AI product engineering, stating the shift is smaller than job descriptions suggest. He said AI product engineering requires writing evals for nondeterministic LLM behavior and balancing human interfaces against handing capabilities to agents via MCP and Skills.
Event version 1.
Key insights
Transition smaller than expected
Martin stated the move from leading the feature flags team to AI product engineering has not felt as big as job descriptions suggest. He said any engineering role still requires understanding systems, failure modes, and how to prove correctness.
Evals are a core new skill
Martin said writing evals is a major skill in AI product engineering that rarely comes up in traditional product engineering. Evals must be more flexible than traditional tests because the same request can have several good answers and LLM behavior is nondeterministic.
Regression example in analytics agent
Martin described testing a query tool for an analytics agent that answered how many users signed up yesterday but counted signup events instead of unique users when asked the same number by country. He said evals should catch this kind of regression before launch.
Balancing interfaces and agent tools
Martin said the harder problem is balancing human-facing interfaces against handing the same capabilities to agents through MCP and Skills. He prefers interfaces when a person must decide which workflow an agent should run, such as visualizing a feature flag's impact or which users an error affects.