LangChain Details Three-Step Eval-Driven Development Process for AI Agents

LangChain outlined a three-step eval-driven development process for building AI agents: define what good looks like, write the eval before building, and compare changes against a baseline. An internal documentation assistant example showed that adding an IT ticket tool raised passing samples from 5 to 7 of 8 but wrongly opened tickets for VPN questions until the tool description was narrowed, after which all 8 passed.
Event version 1.
Key insights
Three-step EDD process
LangChain's eval-driven development process consists of defining what good looks like for an agent, writing the eval before building, and making a change and comparing it against a baseline.
Evals differ from traditional tests
Agents are non-deterministic, so outputs vary, more than one answer can be right, and the agent chooses its own steps. Evals score runs across a set of examples and can use other LLMs as judges.
Tool addition exposed a regression
In an internal documentation assistant with document search and web search, adding an IT ticket tool raised passing samples from 5 to 7 of 8 but also opened a ticket for a VPN how-to question that should have remained a consultation.
Narrowing tool description fixed failures
Execution records showed the tool description covered VPN and laptop issues too broadly. After the description was narrowed, all 8 samples passed, indicating the issue was the tool description rather than model capability.