AI Agent Evals FAQ: Find Failures First, Then Evaluate
A practical FAQ on AI agent evaluations, based on experience from 700+ engineers, argues that the current industry narrative of building evaluation infrastructure and using LLM-as-a-Judge is backwards. The key insight is to first discover failures, then design evaluations around them.