XNews

Cameron R. Wolfe Releases AI Engineer NYC Talk on Agent Evaluation Roadmap

Cameron R. Wolfe, Ph.D. released his AI Engineer NYC conference talk on YouTube, presenting a simple roadmap for evaluating AI agents on subjective and non-verifiable tasks. The talk covers human evaluations, quality guidelines, LLM judges and rubrics, calibration, data sampling, statistical tools, A/B testing, production monitoring, and continual benchmark evolution, accompanied by slides and a full written guide.

Event version 1.

Key insights

Roadmap targets subjective agent evaluation

The talk presents a simple roadmap for agent evals focused on subjective and non-verifiable tasks, with core concepts applicable to any evaluation. It combines human evaluation, offline evaluation, and production monitoring.

Evaluation challenges stem from agent scope and non-determinism

Agents have broad capabilities and solve long-horizon, realistic tasks, making evaluation expensive. Non-deterministic outputs across repeated executions make it difficult to draw reliable conclusions from benchmark scores.

Statistical and sampling techniques reduce noise

The roadmap includes statistical methods for reducing noise and quantifying uncertainty, plus data curation techniques for selecting diverse, realistic, and informative evaluation tasks. Calibration is applied to both humans and LLM judges.

Written guide expands on talk topics

A full written guide covers all talk topics in more detail and links to further reading resources. Slides from the presentation are also available.

Source posts