Cameron R. Wolfe Publishes Three-Part Roadmap for Evaluating AI Agents on Subjective Tasks

Cameron R. Wolfe, Ph.D. published a roadmap for building reliable evaluations of AI agents on highly subjective tasks, combining human evaluation, offline evaluation, and production monitoring. The framework includes statistical methods such as power analysis, confidence intervals, re-sampling, and paired comparisons to reduce noise and quantify uncertainty. Wolfe also outlined five criteria for selecting evaluation data: diverse, realistic, difficult, discriminative, and accurate.
Event version 1.
Key insights
Three-part evaluation framework
The roadmap combines human evaluation as the quality standard, offline evaluation using calibrated LLM judges or task-specific rubrics as a proxy, and production monitoring with A/B testing. These form a tiered process: offline evaluation, then human evaluation, then production A/B test, then full deployment.
Statistical tools for handling noise
Power analysis estimates the number of evaluation samples needed to detect meaningful improvements. Confidence intervals quantify uncertainty, re-sampling reduces within-task variance, and paired comparisons evaluate models on the same tasks to more reliably detect performance differences.
Five criteria for evaluation data
Evaluation tasks should be diverse, realistic, difficult, discriminative, and accurate. Techniques such as structured task taxonomies, model-based difficulty filtering, Item Response Theory, and LLM-based auditing help construct more informative evaluation sets.
Benchmark scores do not equal capability
An improved benchmark score does not always mean an agent is more capable. Well-designed evaluation tasks, calibrated human and model evaluators, and statistical confidence signals are needed to distinguish meaningful capability improvements from noise.