Garry Tan Open-Sources g-harness-evals Benchmark Comparing AI Agent Harnesses

Garry Tan released g-harness-evals, an open-source benchmark that holds a frontier model and 20 multi-user office tasks fixed while swapping only the agent harness. Across Opus 5.5 and GPT-6.1 Sol configurations, pass rates clustered between roughly 71% and 79%, while cost per solved task ranged from about $0.17 for Codex to $0.44 for OpenClaw on Opus. The author reported that cost and speed, not accuracy, separated the harnesses, and noted the task set is nearing saturation.
Event version 1.
Key insights
Harness accuracy differences not distinguishable
Pass rates across tested harnesses fell roughly between 71% and 79%, with Hermes on Opus at a point estimate of 78.8%. Fourteen of the 20 tasks scored identically across all combinations, and gaps under 8 to 14 points could not be confirmed.
Cost and speed separate harnesses
On Opus, OpenClaw cost about $0.44 per solved task with a median time near 99 seconds, while other harnesses cost about $0.19 to $0.24 and Claude Code and QM ran around 40 seconds. On Sol, Codex cost about $0.17 with a median near 30 seconds.
Grading uses system state and event logs
Scoring checks post-task system state and event logs for answer correctness, data leakage and unauthorized reads rather than relying on agent self-reports. Cost is calculated as total spend divided by passed tasks using metered proxy list prices.
Cross-channel corrections trip all harnesses
Tasks involving corrections raised in later conversations across channels were error-prone for every harness tested. The author noted the results come from a development set rather than a held-out set, with some OpenClaw combinations incomplete and QM network conditions not aligned with other groups.