XNews

Garry Tan Open-Sources g-harness-evals Benchmark Comparing AI Agent Harnesses

Garry Tan Open-Sources g-harness-evals Benchmark Comparing AI Agent Harnesses

Garry Tan released g-harness-evals, an open-source benchmark that holds a frontier model and 20 multi-user office tasks fixed while swapping only the agent harness. Across Opus 5.5 and GPT-6.1 Sol configurations, pass rates clustered between roughly 71% and 79%, while cost per solved task ranged from about $0.17 for Codex to $0.44 for OpenClaw on Opus. The author reported that cost and speed, not accuracy, separated the harnesses, and noted the task set is nearing saturation.

Event version 1.

Key insights

Harness accuracy differences not distinguishable

Pass rates across tested harnesses fell roughly between 71% and 79%, with Hermes on Opus at a point estimate of 78.8%. Fourteen of the 20 tasks scored identically across all combinations, and gaps under 8 to 14 points could not be confirmed.

Cost and speed separate harnesses

On Opus, OpenClaw cost about $0.44 per solved task with a median time near 99 seconds, while other harnesses cost about $0.19 to $0.24 and Claude Code and QM ran around 40 seconds. On Sol, Codex cost about $0.17 with a median near 30 seconds.

Grading uses system state and event logs

Scoring checks post-task system state and event logs for answer correctness, data leakage and unauthorized reads rather than relying on agent self-reports. Cost is calculated as total spend divided by passed tasks using metered proxy list prices.

Cross-channel corrections trip all harnesses

Tasks involving corrections raised in later conversations across channels were error-prone for every harness tested. The author noted the results come from a development set rather than a held-out set, with some OpenClaw combinations incomplete and QM network conditions not aligned with other groups.

Source posts