XNews

CursorBench 4.0: Private Benchmark for AI Coding Agents

CursorBench 4.0: Private Benchmark for AI Coding Agents

Cursor updated its private benchmark CursorBench 4.0, replacing public benchmarks like SWE-bench that have been overfit. The methodology uses 'Cursor Blame': tracing committed code back to agent requests to create query-answer pairs. Features: anti-leak measures (internal codebases, frequent task rotations), heavier tasks (monorepos, production log debugging, long experiments), and deliberately ambiguous descriptions to mimic real developer input. Evaluation covers correctness, code quality, efficiency, and interaction behavior. Online evaluation in production captures regressions not seen offline. An ablation study showed semantic search tool value in large codebase QA. The post also notes Claude Fable 5.1 and Opus 5 are strongest but expensive; GPT-6 Astra not evaluated; Grok 4.6 not impressive.

Key insights

CursorBench 4.0 Designed to Resist Benchmark Gaming

Tasks are sourced from internal codebases and controlled sources, refreshed every few months. The benchmark includes monorepos, multi-workspace, production log debugging, and longer experiments. Task descriptions are deliberately short and ambiguous, simulating real developer instructions.

Dual Online-Offline Evaluation Prevents Regressions

CursorBench measures correctness, code quality, efficiency, and interaction offline. Online controlled experiments on real traffic catch regressions where offline scores are high but user experience worsens.

Model Performance Comparison: Claude Fable 5.1 and Opus 5 Lead

Claude Fable 5.1 and Opus 5 are the strongest models but expensive. GPT-6 Astra is not evaluated. Grok 4.6 is not impressive, slightly better than Muse Spark 1.3 but more costly. Grok 4.7 promised improvements.

Source posts