XNews

HarnessDev Benchmark: LLMs Create Agent Harnesses, but Evolution Generalises Poorly

HarnessDev Benchmark: LLMs Create Agent Harnesses, but Evolution Generalises Poorly

ByteDance Seed et al. released HarnessDev, a benchmark testing whether LLMs can build their own agent harness (execution loop, tools, context management). In creation, six models built harnesses from a weak seed; domain gaps were large (code 69.3 vs. human 80.0, search research 52.6 vs. 92.2). Co‑adaptation was severe: Opus harnesses dropped from 69.3 to 33.0 when run by Gemini due to hardcoded step limits. Evolution over 10 rounds gave gains on feedback tasks (+3 to +14) but shrank to +1.4 to +4.4 on held‑out tasks. With a fixed Gemini executor, only Opus stayed positive; GPT‑5.5 declined to −10. Diagnosis was minimal – trajectory viewer used only twice. Feedback scoring proved unreliable for final version selection: only 2 of 9 self‑selected final versions were optimal on held‑out data.

Source posts