EvolveScaler: A Benchmark for Information Evolution

Tencent introduced EvolveScaler, a benchmark for information evolution where records are retracted, corrected, and backfilled. It defines the world as an executable state machine rendered into natural language. Frontier models score as low as 11.3% on the hardest tier.
Key insights
Benchmark Design
EvolveScaler defines the world as an executable state machine, then renders it into natural language. It includes 117 prototypes, 159 question operators, 5 difficulty tiers, and up to ~1,200 events per sample. The benchmark tests models' ability to handle information that changes after reading.
Model Performance
On the hardest tier, median avg@5 for 14 frontier models falls to 11.3%. Training on EvolveScaler improves average performance by +5.25 across 8 out-of-distribution benchmarks.