XNews

EvolveScaler: A Benchmark for Information Evolution

EvolveScaler: A Benchmark for Information Evolution

Tencent introduced EvolveScaler, a benchmark for information evolution where records are retracted, corrected, and backfilled. It defines the world as an executable state machine rendered into natural language. Frontier models score as low as 11.3% on the hardest tier.

Key insights

Benchmark Design

EvolveScaler defines the world as an executable state machine, then renders it into natural language. It includes 117 prototypes, 159 question operators, 5 difficulty tiers, and up to ~1,200 events per sample. The benchmark tests models' ability to handle information that changes after reading.

Model Performance

On the hardest tier, median avg@5 for 14 frontier models falls to 11.3%. Training on EvolveScaler improves average performance by +5.25 across 8 out-of-distribution benchmarks.

Source posts