Benchmarks and model ranking analysis
Researchers propose BENCHMARK² to evaluate quality of LLM benchmarks. - BENCHMARK² uses three metrics: ranking consistency, discriminability and capability-alignment deviation. - Ranking consistency measures whether benchmark rankings align across datasets using Kendall's τ. - Discriminability measures a benchmark's ability to separate model performance; thresholds provided (>0.4 strong, <0.2 weak). - Capability alignment detects expectation reversals within model families (smaller model outperforms larger unexpectedly). - Applied to 15 benchmarks and 11 models: math benchmarks varied widely (AIME high, MATH-500 low); reasoning benchmarks trade off alignment vs discriminability (ARC high alignment/low discriminability; BBH inverse). - The framework enables constructing compact test sets (35% of items) that retain ranking and stability; authors recommend minimum thresholds for benchmarks.
Key insights
BENCHMARK² Provides Three-Axis Quality Scoring for LLM Benchmarks
The proposed BENCHMARK² framework evaluates benchmarks on ranking consistency, discriminability and capability-alignment deviation. Ranking consistency uses Kendall’s τ to compare benchmark rankings across datasets; discriminability gauges whether a benchmark differentiates models; capability-alignment deviation measures unexpected performance inversions within model families. The framework is intended to surface low-quality tests and reduce misleading evaluations.
Empirical Results Show Trade-offs Across Benchmark Types
Applying BENCHMARK² to 15 benchmarks and 11 models, authors found math benchmarks vary (AIME strong; MATH-500 shows ceiling effects), reasoning benchmarks trade alignment vs discriminability (e.g., ARC high alignment/low discriminability; BBH opposite), and knowledge benchmarks are more stable. Objective-answer tasks show higher alignment scores.
Compact, High-Quality Test Sets Can Replicate Full Benchmark Rankings
Researchers demonstrate that selecting items with high capability-alignment and discriminability can produce a compact benchmark using about 35% of original items while preserving ranking consistency and stability, enabling more efficient evaluation.