XNews

Scale AI Launches SWE-Bench Pro Benchmark

Scale AI has launched the SWE-Bench Pro benchmark, a new evaluation tool designed for AI models that require complex codebase assessments. This benchmark includes features such as multi-file edits and an average of over 100 lines changed, addressing the need for more sophisticated evaluations in AI development. Current leading models have scored as follows: GPT-5 at 23.3% and Claude Opus 4.1 at 22.7%, while other models scored below 15%. This launch signifies a significant advancement in the benchmarking of AI capabilities in handling complex coding tasks.

Key insights

SWE-Bench Pro Benchmark Overview

The SWE-Bench Pro benchmark from Scale AI is a crucial tool for evaluating AI models on complex codebases, focusing on multi-file edits and significant line changes. The benchmark aims to enhance the assessment of AI capabilities in real-world coding scenarios.

Source posts