XNews

Android Bench 2.0 Benchmark for Real-World Android Development by Google

Android Bench 2.0 Benchmark for Real-World Android Development by Google

Google introduced Android Bench 2.0, a benchmark measuring model+agent combinations on real Android development tasks using production apps like Signal and Bitwarden. The 30-task suite includes app conversion, architecture migration, and new feature development. Top scores reached 28%, with GPT-6 Astra+Codex and Claude Fable 5.1+Claude Code leading, while many XML-to-Compose migrations saw zero passes.

Key insights

Benchmark Design and Task Categories

Android Bench 2.0 evaluates model+agent pairs on 30 tasks derived from Google's Android best practices, using real production apps. Categories include app conversion (Flutter/React Native to native Android), building apps from scratch, architecture migration (XML to Compose, Hilt to Koin, etc.), and new feature development (CameraX, Media3, picture-in-picture). Architecture migration is the largest category with 13 tasks.

Results: Top Models Achieve 28% Pass Rate, Many Tasks Unconquered

GPT-6 Astra+Codex and Claude Fable 5.1+Claude Code tied for first with 28% pass rate. Qwen3.8 Max+Qwen Coder and Kimi K3+Kimi Code followed. Google's own Gemini models placed lower but with cost advantages. Several tasks saw zero passes from all models, including Flutter-to-Android conversion and most XML-to-Compose migrations, indicating these remain challenging for current AI.

Source posts