XNews

Advances and Benchmarks for AI Coding Agents

Advances and Benchmarks for AI Coding Agents

Benchmarking and evolutionary coding agents test end-to-end developer automation. - App-Bench evaluates zero-human full‑stack app generation - Six scenarios: finance dashboard, hospital board, legal assistant, pharmacy, Pictionary, rental platform - Scoring: 151 criteria, 4,530 manual evaluations, three attempts per tool - Top ranks: Orchids 76.8%, Claude Code 67.5%, v0 64.9%, Gemini CLI 0.0% - Best tools still cover ~77% of required features - Major failures: multi-role flows, API expiry, UI/UX and missing features - Web generators outperform CLI on stability and auth/UI - AlphaEvolve = Gemini + Evolution; Google announced private beta - AlphaEvolve runs evolutionary search: problem spec + evaluator + seed program - Internal gains: ~0.7% datacenter compute recovered, a kernel 23% speedup, 1% overall training time drop - AlphaEvolve offered as a service API for coded optimization

Key insights

App-Bench assesses zero-human, full‑stack app generation

App-Bench is a high-difficulty benchmark that tests whether an AI agent can generate a complete, usable modern web application from a single natural-language prompt with no human intervention. It uses six complex real-world scenarios (financial dashboard, hospital management board, legal assistant with RAG, pharmacy system, a multiplayer Pictionary-style game, and a rental booking platform). Scoring relied on manual evaluation by senior full-stack developers across 151 criteria, totaling 4,530 evaluations; each tool got three attempts and the best score was recorded.

Benchmark results show capability gaps and ranking details

App-Bench results placed Orchids first at 76.8%, Claude Code second at 67.5%, and v0 (Vercel) third at 64.9%, with Gemini CLI scoring 0.0%. Even the top tools reached only about 77% of required features. Common failure modes included missing multi-role workflows, expired or broken APIs causing runtime errors, and UI/UX issues that affected usability. Web-based generators showed slightly better average performance than CLI tools, which exhibited higher variance in handling authentication and foundational UI logic.

AlphaEvolve turns algorithm design into evolutionary search

AlphaEvolve, described as Gemini + Evolution, is an evolution‑based coding agent that accepts a problem specification, an objective evaluator, and a seed program, then evolves code variants via mutation and selection driven by Gemini models and automated evaluators. Google announced AlphaEvolve is in private beta and reported internal gains: about 0.7% global datacenter compute recovered in scheduling, a 23% speedup on a critical kernel improving training performance, and a 1% reduction in overall training time. The service is exposed via an API for external use.

Benchmark design and failure analysis emphasize end-to-end gaps

App-Bench’s methodology—complex scenarios, manual scoring, and multiple attempts per tool—highlights that current coding agents can write code fragments but struggle with end-to-end product requirements: multi-user roles, realtime sync, auth flows, and production-grade UI/UX. The benchmark’s manual, criterion-rich scoring revealed implementation and integration issues as the main shortfalls, not just code correctness. This suggests current agents need stronger system integration, runtime validation, and update handling to reach production readiness.

Source posts