XNews

Qwen Open-Sources BabelArena Multilingual Agent Benchmark Across 23 Languages

Qwen Open-Sources BabelArena Multilingual Agent Benchmark Across 23 Languages

Qwen released BabelArena, an open-source multilingual benchmark for LLM agents covering 23 languages and roughly 16,000 task instances derived from 699 canonical tasks across four benchmark families and 13 domains. The benchmark was built with BabelFlow, a workflow that preserves task and evaluation semantics through structure-preserving translation, multi-layer verification and human review. Testing five frontier models showed no single model led across all families, and lower-resource languages produced more tool-use and control-flow errors and up to roughly twice the input tokens of English.

Event version 1.

Key insights

Benchmark scale and coverage

BabelArena comprises 699 canonical tasks from four agent benchmark families, spanning 13 domains and adapted into 23 languages for 16,077 task instances. Each language variant keeps the source task identity and evaluation criteria so performance can be compared on exactly the same tasks.

BabelFlow construction pipeline

BabelFlow traces each benchmark's runtime data flow to find text that can be safely translated, translates it consistently across tasks, tools, databases, simulated users and graders, and accepts a variant only after multi-layer verification and human review.

No single model dominates

Experiments with five frontier models show that no single model leads across all four benchmark families. Cross-language disparities extend beyond task success.

Lower-resource language failure patterns

Lower-resource languages show larger shares of tool-use and control-flow errors rather than answer-quality errors alone. On the same tasks, agents in low-resource languages consume up to roughly twice the input tokens of English without proportional increases in interaction length.

Language consistency degrades on structured output

Language consistency degrades further on tasks requiring structured output, where switches are directed overwhelmingly toward English.

Source posts