XNews

Kimi K2.5 and K3: Innovations and Benchmarks

Kimi K2.5 and K3: Innovations and Benchmarks

Moonshot AI released Kimi K3 on July 16, 2026, a 2.8-trillion-parameter model with 1-million-token context, native multimodality, Kimi Delta Attention (up to 6.3x faster decoding), and Attention Residuals (~25% training efficiency gain at <2% cost). Open weights are promised by July 27. On the DeepSWE benchmark, K3 max vs. GPT-5.6 Sol max showed complementary strengths: pass@1 68.5% vs 72.7%, but pass@4 89.4% vs 85.8%; K3 cost ~$4.65/rollout vs Sol's $8.37, yielding 14.7 vs 5.3 solved tasks per $100. K3 excelled in Rust and ops tools, Sol in Python/TS/JS and serialization. A cascade strategy (Kimi first, then Sol) achieved 85.6% solve rate at $7.30/task. Earlier, founder Yang Zhilin detailed K2.5 innovations: MuonClip optimizer replacing Adam, Kimi Linear attention mixing linear and full attention 3:1, Agent Swarm for up to 300 parallel agents, early fusion multimodal training, and Attention Residue replacing residual connections. SemiAnalysis noted K3 significantly outperformed NVIDIA Nemotron 3 Ultra, arguing open-source competition drives progress.

Key insights

Kimi K3 delivers frontier-level performance with open weights and cost efficiency

K3's 2.8T parameters, 1M context, and native multimodality are paired with open weights by July 27. On DeepSWE, it trails GPT-5.6 Sol on pass@1 but surpasses it on pass@2 and pass@4, while costing nearly half per rollout and solving almost 3x more tasks per dollar.

K3 and GPT-5.6 Sol have complementary strengths, enabling effective model routing

K3 is stronger in Rust and ops tools; Sol leads in Python/TS/JS and serialization. Task-level correlation is ~0.46, and the two-model union covers 95.6% of tasks. A cascade of Kimi first then Sol achieves higher coverage at lower cost than either alone.

K2.5 introduced foundational replacements for Adam, full attention, and residual connections

MuonClip optimizer improves token efficiency to nearly double the data; Kimi Linear mixes linear and full attention 3:1, outperforming full attention on short and long contexts; Attention Residue replaces residual connections with attention over previous layers, yielding ~24% token efficiency gain. All are open-sourced.

Agent Swarm enables parallel multi-agent collaboration for complex tasks

K2.5's Agent Swarm organizes a CEO agent that delegates to up to 300 sub-agents (researcher, coder, analyst, fact-checker), converting serial workflows into parallel execution. Training uses three dynamic rewards: instantiation, completion, and final result. This significantly reduces task completion time.

Early fusion multimodal training transfers visual learning to text reasoning

K2.5 mixes visual and text data from the start. Training only on visual tasks improved text reasoning; a strong text base allowed zero visual SFT, with vision emerging from RL. This bidirectional transfer suggests shared representations benefit both modalities.

Open-source competition from China is outpacing some US efforts

SemiAnalysis stated K3 significantly surpassed NVIDIA's Nemotron 3 Ultra and argued that US leadership requires independent lab competition, not consolidation. The rapid iteration of Chinese AI labs was highlighted as a model for progress.

Source posts