Kimi Moonshot: Scaling Advances and Kimi K3 Release

Kimi Moonshot announced Kimi K3 and detailed K2.5→K3 technical advances. - Announced Kimi K3: 2.8 trillion parameters, 1M native context, multimodal. - Kimi Delta Attention enables up to 6.3× faster decoding in million-token contexts. - Attention Residue yields ~24–25% higher token training efficiency at <2% cost. - Kimi K2.5 replaced Adam with MuonClip for higher token utilization. - MuonClip (based on Muon) achieved near doubling of data efficiency on same data. - QK-Clip prevents attention logit spikes (observed >1000) enabling Muon at trillion-scale. - Kimi Linear (KDA) is a linear attention mixing 3:1 with full attention to scale to million+ tokens. - KDA uses multiple decay channels to retain/forget information at different speeds. - Agent Swarm orchestration supports up to 100 agents in K2.5, 300 in K2.6, 1500-step tasks. - Early-fusion multimodal training (mixed visual+text) improved text reasoning and enabled zero-vision SFT. - Attention Residue (block version) allows layers to attend to earlier layer outputs via attention, replacing residuals in blocks. - Moonshot open-sourced MuonClip, Kimi Linear, and Attention Residue; K3 weights planned open by July 27, 2026.
Key insights
Kimi K3 launch and capabilities
Kimi Moonshot announced Kimi K3 as an open frontier intelligence model featuring 2.8 trillion parameters, a native 1 million token context window, and native multimodal support. The official release highlights that Kimi Delta Attention accelerates decoding up to 6.3× in million-token contexts and that Attention Residue improves token training efficiency by roughly 24–25% while adding less than 2% computational cost. K3 is positioned for long-horizon agentic coding and self-evolving workflows with weights planned to be open by July 27, 2026.
MuonClip and QK-Clip improve data efficiency and stability
Moonshot replaced Adam with MuonClip (derived from Muon) to increase token utilization: experiments show similar gains to doubling high-quality token counts when training on the same data. Muon enabled training on >15 trillion tokens but caused attention logit instability (max logits spiking above 1000); Moonshot introduced QK-Clip to monitor and synchronously scale Q and K projections, preventing loss spikes and enabling stable trillion-parameter training runs without divergence.
Kimi Linear (KDA) for million-token contexts
Kimi Linear uses Kimi Delta Attention (KDA), a linear attention mechanism that decomposes memory decay into multiple channels so different information decays at distinct rates. The architecture mixes linear attention with full attention at a 3:1 ratio and targets long-input and long-output tasks. Moonshot reported KDA outperforms full attention across short context and long context tasks and scales efficiently to million+ token inputs with substantial compute savings.
Agent Swarm: parallel multi-agent orchestration
Moonshot described Agent Swarm: hierarchical agent teams where a lead agent (CEO) decomposes goals and dispatches up to hundreds of specialized sub-agents. Reward design includes instantiation, completion, and final-result rewards, dynamically reweighted during training. K2.5 supported up to 100 parallel agents (1500-step tasks); K2.6 raised the parallel limit to 300 agents. Demonstrations include parallel research, large literature reviews, multi-chapter drafting, and concurrent dataset analyses.
Early-fusion multimodal training increases cross-modal transfer
Moonshot shifted from late-fusion to early-fusion training: K2.5 trained ~15 trillion mixed visual+text tokens from the start rather than appending visual SFT later. They observed visual-only training improved text reasoning; strong text base models allowed 'zero visual SFT' (supervised fine-tuning) with subsequent multimodal RL training to reach near state-of-the-art vision performance. Early fusion enabled cross-modal transfer used for tasks like 'see-and-code'.
Attention Residue replaces block residuals to boost efficiency
Moonshot introduced Attention Residue (Block Attention Residual) to let each block attend to outputs of many prior blocks and choose which historical signals to retain. They group layers into blocks (e.g., 16-layer blocks) and use attention between blocks; experiments reported ~24% token efficiency improvement and notable gains on GPQA, MATH, and HumanEval. The team positioned Attention Residue as a replacement for vanilla residual connections to make deep networks more selective.