XNews

Kimi K2.5 Technical Deep Dive and Architecture Innovations

Kimi K2.5 Technical Deep Dive and Architecture Innovations

In a GTC 2026 talk, Moonshot AI founder Yang Zhilin detailed how Kimi K2.5 replaced Adam optimizer, full attention, and residual connections with novel components. These advances were later integrated into the open-source Kimi K3 model launched in July 2026. Core technical contributions: • MuonClip optimizer: replaces Adam; improves data efficiency by nearly 2×, approaching the effect of doubling training data. To stabilize training at trillion-parameter scale, a QK-Clip method was added that clamps attention logits in the forward pass, preventing divergence over 15T tokens. • Kimi Linear (KDA): a linear attention mechanism with multi-decay channels that lets different information streams fade at different rates. It mixes linear and full attention at a 3:1 ratio and claims to be the first architecture to outperform full attention on short, long input, and long output tasks simultaneously. • Agent Swarm: a multi-agent training scheme with three rewards—instantiation (encourage parallelism), completion (prevent empty subtasks), and final result. K2.5 supports 100 parallel agents; K3 raised this to 300. • Early fusion multimodal training: visual and text data mixed from the start led to cross-modal transfer—training on visual tasks alone improved text reasoning, and zero visual SFT was possible with a strong text base. • Attention Residue: replaces standard residual connections with block-level attention across previous layers, yielding ~24% token efficiency gain in reasoning, math, and coding benchmarks. K3 launch: 2.8 trillion parameters, 1M context, native multimodal, Kimi Delta Attention providing up to 6.3× faster decoding, and Attention Residuals delivering ~25% higher training efficiency. Open weights were slated for July 27, 2026.

Key insights

Redesigning Adam, Attention, and Residuals Yields Gains Without Extra Data

Moonshot AI's approach challenges the assumption that progress requires primarily more data or compute. By re-engineering the foundational components of the Transformer, they achieved compound benefits: optimizer efficiency (equivalent to doubling data), linear attention for long contexts, and attention residuals for deeper information flow. These are all open-source and can be independently adopted.

Kimi Linear's Multi-Decay Mechanism Could Set New Standard for Long-Context Efficiency

Unlike traditional linear attention with a single global decay, KDA uses per-channel decay rates, allowing the model to retain long-term information while quickly forgetting irrelevant details. The 3:1 hybrid with full attention demonstrates that efficient architectures can surpass full attention across all tested context lengths, making million-token contexts pragmatic.

Agent Swarm Training Enables Parallel Collaboration, Accelerating Complex Task Completion

By incentivizing sub-task parallelization and verifying completion, Moonshot AI trained models to coordinate multiple agents like a CEO. In practice, the approach reduces overall task time for research, code generation, and data analysis. Scaling to 300 agents in K3 extends this capacity into large-scale autonomous workflows.

Early Fusion Multimodal Training Demonstrates Cross-Modal Knowledge Transfer

Mixing visual and text data from the start yielded a surprising result: improving visual performance also boosted text reasoning, and a strong text base eliminated the need for dedicated visual SFT data. This supports the hypothesis that shared latent spaces allow capabilities learned in one modality to benefit another, potentially reducing the need for large paired datasets.

Attention Residue Breaks the Sequential Information Flow of Classical Residuals

By allowing each layer to attend to all previous layers within a block and select which information to keep, the architecture achieves a 24% token efficiency boost. This reimagines Ilya Sutskever's analogy of residuals as 'rotated LSTMs,' replacing the fixed skip connection with a learned, attention-based gating mechanism. Adoption could influence future deep network designs.

Source posts