XNews

Kimi K2.5 and model optimization advances (Kimi & Gemma 4)

Kimi K2.5 and model optimization advances (Kimi & Gemma 4)

Kimi K2.5 and LiveKit announce AI advances.

Key insights

MuonClip and QK-Clip boost token efficiency and stability

Kimi K2.5 replaces the Adam optimizer with MuonClip, a Muon-based optimizer that increases token utilization so that using the same data yields results comparable to doubling data volume. Muon-based training produced runaway attention logits (max logit >1000), so Kimi introduced QK-Clip to monitor and synchronously rescale Q and K projections per attention head; this preserved convergence and enabled Muon to scale to trillion-parameter models and training runs over 15 trillion tokens without loss spikes. All related components have been open-sourced.

Kimi Linear (KDA) extends usable context efficiently

Kimi Linear implements Kimi Delta Attention (KDA), a linear attention mechanism that splits decay into multiple channels so different information streams retain different memory durations. KDA mixes linear and full attention at a 3:1 ratio, allowing the architecture to outperform full attention on short-context, long-input, and long-output tasks and scale context to million-token ranges with lower compute than full attention.

Agent Swarm converts serial Agent work into parallel teams

Kimi introduces Agent Swarm: a hierarchical multi-agent system where a main Agent acts as CEO and spawns specialist subagents (researcher, programmer, analyst, fact-checker). Training used three dynamic rewards—instantiation, completion, and final-result rewards—to encourage creating parallelizable sub-tasks, ensuring sub-tasks complete, and optimizing final outcomes. K2.5 supports up to 100 parallel agents and 1,500 steps; K2.6 increases the agent cap to 300.

Early visual-text fusion yields cross-modal gains

K2.5 used early fusion training by mixing visual and text tokens from the start, training about 15 trillion mixed tokens. Experiments showed that visual-only training improved text reasoning, and a zero-visual-SFT pipeline—supervised fine-tuning on text only followed by multimodal RL—achieved near-state-of-the-art visual performance. The team reports bi-directional transfer between modalities after placing text and vision in a shared representation space.

Attention Residue replaces residual connections for efficiency

Attention Residue replaces standard residual connections with an attention-based mechanism that lets each layer attend to all previous layer outputs, selecting which historical information to preserve. To control cost, the team uses Block Attention Residual (blocks of ~16 layers) and reports that about eight blocks capture most benefits. Experiments show roughly a 24% token efficiency improvement on reasoning, math, and coding benchmarks.

LiveKit optimizes Gemma 4 31B for low-latency, low-cost inference

LiveKit announced a new service running Google’s Gemma 4 31B on LiveKit Inference. Measured against GPT-4.1, Gemma 4 31B on LiveKit showed 5.2× lower first-word latency (192 ms vs 1,006 ms) and approximately 83% lower cost (about one-sixth). In a hotel front-desk scenario, Gemma 4 31B achieved 88% task completion versus GPT-4.1 at 73% and Gemini 2.5 Flash at 64%.

Source posts