LLM infrastructure & performance techniques

Infrastructure innovations lower cost and latency for LLM apps. - Prompt Caching reuses KV cache to cut compute costs 90% - Prompt Caching reduces first-token latency ~85% - OpenAI uses automatic black-box routing for cache hits - Anthropic exposes manual cache control for 100% hit rates - Cache lifetimes often 5–10 minutes on H100 - MiMo‑V2‑Flash uses Hybrid SWA attention for long context - Fixed KV cache improves deployability and performance - MTP (multi-token prediction) increases accept length and speed - MOPD on-policy distillation fuses RL teachers with low compute cost - Engineering choices emphasize practicality and production readiness
Key insights
Prompt Caching yields major cost and latency savings
Prompt Caching reuses model attention KV matrices to avoid recomputation, reducing compute costs by about 90% and first-token latency by about 85% in measured deployments. OpenAI's automatic cache routing offers developer simplicity with varying hit rates, while Anthropic's manual cache-breakpoint approach achieves stable, near-100% hit rates when configured.
MiMo‑V2‑Flash and hybrid attention improve long-context inference
MiMo‑V2‑Flash leverages a Hybrid SWA attention design and fixed KV cache to outperform linear attention in long-context tasks. Engineering trade-offs include optimal window sizing (128 preferred over 512) and preserving 'sink values'. The design emphasizes deployment efficiency and compatibility with existing infrastructure.
Training techniques increase RL efficiency for production models
Techniques such as MTP (multi-token prediction) and MOPD on-policy distillation enable higher accept length, speedups and teacher-student fusion at much lower compute cost. MTP benefits on-policy RL and MOPD can replicate teacher performance using a fraction of compute, aligning training methods with production constraints.