Xiaomi and NVIDIA Partner on KV Cache and NVFP4 for Efficient Inference

SGLang, Qwen, and NVIDIA implemented NVFP4 KV cache in SGLang, achieving 1.78x memory compression over FP8 with minimal accuracy loss. On agent workloads, NVFP4 enables higher concurrency and cache hit rates, improving decode throughput by up to 78% for 1M context. Xiaomi's MiMo-V2.6 models also offer cost-efficient inference.
Key insights
NVFP4 KV Cache: 4-bit Quantization with Two-Level Scaling
NVIDIA's NVFP4 format uses two-level scaling: 4-bit E2M1 values with FP8 block scaling per 16 values and FP32 global scaling. This achieves 56.25% of FP8 memory usage (1.78x compression). SGLang implements three attention paths: initial prefill uses BF16 Q x FP8 K/V, chunked prefill quantizes to NVFP4, and decode reads NVFP4 directly with on-the-fly dequantization using NVIDIA's cvt.rn.bf16x2.e2m1x2 instruction.
Performance Gains: Up to 78% Decode Throughput Improvement
On a single RTX PRO 6000 Blackwell with Qwen3.8-27B, NVFP4 shows 26-30% decode throughput improvement at equal concurrency. At equal memory capacity, concurrency increases from 44 to 70 (37% throughput gain) for 32K context, and from 1 to 2 for 1M context (78% gain). Prefill TTFT increases only 0.2-0.4% at equal concurrency. Agent workloads benefit doubly from reduced KV read and higher cache hit rates.
Accuracy: Minimal Loss but Task-Dependent
On Qwen3.5-397B-A17B, accuracy is nearly identical to FP8: GSM8K drops 0.08 pp, GPQA and AIME unchanged. On Qwen3.8-27B, GSM8K drops 0.31 pp, GPQA 1.01 pp, but SWE-bench Verified drops 1.60 pp (76.20% vs 77.80%). The authors note that agent tasks with long trajectories are more sensitive to KV quantization. The FP32 global scaling was not calibrated, suggesting potential for further improvement.
MiMo-V2.6: Cost-Efficient Open-Source Models
Xiaomi's MiMo-V2.6 Pro and Flash models offer competitive performance at low cost. Pro scores 46 on Artificial Analysis, matching xAI Grok 4.7, with API pricing at $0.435/$0.87 per million tokens. Flash is even cheaper at $0.14/$0.28, with only a few points lower on agent benchmarks. Both support 1M token context and multimodal input under MIT license.