SGLang integrates NVFP4 KV cache for long-context inference boost

SGLang, Qwen, and NVIDIA collaborated to integrate NVIDIA's NVFP4 format into SGLang's KV cache, achieving about 1.78x capacity compression relative to FP8 with minimal accuracy loss. In agentic workloads with high prefix reuse, NVFP4 enables much higher throughput at high concurrency by doubling the number of concurrent requests that fit in memory. The key innovation uses two-level scaling (4-bit values with block-level FP8 scaling) to maintain precision at 4-bit quantization.