SGLang, Qwen, NVIDIA Collaborate on NVFP4 KV Cache for Long Contexts

SGLang, Qwen, and NVIDIA collaborated to implement NVFP4 KV cache quantization in SGLang, using a two-level scaling scheme (4-bit E2M1 values with FP8 block scaling) to reduce memory usage to about 56% of FP8 while maintaining accuracy. Performance gains increase with context length, with up to 78% decode throughput improvement at 1M tokens under equal memory. Agentic workloads benefit doubly from reduced KV reads and higher cache hit rates.