XNews

SGLang, Qwen, NVIDIA Collaborate on NVFP4 KV Cache for Long Contexts

SGLang, Qwen, NVIDIA Collaborate on NVFP4 KV Cache for Long Contexts

SGLang, Qwen, and NVIDIA collaborated to implement NVFP4 KV cache quantization in SGLang, using a two-level scaling scheme (4-bit E2M1 values with FP8 block scaling) to reduce memory usage to about 56% of FP8 while maintaining accuracy. Performance gains increase with context length, with up to 78% decode throughput improvement at 1M tokens under equal memory. Agentic workloads benefit doubly from reduced KV reads and higher cache hit rates.

Source posts