XNews

SGLang Integrates NVFP4 KV Cache for Long-Context Efficiency

SGLang Integrates NVFP4 KV Cache for Long-Context Efficiency

SGLang, Qwen, and NVIDIA collaborated to implement NVFP4 4-bit quantized KV cache in SGLang, achieving up to 78% decode throughput improvement at 1M context and significant gains in agentic workloads via higher cache hit rates. The two-level scaling (E2M1, FP8 block, FP32 global) provides good accuracy, though SWE-bench shows a 1.6pp drop.

Source posts