XNews

SGLang, Qwen, NVIDIA bring NVFP4 KV cache to cut memory and boost decode

SGLang, Qwen, NVIDIA bring NVFP4 KV cache to cut memory and boost decode

SGLang integrated NVIDIA's NVFP4 4-bit KV cache quantization, achieving ~56% of FP8 footprint with two-level scaling. Performance gains grow with context length: up to ~80% peak decode throughput at 1M tokens; agentic workloads benefit from higher radix cache hits.

Source posts