XNews

SGLang, Qwen, NVIDIA Implement NVFP4 KV Cache, Boosting Decode Up to 78%

SGLang, Qwen, NVIDIA Implement NVFP4 KV Cache, Boosting Decode Up to 78%

SGLang, Qwen, and NVIDIA collaborated to implement NVFP4 KV Cache in SGLang, using NVIDIA's 4-bit format with two-level scaling to halve memory footprint vs FP8 while maintaining accuracy. Decode throughput improved 26-30% at equal concurrency and up to 78.46% at equal capacity, with agent workloads showing the largest gains.

Source posts