XNews

SGLang, Qwen, NVIDIA bring NVFP4 KV cache to long-context inference

SGLang, Qwen, NVIDIA bring NVFP4 KV cache to long-context inference

SGLang, Qwen, and NVIDIA integrated NVIDIA's NVFP4 4-bit KV cache format into SGLang, using two-level scaling to cut KV memory to about 56% of FP8. Accuracy stayed near FP8 on a 397B model, while decode throughput rose up to 78% at 1M context under equal-capacity loading. Agent workloads gained from higher radix cache hit rates.

Source posts