XNews

SGLang × Qwen × NVIDIA: NVFP4 KV Cache boosts long-context and agentic inference

SGLang × Qwen × NVIDIA: NVFP4 KV Cache boosts long-context and agentic inference

SGLang, Qwen, and NVIDIA collaborated to implement NVFP4 KV Cache in SGLang, using NVIDIA's Blackwell FP4 format with two-level scaling (block-level E4M3 and global FP32). This reduces KV cache memory to ~56% of FP8 while maintaining near-lossless accuracy on large models. For agentic workloads with shared prefixes, NVFP4 enables higher concurrency and cache hit rates, delivering up to 78% decode throughput improvement at 1M context.

Source posts