SGLang, Qwen, NVIDIA Integrate NVFP4 KV Cache to Boost Long-Context and Agentic Inference

SGLang, Qwen, and NVIDIA collaborated to integrate NVIDIA's NVFP4 format into SGLang's KV cache, achieving 4-bit precision with minimal accuracy loss. This reduces KV cache memory to 56.25% of FP8, improving decode throughput by 26-30% at equal concurrency and up to 78% at equal capacity for long contexts. For agentic workloads, NVFP4 enables higher cache hit rates and sustained throughput at high concurrency.