SGLang, Qwen, NVIDIA: NVFP4 KV Cache boosts long-context and agentic inference

SGLang, Qwen, and NVIDIA collaborated to implement NVFP4 KV Cache in SGLang, achieving 1.78x capacity over FP8 with minimal accuracy loss. The technique uses two-level scaling and three attention paths. For agent workloads, NVFP4 provides dual benefits: reduced KV reads and higher cache hit rates, improving throughput by up to 78% at 1M context.