NVIDIA and SGLang Advance Inference Optimization with NVFP4 and NIM

NVIDIA and SGLang detailed major inference optimizations. The NVFP4 4-bit floating point format for KV cache reduces memory footprint by 1.78x over FP8 with negligible accuracy loss, significantly boosting throughput for long-context and agentic workloads. NVIDIA's NIM stack optimized Nemotron 3 Ultra to serve 2.78x more concurrent users.
Key insights
NVFP4 KV Cache: Precision and Performance
NVIDIA and SGLang integrated the NVFP4 format into SGLang's KV cache. NVFP4 uses a two-level scaling scheme (4-bit E2M1 values + FP8 block scaling) to achieve 1.78x memory reduction over FP8 with minimal accuracy loss. In tests, Qwen3.5-397B showed almost no degradation. The decode kernel reads NVFP4 directly, converting on-the-fly to avoid memory round-trips, making it ideal for memory-bound decode phases.
NIM Full-Stack Optimization for Nemotron
NVIDIA used its NIM microservices to optimize Nemotron 3 Ultra (550B-A55B MoE+Mamba model) on 4x B200 systems. Through a combination of NVFP4 quantization, expert-parallel execution, prefix caching, Mamba state caching, and MTP speculative decoding, the system throughput increased from 718 to 1,997 tok/s. This allows the same hardware to serve 2.78x more concurrent users at a 50 tok/s interactive target.
Impact on Agentic Workloads
Both optimizations show outsized benefits for agentic workloads. NVFP4's memory savings allow radix trees to retain more shared prefixes (system prompts, tool definitions), drastically increasing cache hit rates and avoiding expensive recomputation. In AgentX benchmarks, NVFP4 maintained throughput at high concurrency where FP8 collapsed. NIM's optimizations were specifically profiled on a 'long-input, short-output' agentic workload with high KV reuse.