NVIDIA Optimizes Nemotron 3 Ultra to Serve 2.5x More Users

NVIDIA used full-stack NIM optimization to increase Nemotron 3 Ultra system throughput from 718 to 1,997 tok/s on 4x B200, serving 2.78x more concurrent users at 50 tok/s per user. Optimizations include NVFP4 quantization, prefix caching, speculative decoding, and MoE-aware execution.
Key insights
Optimization Stack
Five layers: NVFP4 quantization with auto-tuned kernels for MoE and Mamba, TP=4/PP=1 parallelism with expert-aware execution, prefix and state caching (partial-prefix matching, Mamba state cache), scheduler/batch/memory tuning for latency budget, and MTP speculative decoding. The optimizations are interdependent, not additive.
Measurement Methodology
NVIDIA used AIPerf to replay real agentic traffic (64K input, 400 output tokens, 76% KV reuse). They swept concurrency from 1 to 64 to plot throughput-latency Pareto curves, then selected the operating point under a 50 tok/s per user SLO. The 2.5x figure is for fixed latency SLO, not raw throughput.
Implications for Agentic Workloads
The optimization is tailored for agentic scenarios with long inputs, short outputs, and high prefix reuse. Each improvement directly reduces unit token cost. The full-stack synergy (hardware-software co-design) is a moat for NVIDIA, as pure software solutions cannot replicate the vertical integration.