NVIDIA NIM optimizations boost Nemotron 3 Ultra throughput

NVIDIA showed that NIM microservice optimizations increased Nemotron 3 Ultra system throughput from 718 to 1,997 tok/s on a 4x B200 system, enabling 2.78x more concurrent users at 50 tok/s per user. The optimizations span precision, parallelism, prefix caching, scheduling, and speculative decoding.
Key insights
Full-stack NIM optimization achieves 2.5x throughput gain
NVIDIA's NIM microservice applied five layers of optimization: NVFP4 quantization with auto-tuned kernels for MoE and Mamba, TP=4 with expert-aware execution, prefix caching with partial-prefix matching and Mamba state caching, scheduler/batch/memory tuning, and MTP speculative decoding. The result: 1,997 tok/s from 718 tok/s on 4x B200, serving 2.78x more concurrent users at 50 tok/s.
Optimizations are interdependent, not additive
The blog notes that optimizations form an interacting configuration bundle, not independent switches. Quantization changes memory headroom, which affects batch size, which affects speculative decoding acceptance rate. Therefore, individual contribution breakdowns are not provided. This reflects the reality of real system optimization.
Agentic workloads benefit most from these optimizations
The benchmark used agentic workload characteristics: 64K token input, 400 token output, 76% KV reuse rate. This long-input-short-output pattern with high prefix reuse makes prefix caching, state reuse, and speculative decoding particularly effective. The optimizations are tailored for the agentic era where inference capacity is a key cost bottleneck.