XNews

NVIDIA Optimizes Nemotron 3 Ultra for 2.5x Concurrent Users

NVIDIA Optimizes Nemotron 3 Ultra for 2.5x Concurrent Users

NVIDIA used NIM microservices to optimize the inference stack on a 4x B200 system, boosting Nemotron 3 Ultra throughput from 718 to 1,997 tok/s. This allows the same hardware to serve 2.78x more concurrent users at 50 tok/s per user. Key optimizations include NVFP4 quantization, prefix caching, and speculative decoding.

Source posts