XNews

NVIDIA NIM Optimizes Nemotron 3 Ultra, Serving 2.78x More Users

NVIDIA NIM Optimizes Nemotron 3 Ultra, Serving 2.78x More Users

NVIDIA's NIM microservices optimized Nemotron 3 Ultra on 4×B200, increasing throughput from 718 to 1,997 tok/s, enabling ~2.78x more concurrent users at 50 tok/s per user. Optimizations included NVFP4 quantization, expert-aware parallelism, prefix caching, and speculative decoding.

Source posts