XNews

NVIDIA Optimizes Nemotron 3 Ultra to Serve 2.5x More Users on Same Hardware

NVIDIA Optimizes Nemotron 3 Ultra to Serve 2.5x More Users on Same Hardware

NVIDIA demonstrated a full-stack optimization with its NIM microservices, improving the Nemotron 3 Ultra 550B-A55B model's system throughput from 718 to 1,997 tok/s on a 4×B200 system. This allows the same hardware to serve approximately 2.78 times more concurrent users while maintaining a consistent 50 TPS user experience. The optimizations included NVFP4 quantization, expert-aware execution, prefix caching, and speculative decoding.

Source posts