XNews

NVIDIA NIM optimizations boost Nemotron 3 Ultra throughput by 2.5x on B200

NVIDIA NIM optimizations boost Nemotron 3 Ultra throughput by 2.5x on B200

NVIDIA achieved 2.5x more concurrent users for Nemotron 3 Ultra on 4x B200 systems through full-stack NIM optimizations including NVFP4 quantization, prefix caching, Mamba state caching, and speculative decoding. Throughput rose from 718 to 1,997 tok/s under a 50 tok/s per-user SLO.

Source posts