XNews

NVIDIA NIM optimizations boost Nemotron 3 Ultra throughput 2.78x on B200

NVIDIA NIM optimizations boost Nemotron 3 Ultra throughput 2.78x on B200

NVIDIA achieved a 2.78x concurrent user increase on a 4×B200 system by applying five NIM optimizations, including NVFP4 quantization, prefix caching, and MTP speculative decoding. Throughput rose from 718 to 1,997 tok/s under a 50 TPS per-user latency target.

Source posts