XNews

NVIDIA NIM Optimizations on B200 Double Nemotron 3 Ultra Throughput

NVIDIA NIM Optimizations on B200 Double Nemotron 3 Ultra Throughput

NVIDIA demonstrated that full-stack synergy with NIM microservices on 4×B200 systems improved Nemotron 3 Ultra 550B-A55B throughput from 718 to 1,997 tok/s (2.5x), enabling 2.78x concurrent users at a 50 tok/s SLO. Key optimizations included NVFP4 quantization, expert-aware tensor parallelism, prefix caching, and MTP speculative decoding. All optimizations interact non-additively.

Source posts