XNews

NVIDIA NIM Optimizations Boost Nemotron 3 Ultra Throughput by 2.5x

NVIDIA NIM Optimizations Boost Nemotron 3 Ultra Throughput by 2.5x

NVIDIA used its NIM microservices to optimize the Nemotron 3 Ultra model on a 4x B200 system, increasing system throughput from 718 to 1,997 tok/s. This allows the same hardware to serve approximately 2.78 times more concurrent users at a fixed 50 tok/s per user.

Source posts