NVIDIA Explains Dense vs. MoE Parameter Activation

NVIDIA released a technical explainer on how dense and Mixture-of-Experts (MoE) models use parameters differently, addressing how a 30B-parameter model can activate only 3B parameters per token while still leveraging the full model's capacity. It covers implications for throughput, memory, and serving complexity.