XNews

AI Hardware Benchmark and Vendor Responses

AI Hardware Benchmark and Vendor Responses

Hardware benchmark comparisons show that NVIDIA leads in inference costs, while TPU v6e and AMD MI300X have higher costs. • At a reference rate of 30 tokens/second, the cost per million tokens for NVIDIA H100 is $1.06. • AMD MI300X costs $2.24 per million tokens. • Google TPU v6e costs $5.13 per million tokens. • The report states that NVIDIA's tokens per dollar are five times that of TPU v6e and twice that of AMD. • TPU v7 is set to be released soon, with expected significant performance improvements, but pricing is unknown. • NVIDIA officially responded by praising Google's AI advancements while reiterating the industry-leading advantages of its GPU platform in performance, versatility, and interchangeability, emphasizing the generality of GPUs compared to ASICs. • A complete benchmark page is referenced to view results for different models (such as Llama 3.3 70B) on TPU v6e.

Key insights

Benchmark Shows H100 Leading in Inference Costs

The hardware benchmark from Artificial Analysis indicates that at a reference rate of 30 tokens/second, the cost per million tokens for NVIDIA H100 is $1.06, while AMD MI300X is $2.24 and Google TPU v6e is $5.13. The benchmark also notes that NVIDIA's tokens per dollar are five times that of TPU v6e and twice that of AMD, indicating that under these reference rates and testing conditions, H100 has the lowest inference costs.

NVIDIA Responds: Acknowledges Competitive Progress and Emphasizes GPU Versatility Advantage

NVIDIA officially congratulated Google on its advancements while reiterating the leading position of its GPU platform in terms of performance, versatility, and interchangeability. NVIDIA emphasized that GPUs can run various models and be deployed in multiple computing environments, whereas ASICs (like TPU), while efficient, are highly specialized and have lower flexibility in diverse models and rapidly evolving environments.

TPU v7 Set to Launch, Potentially Changing Performance Landscape

The benchmark report mentions that TPU v7 is set to be released soon and is expected to bring a significant leap in performance, although there is currently no public pricing information available. The benchmark results also remind users to pay attention to the upcoming release and pricing changes of the next generation of ASICs when selecting hardware.

Source posts