Debate on GPU / TPU / Accelerator Performance and Market

Core Event: Discussion and benchmark comparisons surrounding the performance and market moat of NVIDIA GPUs versus Google TPUs. · Some argue that the CUDA moat is not deep, as skilled engineers can optimize performance on non-NVIDIA platforms. · It is pointed out that companies like Anthropic have long utilized TPUs, indicating that TPUs are not a recent adoption. · Hardware benchmarks (Artificial Analysis) show: at a reference rate of 30 tokens/second, the cost per million tokens for H100 is $1.06; for AMD MI300X it is $2.24; and for TPU v6e it is $5.13. · Reports indicate that TPU v7 is set to be released, but pricing is unknown. · NVIDIA officially responded by congratulating Google on its progress while reiterating the GPU's lead in performance, versatility, and interchangeability, stating that GPUs can run all models and be deployed in various computing environments; while ASICs (like TPUs) are efficient, they lack versatility. · The discussion suggests that hardware advantages are influenced by engineering talent and kernel/system optimization capabilities, and the market landscape is determined more by engineering implementation than by any single hardware.
Key insights
Skepticism About the Depth of NVIDIA's CUDA Moat
Some question the depth of NVIDIA's CUDA moat, arguing that there have been records of TPU usage prior to GPT, and that skilled engineers can achieve competitive performance on non-NVIDIA platforms through system and kernel optimization. This viewpoint cites companies like Anthropic, which have long used TPUs, emphasizing the decisive impact of engineering and software optimization on final performance rather than purely hardware architecture differences.
Benchmarks Show H100 Clearly Leading in Inference Costs
Artificial Analysis hardware benchmarks at a reference rate of 30 tokens/second indicate that the NVIDIA H100 has a cost of approximately $1.06 per million tokens, while the AMD MI300X costs $2.24 and the TPU v6e costs $5.13. This demonstrates a significant advantage for GPUs in terms of inference unit costs under this test. The report also mentions that TPU v7 is set to be released, but pricing details have not been provided.
NVIDIA Emphasizes the Versatility and Ecosystem Advantages of GPUs
In its response, NVIDIA congratulated Google on its progress but reiterated the leading advantages of GPUs in performance, versatility, and interchangeability. The statement noted that GPUs can run all AI models and be deployed in various computing environments, while ASICs (like TPUs) are more efficient for specific tasks but lack versatility and adaptability to changes in models/frameworks.
Hardware Competition Also Limited by Software and Engineering Capabilities
The discussion emphasizes that the performance differences of hardware are often amplified or diminished by software engineering implementation. The right kernel and system engineering talent can significantly improve performance on non-native hardware. Therefore, market competition is not only a battle of hardware architecture but also a contest of system optimization and engineering capabilities.