Training and Computing Power Competition: TPUv7 and Risks in Training Infrastructure

Analysis indicates that TPUv7 (Ironwood) is close to or matches NVIDIA Blackwell in terms of single-chip computing power, memory, and bandwidth, while having advantages in large-scale interconnectivity and total cost of ownership. However, researchers warn that training infrastructure is a center of risk. - TPUv7 FP8 computing power is approximately 4.6 PFLOPS, compared to NVIDIA B200's 4.5 PFLOPS. - TPUv7 is equipped with 192GB HBM3e and has a bandwidth of about 7.4 TB/s. - Google can connect up to 9,216 TPUv7 chips into a super-large cluster using optical switching and 3D Torus interconnect. - Estimates show that the total cost of ownership for TPUv7 systems is about 44% lower than that of NVIDIA GB200 solutions. - Anthropic has committed to purchasing and using large-scale TPU clusters, indicating Google's shift towards commercializing its computing power. - Andrew Ng breaks down AI into three layers: applications, inference, and training, noting that training infrastructure is capital-intensive and carries the highest risk; the application layer is underestimated, and the inference layer faces supply constraints.
Key insights
TPUv7 Matches or Exceeds Blackwell in Single Chip and Memory Specifications
Analysis indicates that TPUv7's FP8 computing power is approximately 4.6 PFLOPS, equipped with 192GB HBM3e and about 7.4 TB/s memory bandwidth, placing its single-chip performance metrics on par with NVIDIA Blackwell (B200), demonstrating a significant breakthrough in hardware design by Google.
TPUv7's Scalable Interconnect and Cluster Size as a Differentiated Advantage
Google's optical switching and 3D Torus interconnect architecture can connect thousands to 9,216 TPUv7 chips into a unified super-large cluster. This large-scale, low-latency interconnect significantly reduces communication bottlenecks when training ultra-large models, which is key to its competitive advantage.
TPUv7's Total Cost of Ownership is Highly Competitive
Analysis estimates that Google's vertically integrated TPUv7 solution has a total cost of ownership approximately 44% lower than deploying NVIDIA GB200 server solutions, making it economically attractive for organizations requiring large-scale training computing power.
Training Infrastructure as a Concentration of Capital and Risk, with the Application Layer Underestimated
Andrew Ng proposes breaking down AI into applications, inference, and training layers: the application layer is currently underestimated and its long-term value overlooked; the inference layer faces supply constraints; the training layer is capital-intensive and carries the highest risk, with open-source models and hardware advancements potentially weakening the moat of any single vendor.