Discussion on Scaling, Training Infrastructure, and the AI Bubble

Ilya clarifies that continuing to scale in the current manner will lead to improvements, but will 'miss something important'; Andrew Ng breaks down AI into three layers: applications, inference, and training, pointing out that the training infrastructure is the most at risk of a bubble; an Apollo report shows that the rate of enterprise AI adoption is flattening or declining.
Key insights
Ilya: Continuing to Scale Will Improve, but Key Elements Are Missing
Ilya clarified in a tweet that continuing to scale through stacking computational power, data, and training environments will bring ongoing improvements and will not stagnate; however, he also pointed out that this approach will always 'miss something important.' He used an analogy with two students to illustrate that while continuous training can enhance performance, it cannot replace the ability to generalize and learn efficiently with minimal training.
Andrew Ng: Breaking Down AI into Layers to Assess Bubble Risks
In his article, Andrew Ng proposed dividing AI into three parts: the application layer, inference infrastructure, and training infrastructure. He believes that the application layer is underestimated and still has significant growth potential; the inference layer currently faces supply constraints while demand is high; the training layer is capital-intensive and easily replaceable by open-source solutions, making it the most likely sector to experience asset bubbles and return risks.
Signals of Slowing Enterprise AI Adoption Rates
A report from Apollo Academy, citing data from the U.S. Census Bureau and Ramp, indicates that the pace of AI adoption among large enterprises is stagnating or even declining, suggesting that there are barriers to transition from deployment to sustained adoption, and that short-term adoption rates are not on a continuously exponential rise.
Investment Risks and Market Impact of Training Infrastructure
Andrew Ng pointed out that training infrastructure is a capital-intensive and risk-concentrated area. The performance improvements of open-source large models and optimizations in algorithms/hardware may quickly reduce training costs, thereby weakening expectations of proprietary models as a competitive moat. If the training sector overheats and collapses, it could trigger a chain reaction of capital withdrawal from the application side.