Unlock Editor's Digest without spending a dime
Roula Khalaf, editor of the FT, picks her favorite stories on this weekly newsletter.
The computational “law” that made Nvidia the most useful company on the planet is starting to interrupt. This just isn’t the famous Moore's Law, the semiconductor industry's maxim that chip performance increases by doubling transistor density every two years.
For many in Silicon Valley, Moore's Law has been replaced because the dominant predictor of technological progress by a brand new concept: the “scaling law” of artificial intelligence. This assumes that feeding more data right into a larger AI model – which in turn requires more computing power – results in smarter systems. This realization accelerated the progress of AI and shifted the main target of development from solving difficult scientific problems to the simpler technical challenge of constructing ever-larger chip clusters – often those made by Nvidia.
The law of scaling reached its peak with the introduction of ChatGPT. The rapid pace of improvement in AI systems within the two years since appeared to suggest that the rule could hold until we reach some form of “superintelligence,” perhaps inside this decade. However, over the past month, rumors have grown louder within the industry that the newest models from firms equivalent to OpenAI, Google and Anthropic haven’t shown the expected improvements according to the Scaling Act's predictions.
“The 2010s were the age of scaling, now we’re back within the age of wonder and discovery,” says OpenAI co-founder Ilya Sutskever told Reuters recently. This is the person who said a yr ago, he Thought It is “quite likely that the complete surface of the Earth can be covered with solar panels and data centers” to power AI.
Until recently, the scaling law was applied to “pre-training”: the basic step in constructing a big AI model. Now, AI executives, researchers and investors acknowledge that AI model capabilities – as Marc Andreessen put it in his podcast – “peak” at pre-training alone, meaning more work is required after the model is built to take care of progress Coming.
Some of the scaling law's early proponents, equivalent to Microsoft CEO Satya Nadella, have tried to rewrite its definition. It doesn't matter whether pre-training results in diminishing returns, advocates argue, because models can now “reason” when asked a fancy query. “We are witnessing the emergence of a brand new scaling law,” Nadella said recently, referring to OpenAI’s recent o1 model. But this type of fumbling is prone to make Nvidia's investors nervous.
Of course, the scaling law was never an iron rule, just as there was no inherent factor that allowed engineers at Intel to further increase transistor density in accordance with Moore's Law. Rather, these concepts function organizing principles for the industry and drive competition.
Still, the scaling law hypothesis has fueled “fear of missing out on the following big technology shift,” which has led to unprecedented investments in AI by major tech firms. According to Morgan Stanley, capital spending at Microsoft, Meta, Amazon and Google will exceed $200 billion this yr and exceed $300 billion next yr. Nobody desires to be the last to construct superintelligence.
But if greater not means higher in AI, will these plans be limited? Nvidia will likely suffer greater than most if this happens. When the chipmaker reported earnings last week, the primary query from analysts was about scaling laws. Nvidia CEO Jensen Huang insisted that pre-training scaling was “intact,” but admitted that it alone “isn’t enough.” The excellent news for Nvidia, Huang argued, is that the answer would require much more of its chips: so-called “test time scaling,” since AI systems like OpenAI's o1 could have to “think” longer to search out smarter answers.
That could be true. While training has taken up most of Nvidia's chips thus far, demand for computing power for “inference” – or how models reply to each individual request – is predicted to grow quickly as more AI applications emerge.
Those involved in constructing this AI infrastructure consider the industry could have a minimum of one other yr of playing catch-up in the case of inference. “Right now, it is a market that needs more chips, not fewer,” Microsoft President Brad Smith told me.
But in the long term, the seek for chips to power larger and bigger models before they hit the market has been replaced by something more closely tied to AI use. Most firms are still on the lookout for the killer app for AI, especially in areas that will require o1's emerging “reasoning skills.” Nvidia became the most useful company on the planet within the speculative phase of AI expansion. The scaling law debate underscores how much the longer term of business will depend on Big Tech realizing tangible returns on these huge investments.
