GoKawiilSince about 2020, AI has largely focused on training bigger and better models. Large language models (LLMs) ballooned from millions of parameters to trillions. This proved effective: The largest version of OpenAI’s GPT-3, released in 2020, correctly answered just 43.9 percent of questions on a popular knowledge-and-reasoning benchmark. Just four years later, GPT-4o reached a score of 88.7 percent on the same exam, effectively matching those of human experts.
Advanced AI labs are still training ever larger models, but that training has somewhat receded to the background of the AI conversation. In 2026, inference—the use of trained models to produce code, write essays, or make images of ourselves as elves—has come to the forefront.
“It’s like training is yesterday’s news,” says Matt Kimball, principal data-center analyst at Moor Insights & Strategy. “All that any chief information officer wants to talk about is inference.” Nvidia CEO Jensen Huang, speaking at the company’s GTC 2026 conference, touted this change as the “inflection point of inference.”
Part of what’s caused the shift is very simple: LLMs are becoming useful, so people are using them. On top of that, many models on the market today are reasoning models. In response to a user’s query, they run inference not just once but multiple times, reprompting themselves in a process called chain of thought. Reasoning models generate longer outputs, and models with high reasoning effort can produce up to 20 times as much text as those with low or no effort. Adding even more to the world’s inference workload, the rise of agentic AI has resulted in inference running not just as a real-time response to a user’s query but also around the clock, working autonomously toward a user-defined goal.
Amazon’s Trainium chip was originally designed for AI training. However, Amazon Web Services chose to break up AI inference into two parts, with Trainium running the more computationally complex portion and Cerebras’s wafer-scale engine taking on the more memory-intensive portion. Amazon
The resulting explosion in inference demand has led to unexpected alliances among tech giants. OpenAI and Amazon have deployed chips the size of a dinner plate designed by Cerebras, despite Amazon having its own Trainium chips. Nvidia bought key talent and intellectual property from AI-inference startup Groq in a controversial deal worth US $20 billion. And Anthropic is paying LLM competitor SpaceXAI over a billion dollars per month to lease spare compute.
Although they might seem similar, AI training and AI inference are computationally different. These big moves from tech giants signal that in order to support the inference demand, we’re going to need a very different mix of hardware than experts may have expected even a couple of years ago.
How does AI inference differ from AI training?
An untrained LLM is like a jumble of Scrabble tiles on a table. Instead of single letters, though, the tiles show fragments of words, called tokens. Everything you’d need to write almost anything is present, but nothing makes sense.
Training a model organizes this jumble using a guessing game played at scale. The model is shown real text with the next token hidden and asked to predict what comes next. After each guess, the correct token is revealed and then compared to the prediction, and the difference is used to calculate the model’s accuracy. The game is played not with a single sentence but over billions of passages.
... continue reading