GoKawiilAt the 2018 International Symposium on Computer Architecture, John Hennessy and David Patterson delivered their Turing Lecture: "A New Golden Age for Computer Architecture".
In the 1980s, when Hennessy and Patterson did their Turing Award-winning research,
single-threaded CPU performance grew 52% a year. By 2018, with the end of Moore's Law and Dennard Scaling, the rate was 3%.
There was a need for domain-specific architectures (DSAs). Their worked example was Google's TPU v1, already in production: 29× the throughput of a CPU on neural-network inference, at 80× better energy efficiency. The closing prediction: "the next decade will see a Cambrian explosion of novel computer architectures."
This prediction came true. Today, we now have dozens of architectures in serious development. GPUs, TPUs, LPUs, NPUs, DPUs, ASICs, wafer-scale engines, reconfigurable dataflow, neuromorphic, photonic, analog. Particularly, these architectures focus on compute for AI.
The architectures that have won real deployment so far: GPUs (NVIDIA, AMD), systolic-array accelerators (TPU, Trainium), the Cerebras Wafer-Scale Engine, and the Groq LPU.
NVIDIA is the clear frontrunner; AMD follows, with 6 GW commitments from both OpenAI and Meta. TPUs train Gemini and will serve Anthropic with up to a million chips; Anthropic also runs Claude on over a million Trainium chips. Cerebras now serves OpenAI inference; the Groq LPU was folded into NVIDIA via a $20B acquihire.
This post aims to survey these varying approaches - their philosophy, architecture, scaling methods (scale-up and scale-out), and software stack (how you program the chip).
The Problem
AI compute is dominated by matrix multiplication. A transformer is a sequence of matmuls: Q/K/V projection, attention, output projection, FFN - interleaved with element-wise ops: normalisation, activation, residual adds. Training a frontier model performs 10 25 10^{25} 1025 multiply-accumulate operations (matmuls are a sequence of multiply-accumulates).
... continue reading