GoKawiilAI hardware has been a pretty simple game for years: Whose GPU has the most FLOPS, the most memory, or the fastest clock? But as hyperscalers want more performance from the same size of facilities, who controls the fabric between the chips is becoming another question frequently asked – in large part because it can dictate whether AI hardware follows the principles of Moore’s law that have held for decades, or breaks out into a new paradigm.
“Moore's Law is deader than a doornail, absolutely just toast,” said Nick Harris, chief executive of photonics firm Lightmatter, in an interview with Tom's Hardware Premium. The way to make a processor faster used to be to pack in more silicon, but there is a physical limit to how big a single chip can be – so performance now depends on networking dozens, then hundreds, of chips together. “Ultimately, networking is the future of computing,” Harris said. “The performance of these AI systems is completely hamstrung and bottlenecked by your ability to achieve low latency and very, very high bandwidth.”
It all comes down to scale-up and scale-out. Scale-up is the ultra-low-latency interconnect inside a single server, rack, or pod that makes many accelerators behave like one giant machine. Scale-out is the wider network, combining those machines into clusters. There is a third factor, called scale-across, that links whole data centers together. As Polina Bayvel, professor of optical communications and networks at University College London, told us, the hyperscalers are "on a drug which says that the more you put in, the better is the answer" – and between 72 and 144 GPUs now go into a single scale-up rack before it needs to reach outward.
Latest Videos From Watch full video here:
Copper was sufficient within racks for years, but it has hit a wall, and a supply shortage. “Your arm reach, my arm reach, is about two metres – the copper cable is a little shorter than that,” Harris explained, “and we’ve already hit the limit.” Thus, optical interconnects are becoming more important, and moving from rack-to-rack cabling toward the accelerator package itself. Earlier this month, Elon Musk made moves to purchase Mesh Optical, as part of TeraFab and SpaceX's ongoing AI ambitions.
Moving light towards the chip
An example of Nvidia's Spectrum-X co-packaged optics. (Image credit: Nvidia)
Today, optical signals usually arrive through pluggable transceivers, the standard modules that sit around 18 inches from the switch. The next step is near-package optics, which brings that down to roughly six inches. After that comes co-packaged optics, or CPO, where optical chiplets sit around the GPU or switch. The most ambitious version puts the optics on an interposer beneath the chip itself. “As you progress, you increase the speed and you reduce the energy consumption with each one of those leaps,” said Harris.
For now, the industry’s attention is on the step before the really radical one. “I think 2027 and 2028 are going to be very, very big years for near-package optics,” Harris said. He describes it as the test bed: the last stop before the optical transceiver effectively ships as part of the accelerator or switch, rather than as a separate component bolted on nearby.
Lightmatter is already building CPO parts at TSMC and GlobalFoundries, which it expects to ship in 2028. At the same time, it is working with customers on the more integrated interposer approach – the one for companies that, in Harris’s words, “realise they’d like to have an advantage in the market.”
... continue reading