GoKawiilOur local AI testing in 2026 has focused on large language models that can fit entirely into the 128GB of unified memory on Nvidia GB10 and AMD Strix Halo systems. Useful as those smaller open models can be, there is sometimes no replacement for displacement. Today, we’re exploring what’s possible from a local AI cluster with a pair of Nvidia GB10 systems, namely Dell’s Pro Max with GB10 (henceforth Pro Max), which gives us 256GB of RAM for a local AI sandbox.
Quantizing an AI model from higher-precision to lower-precision data types involves tradeoffs for performance and accuracy. And even in quantized form, some advanced open models are still too large to fit within 128GB. But those models can be distributed across multiple local systems using the network as a scale-out backbone, just as they are in the data center.
Why scale out GB10 systems (or Strix Halos, or Macs)? Local token factories with large VRAM pools built up from discrete GPUs can get crazy, fast. Scaling one to even 128GB of VRAM requires a costly host system with enough PCI Express slots and bandwidth to feed those cards, and going beyond 128GB means spending $20K or more in Nvidia GPUs at a minimum, even if you're building up from older 48GB Ada cards.
Latest Videos From Watch full video here:
The preferred recipe for this kind of setup typically includes a Threadripper Pro or Epyc platform, which means a costly CPU, motherboard, and DDR5 kit even before you start adding graphics cards. The power requirements for such a system can quickly get beyond the capabilities of a standard USA 15A circuit (1,800W maximum).
And having four discrete GPUs running their blower fans at high speeds under load, along with whatever other active cooling you might need for what is essentially a GPU server, is not going to make for the most pleasant company if you’re sharing a space with it.
While a GPU server with four RTX Pro 5000 or RTX Pro 6000 cards is useful for getting the absolute best performance for a given application, those potentially high costs, platform challenges, and quality of life concerns have led local AI enthusiasts to explore other ways of achieving large local memory pools with acceptable LLM inference performance, like the GB10 cluster we’re building today.
(Image credit: Tom's Hardware)
Nvidia made the DGX Spark and its Spark-alikes scalable, cluster-able systems right out of the box thanks to their built-in ConnectX 7 200Gbps NICs. These high-end interfaces support Remote Direct Memory Access over Converged Ethernet, or RoCE, so two (or more) GB10 boxes can use them as the backbone for a distributed AI computing cluster.
Stay On the Cutting Edge: Get the Tom's Hardware Newsletter Get Tom's Hardware's best news and in-depth reviews, straight to your inbox. Contact me with news and offers from other Future brands Receive email from us on behalf of our trusted partners or sponsors
... continue reading