Would you like to run 235B parameter LLMs at home, but your lowly $10,000 budget restricts you to “consumer GPUs that can barely handle 70B parameter models”? This was the situation developer David Noel Ng found himself in, until he stumbled across an “obviously fake” Nvidia Grace-Hopper platform being sold on Reddit, of all places. Ng took the gamble and, according to his blog post, it paid off royally. He’s managed, with a bit of tinkering and fixing up, to get an enterprise system that would usually cost ~ $80,000 for a tenth of that sum. The included 960GB of LPDDR5X memory, alone, is now worth more than he paid for the full system. Hilariously, he even lowballed the seller, noting the original listing on Reddit was for 10,000 EUR before offering just 7,000 EUR.
Why Ng made the offer on an ‘obviously fake’ listing
As we mentioned in the intro, the deal Ng found seemed a little too good to be true. However, he researched the seller, who seemed to be a legitimate server equipment reseller within two hours' driving distance, so he quickly made an offer to get first in line.
There were some underlying, but not insurmountable, issues with the Grace Hopper system as sold, which meant it wouldn’t be widely popular on a consumer marketplace. Specifically, it was “a Frankensystem converted from liquid-cooled to air-cooled” operation. It also looked a bit of a mess, wasn’t rackable, and ran using a 48V power supply.
On the other hand, even if this were just a collection of components, the offer seemed irresistible. The specs of the system, as sold, were as follows:
2x Nvidia Grace-Hopper Superchip
2x 72-core Nvidia Grace CPU
2x Nvidia Hopper H100 Tensor Core GPU
2x 480GB of LPDDR5X memory with error-correction code (ECC)
2x 96GB of HBM3 memory
... continue reading