GoKawiilDeveloper builds gpu-rebar-layout tool to enable 32 GB Resizable BAR on multi-GPU Linux rig
A hobbyist detailed how they got peer-to-peer memory access working across four AMD Radeon AI PRO R9700 cards and two RTX 3090s connected through a Broadcom PEX88096 PCIe switch on Linux. They released a kernel module called gpu-rebar-layout on GitHub that manually resizes each card's BAR to 32 GB after boot, working around a BIOS limitation and a Linux kernel bug that prevents the system from allocating enough address space for six large BARs at once.
GoKawiil's interpretation of the reporting above, not reported fact.
The write-up illustrates a practical barrier facing hobbyists and small labs building multi-GPU AI rigs from consumer and prosumer cards: without full 32 GB Resizable BAR mapping, AMD's ROCm won't enable direct GPU-to-GPU data transfer, limiting performance in workloads like local AI training or inference. The author notes the fix relies on undocumented kernel internals and a yet-unmerged mailing list patch, suggesting this remains a fragile, DIY workaround rather than an officially supported configuration.
- A custom Linux kernel module, gpu-rebar-layout, forces 32 GB Resizable BAR sizing across six GPUs behind a PCIe switch.
- AMD ROCm requires each card's full 32 GB to be BAR-mapped before it will enable peer-to-peer transfers.
- The approach depends on unstable kernel internals and an unmerged patch, making it experimental and version-sensitive.
AMD Radeon AI PRO R9700 GPU — This article is literally built around squeezing maximum performance out of multiple R9700s behind a PCIe switch, making it a natural centerpiece for anyone tackling multi-GPU ROCm workloads. If you're experimenting with Resizable BAR and peer-to-peer transfers like this build, having one of these cards on hand lets you follow along and test configurations yourself.
See AMD Radeon AI PRO R9700 GPU on Amazon → Affiliate link — we may earn a commission on purchases, at no extra cost to you. Product picked by AI based on this article; it is not a tested recommendation.Source: doug.sh — Doug Calobrisi, 2026-11-12
Published there as: “Fitting Four 32 GB Resizable Bars Behind a PCIe Switch on Linux”
Read the original report → The summary and analysis above are GoKawiil's own, written from reporting by the source above. Facts and quotes belong to the original publisher.