GoKawiilHot Chips 2026 tutorial details how High Bandwidth Flash could reshape ML memory
At Hot Chips 2026, researchers Anurag Agarwal and Radhakrishna Giduthuri presented a tutorial on High Bandwidth Flash (HBF), a package-integrated flash memory technology built like HBM but functioning more like an SSD. Since no HBF hardware exists yet, the talk relied on simulations and projections, examining how software—especially inference frameworks like vLLM—would need to be redesigned to exploit HBF's large capacity despite its coarse-grained, storage-like access pattern.
GoKawiil's interpretation of the reporting above, not reported fact.
HBF could dramatically expand on-package memory capacity for AI accelerators without the cost and power penalties of stacking more HBM, but only if software stacks are rearchitected to handle SSD-like access constraints such as data retention and write leveling. Ideas like storing MoE experts or KV cache in HBF suggest a path toward serving larger models cheaply, but the lack of fine-grained random access means it won't be a simple drop-in replacement for existing memory hierarchies.
- HBF mimics HBM's physical form but behaves internally like flash storage, requiring DMA-based access in large chunks.
- No HBF hardware exists yet; the Hot Chips talk was based on simulation and software projections for ML workloads.
- Potential uses include storing MoE expert weights and sparse-attention KV caches, but software must handle SSD-style management overhead.
Samsung 990 PRO NVMe SSD — HBF may still be simulations and projections, but the flash technology behind it is the same NAND powering today's fastest consumer drives. The Samsung 990 PRO is a PCIe 4.0 NVMe SSD built for heavy read/write workloads like local AI model storage, so you can load big datasets fast while waiting for on-package flash to become real.
See Samsung 990 PRO NVMe SSD on Amazon → Affiliate link — we may earn a commission on purchases, at no extra cost to you. Product picked by AI based on this article; it is not a tested recommendation.Source: chipsandcheese.com — Chester Lam, 2026-08-24
Published there as: “Hot Chips 2026: Applying High Bandwidth Flash (HBF)”
Read the original report → The summary and analysis above are GoKawiil's own, written from reporting by the source above. Facts and quotes belong to the original publisher.