Processing in Memory: DRAM Is About to Do Math

At Hot Chips 2026, Samsung presented a 16 GB LPDDR5X memory package that delivers 614 GB/s of internal bandwidth to its own compute units.

For scale, 614 GB/s matches the memory bandwidth of a top-spec Apple M5 Max across its entire unified memory system: the 40-core GPU configuration in the $3,499 MacBook Pro. Samsung claims that number from inside one memory package, against the 76.8 GB/s that escapes through its external pins. Tom's Hardware reported the resulting eightfold difference.

That eight-to-one ratio makes the case for Processing in Memory. DRAM banks already provide most of the bandwidth; the external pins cannot expose it.

The terms#

Bank. DRAM contains banks, independent arrays that can operate in parallel. A modern LPDDR5X die has dozens. Together they deliver enormous internal throughput, but they share a narrow external interface.

GEMV vs. GEMM. Matrix-vector multiply versus matrix-matrix multiply. Autoregressive decoding at batch size 1 uses GEMV: one token's activations multiplied against the entire weight matrix. Prefill and batched serving use GEMM. PIM helps most with GEMV.

PIM. Processing-in-memory hardware places compute units next to the banks, inside the memory die, where they can use bank-level bandwidth instead of interface bandwidth.

Arithmetic intensity. FLOPs performed per byte loaded. High intensity makes a workload compute-bound. Low intensity makes it bandwidth-bound, leaving matrix engines idle.

The bottleneck is memory bandwidth#

Generating one token requires reading every parameter from DRAM, multiplying it once, and discarding it. Each token depends on the preceding token, so batch-1 decoding cannot reuse the weights across tokens. Arithmetic intensity sits near the floor.

... continue reading