A newly published technical handbook explains FlashAttention, the exact attention algorithm that speeds up Transformer models by minimizing memory traffic rather than reducing computation. It walks through the standard attention implementation's repeated reads and writes to GPU high-bandwidth memory, then details how tiling, online softmax, and recomputation eliminate most of that overhead across FlashAttention-1 through FlashAttention-4.
chizkidd.github.io
· 2026-09-14
Shenzhen Suqiao Intelligent Technology, an established Chinese OEM/ODM, is listing a customized GeForce RTX 5090 with 96GB of memory—triple the stock 32GB—for $3,888 on Alibaba, undercutting the standard U.S. price by roughly 35%. The listing contains dubious specs like GDDR6X memory and 14 Gbps speeds, raising questions about accuracy, though the underlying GB202 chip used in Nvidia's 96GB RTX Pro 6000 Blackwell suggests such a memory configuration is technically feasible.
tomshardware.com
· 2026-09-11
System76 has introduced Thelio Mira AI, a Linux-based desktop workstation aimed at local AI development that supports GPU configurations offering up to 192 GB of memory. The machine is designed to let developers train, fine-tune, and iterate on demanding AI models without relying on cloud infrastructure, and it ships with standard System76 hardware warranty options of one, two, or three years.
system76.com
· 2026-09-10
A technical deep-dive follows the STG.E instruction—the global memory store operation in a simple vector-add CUDA kernel—through an RTX 4090's pipeline, from register read through the load/store unit, coalescer, L1 cache, crossbar, and L2 slice, down to DRAM. The author benchmarks how frequently a single warp can issue store instructions, finding a new STG.E can be pushed through the register file, LSU, and coalescer roughly every 6.1 cycles (about 2.3 nanoseconds at 2.6 GHz).
blog.doubleword.ai
· 2026-09-08
GoKawiil