Softmax - GoKawiil

Explainer details math behind the Softmax function and its derivative

A technical post explains the Softmax function, which converts an N-dimensional vector of real numbers into another N-dimensional vector of values between 0 and 1 that sum to 1. Using worked numeric examples, it shows how Softmax acts as a smoothed version of the maximum function, giving the largest input a disproportionately larger share of the output while still assigning smaller shares to other elements.

eli.thegreenplace.net · 2026-10-01

New handbook breaks down how FlashAttention speeds up Transformer attention without approximation

A newly published technical handbook explains FlashAttention, the exact attention algorithm that speeds up Transformer models by minimizing memory traffic rather than reducing computation. It walks through the standard attention implementation's repeated reads and writes to GPU high-bandwidth memory, then details how tiling, online softmax, and recomputation eliminate most of that overhead across FlashAttention-1 through FlashAttention-4.

chizkidd.github.io · 2026-09-14