GoKawiilBuild and understand one of the most complex GPU kernels on the latest hardware
Download the video You’ll generate these videos with B200 attention kernel built below Generated by video model using the Blackwell B200 attention kernel implemented in this post. See Capstone Project below.
Percent of same-run Stock FA4 across 14-kernel progression and minor optimizations covered at the end. Each implementation is timed on its preferred contiguous layout; see Benchmark Calibration.
What this blog is about
In this blog, we build a dense B200 attention kernel from scratch in CUDA and a little PTX, from baseline to 94.4% of FlashAttention-4 performance on 4K, 8K, and 16K shapes used in the FA4 paper.
The main contribution is the visual guide: a beginner-friendly progression built around 60 diagrams. We first build an intuitive understanding of how the naive kernel works, then add one optimization at a time, with detailed diagrams, concise explanations, and code.
For the capstone project, we plug the final kernel into a video-generation model.
The focus here is not squeezing every last percent of performance, that’ll be the focus of my next blog.
It’s one of the hardest kernels out there, running on the latest hardware, so it’ll be fun.
Research: This is meant to give you a foundation for doing your own GPU kernel research on the latest hardware. We focus on B200 attention, but many of the concepts and mental models apply beyond this kernel. By the end, you’ll understand this kernel and be able to come up with your own optimization ideas.
... continue reading