GoKawiilDeveloper builds B200 attention CUDA kernel reaching 94.4% of FlashAttention-4 speed
A technical blog post walks through building a custom attention kernel for Nvidia's Blackwell B200 GPU entirely from scratch, using CUDA and some PTX code. Through a 14-step progression of optimizations illustrated with 60 diagrams, the kernel reaches 94.4% of FlashAttention-4's performance on 4K, 8K, and 16K sequence shapes, and is demonstrated by plugging it into a video-generation model.
GoKawiil's interpretation of the reporting above, not reported fact.
Attention kernels are the computational bottleneck in modern AI models, and squeezing performance out of new hardware like Blackwell requires deep, low-level GPU expertise that few engineers possess. By documenting the process step-by-step, this work lowers the barrier for other developers to understand and optimize kernels on cutting-edge chips, which matters as AI compute costs and hardware complexity keep rising.
- The kernel achieves 94.4% of FlashAttention-4 performance on B200 GPUs across multiple sequence lengths.
- The build process is documented visually through 60 diagrams and 14 incremental optimization steps.
- A capstone demo integrates the finished kernel into a video-generation model to show real-world applicability.
Programming Massively Parallel Processors, 4th Edition — If this deep dive into hand-writing a Blackwell B200 attention kernel in CUDA and PTX hooked you, this is the classic text for learning GPU programming properly. It walks through memory hierarchies, tiling, and kernel optimization patterns that underpin exactly the step-by-step progression the article demonstrates.
See Programming Massively Parallel Processors, 4th Edition on Amazon → Affiliate link — we may earn a commission on purchases, at no extra cost to you. Product picked by AI based on this article; it is not a tested recommendation.Source: iaroslavelistratov.github.io — Iaroslav Elistratov, 2026-09-02
Published there as: “B200 Attention Kernel from Scratch to Near-SOTA in 60 Diagrams”
Read the original report → The summary and analysis above are GoKawiil's own, written from reporting by the source above. Facts and quotes belong to the original publisher.