GoKawiilDeveloper builds B200 attention CUDA kernel reaching 94.4% of FlashAttention-4 speed
A technical blog post walks through building a custom attention kernel for Nvidia's Blackwell B200 GPU entirely from scratch, using CUDA and some PTX code. Through a 14-step progression of optimizations illustrated with 60 diagrams, the kernel reaches 94.4% of FlashAttention-4's performance on 4K, 8K, and 16K sequence shapes, and is demonstrated by plugging it into a video-generation model.