Attention Wasn't All We Needed

Attention Wasn't All We Needed There's a lot of modern techniques that have been developed since the original Attention Is All You Need paper. Let's look at some of the most important ones that have been developed over the years and try to implement the basic ideas as succinctly as possible. We'll use the Pytorch framework for most of the examples. Note that most of these examples are highly simplified sketches of the core ideas, if you want the full implementation please read the original paper or the production code in frameworks like PyTorch or Jax. Group Query Attention Ok starting off in no particular order, Grouped Query Attention is a technique to reduce the memory usage of the KV cache during inference. Group Query Attention is an architectural optimization for the standard multi-head attention mechanism. The core idea behind GQA is based on the observation that the computational bottleneck and memory footprint in MHA are heavily influenced by the size of the K and V project ... Read full article.