Self Attention - GoKawiil

Technical essay models transformer layers as memory stores with measurable recall capacity

A technical blog post argues that linear layers, MLPs, and self-attention can each be understood as a type of memory store that holds key-value pairs and retrieves them later. The author introduces an 'interference-to-signal ratio' to measure how well each layer type recalls stored information as the number of memories grows, estimating capacities of roughly 64 memories for linear layers, 2,700 for MLPs, and over a million for self-attention with 64-dimensional keys.

alexwang.ai · 2026-10-16

Interactive Explainer Breaks Down GPT-2's Transformer Architecture Visually

A visual explainer tool called Transformer Explainer illustrates how Transformer-based neural networks work, using the 124-million-parameter GPT-2 (small) model as its example. It walks through core components like tokenization, embeddings, attention mechanisms, and Transformer blocks to show how these models predict the next word in a sequence.

poloclub.github.io · 2026-09-21