GoKawiilTechnical essay models transformer layers as memory stores with measurable recall capacity
A technical blog post argues that linear layers, MLPs, and self-attention can each be understood as a type of memory store that holds key-value pairs and retrieves them later. The author introduces an 'interference-to-signal ratio' to measure how well each layer type recalls stored information as the number of memories grows, estimating capacities of roughly 64 memories for linear layers, 2,700 for MLPs, and over a million for self-attention with 64-dimensional keys.