unifies linear/MLP/attention as associative memory via interference-to-signal ratio
You could have invented transformers from memory (external)
If you build or debug transformers, this gives you a back-of-envelope formula for how many distinct facts each layer type can recall before interference drowns the signal: ~64 for a linear layer, ~2,700 for a squared-ReLU MLP, ~1.2 million for softmax attention at 64-dim keys.
- The unifying metric is the interference-to-signal ratio: query a store with a stored key k_j, recover v_j (signal) plus a weighted sum of all other values (interference); capacity is the number of memories M at which that ratio hits 1.
- Outer-product linear store W = VᵀK has interference-to-signal ~sqrt(M/d_k), giving capacity M ≈ d_k (64 at d_k=64). The optimal least-squares store whitens keys via (KᵀK)⁻¹, cancels d_k memories' worth of interference (ratio sqrt(M/d_k − 1)), and doubles capacity to 2d_k.
- A two-layer MLP Vᵀσ(Kx) is derived as a nonlinear store: rows of W1 are keys, columns of W2 are values. ReLU^p shrinks each interference term to d_k^(−p/2), pushing capacity to ~d_k^p; squared ReLU gives ~2d_k²/3 ≈ 2,700, cubed ReLU ~35,000. Squared ReLU equals a GLU with shared gate/input weights; SwiGLU nearly matches it.
Full summary
The post reframes the standard transformer layers (linear maps, MLPs, self-attention) as associative memory stores and estimates each one's recall capacity using a single quantity, the interference-to-signal ratio: when you query with a stored key, you get the target value plus interference from every other stored memory, and capacity is the number of memories M at which interference equals signal. An outer-product linear store holds about d_k memories; whitening the keys (the optimal least-squares store) doubles that to 2d_k; passing similarities through a nonlinearity ReLU^p suppresses low-similarity interference and raises capacity to roughly d_k^p, so a squared-ReLU MLP reaches ~2,700 and a cubed-ReLU ~35,000 at d_k=64. Softmax attention is the exponential limit of this power trick, with capacity ~e^(2β(1−β/d_k)), about 1.2 million at β=sqrt(d_k)=8, and because its keys and values are computed from context tokens it is non-parametric and grows with context length. The author connects each store to real architectures (GLU/SwiGLU MLPs, Vaswani et al.'s deleted v2/v3 note describing feedforward layers as attention) and to long-context logit-scaling schemes (Su, Scalable-Softmax, LogN-Scaling, YaRN) that raise β with log M. The author explicitly notes the estimate counts only random-vector recall in one layer and ignores the training data, and points to complementary bits-per-parameter measurements (Morris et al. ~3.6, Allen-Zhu & Li ~2).