From Selective Filter to All-to-All Weights

How Transformers Replaced Sequential Memory With Scaled Dot-Product Attention

For half a billion years, every biological nervous system on Earth evolved under the same unyielding physical constraint. Because wiring an eyeball or a cerebral cortex with all-to-all high-resolution connections would consume more metabolic energy and skull volume than an organism could survive, biological attention had to act as a ruthless subtractive filter. By aiming a two-degree fovea, damping rival signals in the thalamus, and funneling sensory traffic through a fifty-bit serial bottleneck, the living brain survives by throwing away ninety-nine point nine nine nine percent of incoming reality. When computer scientists in the late twentieth century began building artificial neural networks to translate languages and parse text, they unwittingly recreated a digital version of that sequential memory bottleneck.

Through the early 2010s, state-of-the-art sequence modeling relied on Recurrent Neural Networks and Long Short-Term Memory architectures that read a sentence strictly one word at a time from left to right. As each word entered the network, the model updated a single fixed-length hidden state vector and passed that compressed summary forward to the next time step. Whenever a paragraph stretched beyond thirty or forty words, early words at the beginning of the sentence were overwritten and diluted inside that fixed-size vector, just as human working memory loses the beginning of a spoken list once its four slots overflow.

In 2014, computer scientist Dzmitry Bahdanau, working with Kyunghyun Cho and Yoshua Bengio at the University of Montreal, broke that fixed-vector bottleneck by introducing a differentiable attention mechanism for neural machine translation. Instead of forcing the decoder to generate an entire translated sentence out of one static summary vector, Bahdanau allowed the network to look back across all the encoder's hidden states at every step and compute a learned alignment weight for each source word. That alignment dynamically highlighted whichever French words were most relevant to the specific English word being generated right then.

Three years later, in June 2017, Ashish Vaswani and seven colleagues at Google Brain and Google Research published Attention Is All You Need, asking a radical architectural question. If an attention matrix can link any two words across a sequence directly, why keep the slow, sequential recurrent backbone at all? Discarding recurrence completely, Vaswani's team built the Transformer around scaled dot-product self-attention, an operation that processes every token in a passage simultaneously in parallel. Each input token is projected through learned weight matrices into three vectors of dimension $d_k$: a Query vector $Q$ representing what the token is looking for, a Key vector $K$ representing the label the token advertises to others, and a Value vector $V$ holding the content the token transmits when selected.

In a single parallel matrix multiplication computed as $\text{Attention}(Q, K, V) = \text{softmax}(QK^T / \sqrt{d_k})V$, every token takes the dot product of its Query against the Keys of every other token in the context window. It then scales the products by $\sqrt{d_k}$ so large dimensions do not push the softmax function into vanishing-gradient saturation, and normalizes the scores into probability weights that blend the Value vectors together. By splitting this calculation across multiple parallel attention heads, one head can link a pronoun directly to its noun five hundred words earlier while another head tracks verb tense or syntactic clauses across the same passage. By discarding recurrent step-by-step memory altogether and computing scaled dot-product compatibility scores between every pair of tokens in parallel, the 2017 Transformer turned attention from a biological survival filter that throws away 99.999 percent of experience into a differentiable all-to-all routing matrix.