Attention is born

Look back, softly

A human translator does not memorize the whole sentence and then write; they glance back at the relevant source words while producing each target word. Bahdanau, Cho and Bengio (2015) gave the decoder the same ability: at every output step it computes a weighted average of all encoder states, with weights that depend on what it is currently trying to produce. The weights are computed by a small network and trained end-to-end with the rest of the model — no alignment supervision, just backprop (4.4 Backpropagation, fully derived).

Bahdanau (additive) attention

Let the encoder (bidirectional, so each state sees the whole source) produce states $\mathbf{h}_1,\dots,\mathbf{h}_S$, and let the decoder's previous state be $\mathbf{s}_{t-1}$. For output step $t$:

The score function is a one-hidden-layer network, hence "additive": it adds projections of the decoder and encoder states before the $\tanh$. The context vector is now different at every step — a fresh summary of the source focused on what matters right now — so no single vector has to carry everything. Plotting $\alpha_{tj}$ as a matrix of target × source positions gives the famous alignment heatmaps: a near-diagonal for English→French with visible swaps where word order differs ("European Economic Area" ↔ "zone économique européenne"). Attention learned alignments as a by-product of translation.

Luong (multiplicative) attention

Luong, Pham and Manning (2015) simplified the scoring and compared variants using the current decoder state $\mathbf{s}_t$:

The dot-product score is a single matrix multiplication for all positions at once, so it is far cheaper on GPUs, and in practice it works as well as the additive form once dimensions are matched. Luong also distinguished global attention (over all source positions) from local attention (a window around a predicted position) and introduced input feeding (passing the previous attention output into the next decoder step). The transformer chose the dot product and fixed its one weakness — the growth of score variance with dimension — by dividing by $\sqrt{d_k}$ (7.2 Scaled dot-product and multi-head attention, fully derived).

Attention as a soft dictionary

Strip away the RNNs and a pattern remains. A query $\mathbf{q}$ (what I am looking for: the decoder state) is compared with a set of keys $\mathbf{k}_j$ (what each item is about: the encoder states); the similarities become weights; the weights blend the corresponding values $\mathbf{v}_j$ (what each item contains: also the encoder states, in seq2seq):

A hard dictionary returns the single value whose key matches exactly; attention returns a soft, differentiable mixture weighted by how well each key matches. That differentiability is the whole trick: it lets gradient descent learn what to look up.

Worked example. Three source positions with dot-product scores $(2,\,0,\,1)$ for the current query. Softmax: $e^2=7.389$, $e^0=1$, $e^1=2.718$, sum $11.107$, so $\alpha=(0.665,\,0.090,\,0.245)$. With values $\mathbf{v}_1=(1,0)$, $\mathbf{v}_2=(0,1)$, $\mathbf{v}_3=(1,1)$ the context is $0.665(1,0)+0.090(0,1)+0.245(1,1)=(0.910,\,0.335)$ — mostly the first item, a little of the third.

Math check (why scaling will matter). If the key dimension is $d_k=2$ and we divide the scores by $\sqrt{d_k}=1.414$ before the softmax, the scores become $(1.414,0,0.707)$ and the weights $(0.576,0.140,0.284)$ — flatter. Larger raw scores make the softmax sharper and its gradients smaller; the transformer's $\sqrt{d_k}$ keeps the softmax in its well-behaved range as dimensions grow (7.2 Scaled dot-product and multi-head attention, fully derived).

Three kinds of attention follow from the same formula by choosing where queries and keys come from: cross-attention (queries from the decoder, keys/values from the encoder — Bahdanau's case), self-attention (queries, keys and values all from the same sequence, so each position gathers information from the others — proposed for sentence encoding by Cheng et al. and Parikh et al. in 2016) and causal self-attention (self-attention restricted to earlier positions, for generation — 7.6 Decoder-only: GPT and autoregressive generation).

What attention fixed — and what it did not

Fixed: the bottleneck (quality no longer collapses with sentence length), a few BLEU points, and interpretability through alignment plots. Not fixed: the encoder and decoder were still RNNs, so training remained sequential in time (no parallelism across positions) and information between two distant words still travelled through $O(S)$ recurrent steps, with the gradient problems of 6.4 Recurrent neural networks and BPTT along the way. The 2017 question was: if attention is doing the work of connecting positions, do we need the recurrence at all? The answer — replace every RNN with self-attention, add positional encodings, stack the blocks — is 7.1 Attention Is All You Need: the big picture.

Where this shows up later

• Scaled dot-product and multi-head attention — 7.2 Scaled dot-product and multi-head attention, fully derived; positional encodings restore the order information RNNs gave for free — 7.3 Positional information: sinusoidal, learned, RoPE, ALiBi; cross-attention in encoder–decoder transformers — 7.7 Encoder–decoder: T5, BART and choosing an architecture.
• Cross-attention conditions image generators on text — 8.10 Multimodal models; attention weights as (unreliable) explanations and as circuits — 8.11 Interpretability: what is happening inside.
• Retrieval is attention over a database: a query embedding scored against stored keys, returning the top values — 9.2 Embeddings deep dive, 9.3 Vector search and indexes; agents' memory retrieval does the same — 10.4 Memory and context engineering.