Attention Is All You Need: the big picture

Why recurrence had to go

By 2016 the best translation systems were encoder–decoder RNNs with attention (6.6 Seq2seq, encoder–decoder and beam search, 6.7 Attention is born). They worked, but two costs were baked into the recurrence $\mathbf{h}_t = f(\mathbf{h}_{t-1}, \mathbf{x}_t)$. First, sequential computation: position $t$ cannot be processed before position $t-1$, so a sentence of $n$ tokens needs $n$ dependent steps while GPUs, which love parallel work, sit idle. Second, path length: information from token 1 reaches token $n$ only after passing through $n$ transformations, each an opportunity to forget or to shrink a gradient. LSTM gates (6.5 LSTM and GRU) soften the second problem; nothing fixes the first.

Attention had already shown that letting the decoder look directly at every encoder state removes the bottleneck of a single summary vector. The 2017 paper by Vaswani et al. asked the radical question: what if attention is the only mechanism that mixes information across positions?

Intuition. An RNN passes a note down a line of people, one at a time. Self-attention puts everyone in a room: each person looks at everyone else and decides whom to listen to, all at once.

The cost table that justified the bet

Table 1 of the paper compares one layer of each design for a sequence of $n$ tokens with representation size $d$:

Self-attention pays $n^2$ in compute but wins everywhere else: no sequential dependency, and every pair of positions is one step apart. When $n < d$ (typical sentences, $d = 512$) it is even cheaper than recurrence. The $n^2$ term only became painful years later with long contexts — the whole story of 7.10 Efficient inference: KV cache, GQA, FlashAttention, speculative decoding, quantization and 7.12 Beyond vanilla attention: state space models, long context and hybrids.

A tour of the architecture

The original model is an encoder–decoder for translation. Read the diagram bottom to top.

1. Embeddings + positions. Each token id becomes a learned vector of size $d_{\text{model}} = 512$, scaled by $\sqrt{d_{\text{model}}}$, and a positional encoding is added (7.3 Positional information: sinusoidal, learned, RoPE, ALiBi) because attention alone cannot tell order.
2. Encoder stack. $N = 6$ identical blocks. Each block: multi-head self-attention (7.2 Scaled dot-product and multi-head attention, fully derived) → add the residual and apply LayerNorm → position-wise feed-forward network (two linear layers with a ReLU, inner size $d_{ff} = 2048$) → add & norm (7.4 Inside the block: residuals, LayerNorm, FFN and embeddings). The encoder reads the whole source sentence bidirectionally.
3. Decoder stack. Also 6 blocks, each with three sub-layers: masked self-attention over the target tokens generated so far (a position may not look at later positions), cross-attention whose queries come from the decoder and whose keys and values come from the encoder output, and the feed-forward network.
4. Output. A linear layer maps each decoder position to vocabulary logits, and a softmax gives $p(\text{next token})$. The output projection shares its weights with the input embeddings.

The base model has about 65M parameters with $h = 8$ heads of size $d_k = d_v = 64$; the "big" model doubles $d_{\text{model}}$ to 1024 with 16 heads (213M parameters).

How it was trained and run

Training is teacher forcing (6.6 Seq2seq, encoder–decoder and beam search): feed the whole target sentence shifted right by one token, apply the causal mask, and compute cross-entropy on all positions in parallel — one forward pass yields $n$ predictions. Details that became standard: Adam with $\beta_2 = 0.98$, a learning rate that warms up linearly for 4,000 steps and then decays like $\text{step}^{-0.5}$ (the formula is $\eta = d_{\text{model}}^{-0.5}\min(\text{step}^{-0.5},\ \text{step}\cdot 4000^{-1.5})$), dropout 0.1, label smoothing 0.1, and byte-pair-encoded vocabularies of roughly 32–37k tokens (6.1 Text to numbers: tokenization). The base model trained for 12 hours on 8 P100 GPUs and reached 27.3 BLEU on English→German; the big model took 3.5 days and set new records (28.4 BLEU En→De, 41.8 En→Fr) at a fraction of the compute of previous systems.

At inference the decoder is autoregressive: start from a begin token, run the whole stack, pick a next token (beam size 4 with a length penalty in the paper), append it, and repeat. The encoder runs once; the decoder runs once per output token — the asymmetry that later motivates the KV cache (7.10 Efficient inference: KV cache, GQA, FlashAttention, speculative decoding, quantization).

What changed between 2017 and 2026

The paper made three contributions that survived: self-attention as the mixing operation, multi-head attention, and the idea that positions must be injected explicitly. Almost everything else was revised:

• Decoder-only won for general-purpose models. GPT dropped the encoder entirely (7.6 Decoder-only: GPT and autoregressive generation); BERT kept only the encoder (7.5 Encoder-only: BERT and friends (and sentence embeddings)); T5-style encoder–decoders remain for translation, speech and some summarization (7.7 Encoder–decoder: T5, BART and choosing an architecture).
• Normalization moved before each sub-layer (pre-norm), making deep stacks train without fragile warmup (7.4 Inside the block: residuals, LayerNorm, FFN and embeddings).
• Components were swapped: sinusoidal positions → rotary embeddings, ReLU → GELU/SwiGLU, LayerNorm → RMSNorm, dense FFNs → mixture-of-experts (7.11 Mixture of Experts), and dropout mostly disappeared at scale.
• Scale became the method. The same block, stacked 96 times with $d = 12{,}288$, is GPT-3 (7.9 Scaling laws and compute-optimal training).

The rest of this phase follows that arc: the attention math (7.2), positions (7.3), the block (7.4), the three architectural families (7.5–7.7), building one yourself (7.8), scaling (7.9), inference efficiency (7.10), MoE (7.11) and what lies beyond attention (7.12).

Where this shows up later

• The attention equation is derived and computed by hand in 7.2 Scaled dot-product and multi-head attention, fully derived; positional encodings in 7.3 Positional information: sinusoidal, learned, RoPE, ALiBi; the block and its parameter count in 7.4 Inside the block: residuals, LayerNorm, FFN and embeddings.
• The encoder/decoder split becomes the taxonomy of modern models: 7.5 Encoder-only: BERT and friends (and sentence embeddings), 7.6 Decoder-only: GPT and autoregressive generation, 7.7 Encoder–decoder: T5, BART and choosing an architecture.
• Teacher forcing and autoregressive decoding reappear in sampling (8.5 Decoding and sampling) and inference optimization (7.10 Efficient inference: KV cache, GQA, FlashAttention, speculative decoding, quantization).
• Vision Transformers apply exactly this encoder to image patches (5.7 Vision Transformers); Whisper is this encoder–decoder applied to audio (8.10 Multimodal models).