Positional information: sinusoidal, learned, RoPE, ALiBi

Attention is blind to order

Permute the rows of $\mathbf{X}$ and self-attention permutes its output rows the same way — nothing else changes, because every score $\mathbf{q}_i\cdot\mathbf{k}_j$ is computed independently of where $i$ and $j$ sit. Formally, for any permutation matrix $\mathbf{P}$, $\operatorname{Attention}(\mathbf{P}\mathbf{X}) = \mathbf{P}\operatorname{Attention}(\mathbf{X})$: the layer is permutation-equivariant. A transformer without positional information treats "dog bites man" and "man bites dog" identically. So order has to be injected, and the design space is: where (added to the input, or inside the attention scores) and how (absolute positions, or relative offsets).

Sinusoidal encodings (the original)

The 2017 paper adds to each token embedding a fixed vector $\mathbf{p}_{pos}\in\mathbb{R}^d$ with

Pair up dimensions $(2i, 2i+1)$: each pair is a point on the unit circle rotating with angular frequency $\omega_i = 10000^{-2i/d}$. Low $i$ spins fast (wavelength $2\pi$), high $i$ spins slowly (wavelength $10000\cdot 2\pi$) — a binary-counter-like clock in continuous form, visible as stripes in the positional-encoding demo below.

The relative-shift property. Why sinusoids rather than, say, a single counter? Because a fixed offset $k$ acts as a rotation that does not depend on $pos$:

which is just the angle-addition formulas $\sin(a+b)=\sin a\cos b+\cos a\sin b$ and $\cos(a+b)=\cos a\cos b-\sin a\sin b$. So $\mathbf{p}_{pos+k} = \mathbf{R}_k\,\mathbf{p}_{pos}$ for a block-diagonal rotation $\mathbf{R}_k$ determined by $k$ alone: a linear layer can learn "attend three tokens back" without knowing the absolute position. The dot product $\mathbf{p}_{pos}\cdot\mathbf{p}_{pos+k} = \sum_i \cos(\omega_i k)$ depends only on $k$ and decays with distance.

Learned absolute positions

BERT and GPT-2 instead learn a table $\mathbf{P}\in\mathbb{R}^{n_{\max}\times d}$ (GPT-2: $1024\times 768$) and add row $pos$ to each token. Simple and as good as sinusoids in-distribution, but position 1025 has no row at all — the model cannot run past its training length. Both schemes also entangle "what" and "where" in the same vector, which attention then has to disentangle.

Relative positions inside attention

Shaw et al. (2018), Transformer-XL and T5 moved position out of the embeddings and into the attention scores: T5 adds a learned scalar bias $b_{\text{bucket}(i-j)}$ to $s_{ij}$, with offsets bucketed logarithmically so that far-apart pairs share parameters. This encodes relative order directly, which is what language mostly cares about.

Rotary position embeddings (RoPE)

RoPE (Su et al., 2021) is the scheme used by Llama, Qwen, Gemma and most open models. The goal: make the attention score depend on the relative offset while keeping the ordinary dot product. Split $\mathbf{q}$ and $\mathbf{k}$ into 2-D pairs and rotate pair $i$ at position $m$ by the angle $m\theta_i$:

Rotations compose by adding angles, so $\mathbf{R}(a)^\top\mathbf{R}(b) = \mathbf{R}(b-a)$ and

The score is a function of content and of $n-m$ only — relative position, achieved by multiplying rather than adding, and applied to $\mathbf{q}$ and $\mathbf{k}$ only (values are untouched). Because high-frequency pairs rotate quickly, attention between distant tokens is naturally damped. In code the rotation is two element-wise products with precomputed $\cos$ and $\sin$ tables:

import torch

def rope(x, base=10000.0):                      # x: (..., n, d) with d even
    n, d = x.shape[-2], x.shape[-1]
    theta = base ** (-torch.arange(0, d, 2) / d)          # (d/2,) frequencies
    ang = torch.arange(n)[:, None] * theta[None, :]        # (n, d/2) = position * theta
    cos, sin = ang.cos(), ang.sin()
    x1, x2 = x[..., 0::2], x[..., 1::2]
    return torch.stack([x1 * cos - x2 * sin, x1 * sin + x2 * cos], -1).flatten(-2)

Apply rope to queries and keys before the dot product and the attention function from 7.2 Scaled dot-product and multi-head attention, fully derived needs no other change.

Math check. With $d_k = 2$, one pair, $\theta = 1$: a query at position 3 and a key at position 1 give $\mathbf{q}^\top\mathbf{R}(-2)\mathbf{k}$; a query at position 10 and a key at position 8 give the same $\mathbf{q}^\top\mathbf{R}(-2)\mathbf{k}$. Same offset, same score — that is the whole point.

ALiBi: no embeddings at all

ALiBi (Press et al., 2021) adds a penalty proportional to distance directly to the scores, $s_{ij} \leftarrow s_{ij} - m_h\,(i-j)$ for $j\le i$, with a fixed slope $m_h$ per head (a geometric sequence, e.g. $1/2, 1/4, \dots, 1/256$ for 8 heads). Nearby tokens are favoured, each head at a different range, and nothing is tied to a maximum length — ALiBi models extrapolate to sequences several times longer than they were trained on, at some cost in long-range precision.

Extending context after training

Positions are also what breaks when you ask for a longer context. RoPE trained on 4k tokens produces unseen rotation angles at position 8k. Two fixes: position interpolation rescales positions by $L_{\text{train}}/L_{\text{new}}$ so they stay inside the trained range, followed by a short fine-tune; NTK-aware scaling and YaRN rescale the base frequency instead (raising the $10000$ base, which Llama 3 pushes to $500{,}000$) so that low frequencies stretch while high frequencies keep their resolution. Long-context models then continue pretraining on long sequences. Vision Transformers face the same problem in 2-D and use learned patch positions or 2-D RoPE variants (5.7 Vision Transformers).

Where this shows up later

• Llama-style decoders combine RoPE with RMSNorm, SwiGLU and GQA — 7.6 Decoder-only: GPT and autoregressive generation, 7.10 Efficient inference: KV cache, GQA, FlashAttention, speculative decoding, quantization.
• Context extension and the quadratic cost of long sequences — 7.12 Beyond vanilla attention: state space models, long context and hybrids; the limits of long context versus retrieval — 9.1 Why RAG, and the canonical pipeline.
• The same "inject position, then attend" recipe is used for image patches in ViT (5.7 Vision Transformers) and for audio frames in Whisper (8.10 Multimodal models).