Decoder-only: GPT and autoregressive generation

One objective for everything

A decoder-only transformer keeps the causal mask of 7.2 Scaled dot-product and multi-head attention, fully derived and drops the encoder. Its training objective is the chain rule of probability turned into a loss (6.2 Classical NLP and n-gram language models):

Because of the mask, one forward pass over a $T$-token sequence produces all $T$ next-token distributions, so every token is a training example — the full 100% signal that MLM gives up (7.5 Encoder-only: BERT and friends (and sentence embeddings)). The same objective covers code, dialogue, translation and arithmetic: anything expressible as text becomes "predict what comes next".

GPT-1 → GPT-2 → GPT-3: what scaled

Check the parameter rule of 7.4 Inside the block: residuals, LayerNorm, FFN and embeddings on GPT-3: $12\times 96\times 12288^2 \approx 174$B plus a 617M embedding matrix — 175B. The architecture barely changed across the three generations; width, depth, data and compute did (7.9 Scaling laws and compute-optimal training).

GPT-3's result reframed the field. Given a few input→output examples in the prompt, the frozen model performs the task — no gradient updates. This in-context learning is what makes prompting (8.4 Prompting as programming) a programming interface, and interpretability later traced part of the mechanism to induction heads that copy earlier patterns (8.11 Interpretability: what is happening inside).

The generation loop

Inference is a loop around a single forward pass:

import torch, torch.nn.functional as F

@torch.no_grad()
def generate(model, ids, max_new=50, temperature=1.0):
    for _ in range(max_new):
        logits = model(ids)[:, -1, :]              # only the last position matters
        probs = F.softmax(logits / temperature, -1)
        nxt = torch.multinomial(probs, 1)          # sample (or argmax for greedy)
        ids = torch.cat([ids, nxt], dim=1)
    return ids

Two phases hide inside it. Prefill runs the prompt through the model once, in parallel; decode then appends one token per step. Written naively as above, step $t$ re-processes all $t$ tokens, so generating $T$ tokens costs $O(T^2)$ forward work; the KV cache stores every layer's keys and values so each new token costs $O(T)$ attention plus one pass through the weights (7.10 Efficient inference: KV cache, GQA, FlashAttention, speculative decoding, quantization). Generation stops at an end-of-sequence token or a length limit; how the next token is chosen — greedy, beam, temperature, top-k, top-p — is the subject of 8.5 Decoding and sampling.

Intuition. The model never "plans a sentence". It emits one distribution over the vocabulary, a token is drawn, and that token becomes part of the context for the next distribution. Everything that looks like planning is implicit in those distributions.

Why decoder-only won

1. Signal efficiency: every position is supervised.
2. One interface: input and output are the same stream, so any task is "continue the text" — no task-specific heads.
3. Generation is native: encoders cannot produce text; encoder–decoders need two stacks.
4. Scaling behaviour: clean, predictable power laws (7.9 Scaling laws and compute-optimal training) made it the safest bet for enormous compute budgets.
5. In-context learning turned the model into a general tool rather than a feature extractor.

Encoder–decoders remain competitive for input-heavy tasks (7.7 Encoder–decoder: T5, BART and choosing an architecture), and encoders for representations (7.5 Encoder-only: BERT and friends (and sentence embeddings)) — but the general-purpose "LLM" is a decoder.

The modern decoder recipe

Llama (2023) consolidated the changes that most open models now share:

• Pre-norm with RMSNorm instead of post-norm LayerNorm (7.4 Inside the block: residuals, LayerNorm, FFN and embeddings).
• RoPE instead of learned absolute positions (7.3 Positional information: sinusoidal, learned, RoPE, ALiBi).
• SwiGLU feed-forward with $d_{ff}\approx 8d/3$ (7.4 Inside the block: residuals, LayerNorm, FFN and embeddings).
• Grouped-query attention to shrink the KV cache (7.10 Efficient inference: KV cache, GQA, FlashAttention, speculative decoding, quantization).
• No biases, untied embeddings, a large vocabulary (128k tokens in Llama 3) and very long training — 15T tokens for Llama 3, far beyond the compute-optimal point because inference cost dominates (7.9 Scaling laws and compute-optimal training).

Read any model's config.json on the Hugging Face Hub and you can now decode it: hidden_size is $d$, intermediate_size is $d_{ff}$, num_key_value_heads reveals GQA, rope_theta is the RoPE base, tie_word_embeddings says whether the unembedding is shared.

From base model to assistant

A pretrained decoder is a base model: it continues text. Ask it a question and it may produce more questions, because that is what text on the internet does. Turning it into an assistant takes supervised fine-tuning on instruction–response pairs and preference optimization (8.3 Alignment: RLHF, PPO, DPO, GRPO and Constitutional AI); reasoning models add reinforcement learning on verifiable tasks (8.6 Reasoning models and test-time compute). The architecture is unchanged — the same loop above, with a chat template marking which tokens the user wrote and where the model should start producing.

Where this shows up later

• Building this exact model and training loop from scratch — 7.8 Build GPT from scratch (nanoGPT walkthrough); how big to make it — 7.9 Scaling laws and compute-optimal training.
• The KV cache, GQA and batching that make the generation loop fast — 7.10 Efficient inference: KV cache, GQA, FlashAttention, speculative decoding, quantization; decoding strategies — 8.5 Decoding and sampling.
• Prompting and in-context learning as the programming interface — 8.4 Prompting as programming; alignment that turns a base model into a chat model — 8.3 Alignment: RLHF, PPO, DPO, GRPO and Constitutional AI.
• Decoders are the brains of RAG systems (9.1 Why RAG, and the canonical pipeline) and agents (10.1 What is an agent? Loops, autonomy and workflows vs agents).