Build GPT from scratch (nanoGPT walkthrough)

The project

Karpathy's two-hour video builds a character-level GPT on the Tiny Shakespeare corpus (about 1 MB of text) and ends with a model that writes plausible fake Shakespeare. Everything from this phase appears in order: tokens → embeddings → attention head → multi-head → block → stacked GPT → training loop → sampling. Do not just watch — type every line, break things, and read the losses.

Stage by stage

1. Data and tokenizer. The text has 65 distinct characters; the "tokenizer" is a dictionary from character to integer (6.1 Text to numbers: tokenization). Encode the whole file into one tensor and split 90/10 into train and validation.
2. Batching. Pick block_size (the context length, 8 at first); a training example is a window of block_size inputs with the same window shifted by one as targets. One window yields block_size prediction problems at once — the causal objective of 7.6 Decoder-only: GPT and autoregressive generation in miniature.
3. Bigram baseline. An embedding table of shape $V\times V$ turns each character into logits for the next one. The initial loss should be about $\ln 65\approx 4.17$ (uniform guessing, 1.9 Information theory: entropy, cross-entropy and KL divergence); training brings it to ≈2.5. Sampling already produces letter-like gibberish.
4. A single attention head. Keys, queries and values from linear layers; scores masked with torch.tril; scaled by $\sqrt{d_k}$; softmax; weighted sum of values — the function from 7.2 Scaled dot-product and multi-head attention, fully derived written out. The loss improves because characters can now see their context.
5. Multi-head, FFN, block. Several heads concatenated, a position-wise MLP, then residual connections and LayerNorm (pre-norm) so that deeper stacks train (7.4 Inside the block: residuals, LayerNorm, FFN and embeddings). Dropout regularizes the small dataset.
6. Scale up. n_embd=384, 6 layers, 6 heads, block_size=256, dropout 0.2, learning rate 3e-4, about 5,000 steps on a GPU → validation loss ≈1.48 and genuinely Shakespeare-shaped output.

The model in forty lines

import torch, torch.nn as nn, torch.nn.functional as F

class CausalSelfAttention(nn.Module):
    def __init__(self, d, n_head, block_size, dropout):
        super().__init__()
        self.h, self.qkv, self.proj = n_head, nn.Linear(d, 3 * d), nn.Linear(d, d)
        self.drop = nn.Dropout(dropout)
        mask = torch.tril(torch.ones(block_size, block_size))
        self.register_buffer('mask', mask.view(1, 1, block_size, block_size))

    def forward(self, x):
        B, T, C = x.shape
        q, k, v = self.qkv(x).split(C, dim=2)
        q, k, v = [t.view(B, T, self.h, C // self.h).transpose(1, 2) for t in (q, k, v)]
        att = (q @ k.transpose(-2, -1)) / (C // self.h) ** 0.5
        att = att.masked_fill(self.mask[:, :, :T, :T] == 0, float('-inf'))
        y = self.drop(F.softmax(att, dim=-1)) @ v
        return self.proj(y.transpose(1, 2).reshape(B, T, C))

class Block(nn.Module):
    def __init__(self, d, n_head, block_size, dropout):
        super().__init__()
        self.ln1, self.ln2 = nn.LayerNorm(d), nn.LayerNorm(d)
        self.attn = CausalSelfAttention(d, n_head, block_size, dropout)
        self.mlp = nn.Sequential(nn.Linear(d, 4 * d), nn.GELU(),
                                 nn.Linear(4 * d, d), nn.Dropout(dropout))

    def forward(self, x):
        x = x + self.attn(self.ln1(x))
        return x + self.mlp(self.ln2(x))

class GPT(nn.Module):
    def __init__(self, vocab, d=384, n_layer=6, n_head=6, block_size=256, dropout=0.2):
        super().__init__()
        self.tok, self.pos = nn.Embedding(vocab, d), nn.Embedding(block_size, d)
        self.blocks = nn.Sequential(*[Block(d, n_head, block_size, dropout) for _ in range(n_layer)])
        self.ln_f, self.head = nn.LayerNorm(d), nn.Linear(d, vocab, bias=False)

    def forward(self, idx, targets=None):
        B, T = idx.shape
        x = self.tok(idx) + self.pos(torch.arange(T, device=idx.device))
        logits = self.head(self.ln_f(self.blocks(x)))
        loss = None if targets is None else F.cross_entropy(logits.view(-1, logits.size(-1)), targets.view(-1))
        return logits, loss

Pair it with the generate loop from 7.6 Decoder-only: GPT and autoregressive generation (crop idx to the last block_size tokens before each forward pass) and an AdamW training loop (4.5 Optimizers and learning-rate schedules).

What to watch while it trains

• Initial loss ≈ ln V. If it is much higher, the output layer's initialization is too large; nanoGPT scales the residual projections by $1/\sqrt{2L}$ for this reason.
• Train vs validation gap. With 1 MB of text and a 10M-parameter model you will overfit; that is dropout's job here, and it is why real models train on trillions of tokens for a single epoch (7.9 Scaling laws and compute-optimal training).
• Samples over time. Gibberish → words → grammar → character names and stage directions. Perplexity is $e^{\text{loss}}$ (1.9 Information theory: entropy, cross-entropy and KL divergence).
• Throughput. Measure tokens per second; then turn on bf16 autocast and torch.compile and measure again (4.9 PyTorch II: real training pipelines).

Math check. Parameter count with the formula of 7.4 Inside the block: residuals, LayerNorm, FFN and embeddings: $12\times 6\times 384^2 = 10.6$M for the blocks, plus $65\times 384$ and $256\times 384$ for the embeddings — about 10.7M. The video reports 10.8M including LayerNorm and bias terms. ✓

From toy to GPT-2 124M

nanoGPT's repository is the same model with a BPE tokenizer (GPT-2's 50,257 tokens), weight tying, and a data pipeline for OpenWebText or FineWeb. Karpathy's follow-up video reproduces GPT-2 (124M) on 10B tokens with careful attention to hyperparameters (warmup, cosine decay, weight decay 0.1, gradient clipping at 1.0, a batch of about 0.5M tokens) and evaluates on HellaSwag; llm.c does the same in raw C/CUDA, and nanochat (2025) extends the pipeline to tokenizer training, pretraining, SFT and a chat UI for about $100 of rented GPU time. Reading these repositories after this lesson is the single best way to make the whole phase concrete; Stanford's CS336 assignment 1 asks for the same build with a BPE tokenizer and RoPE.

Exercises, from easy to hard

1. Fuse the three per-head linear layers into one, as in the code above, and verify identical outputs against separate layers.
2. Train on your own text (lyrics, code, your chat history). Which block_size helps most?
3. Implement a KV cache in generate and confirm that per-token latency becomes flat (7.10 Efficient inference: KV cache, GQA, FlashAttention, speculative decoding, quantization).
4. Swap the learned position table for RoPE (7.3 Positional information: sinusoidal, learned, RoPE, ALiBi) and test on sequences longer than block_size.
5. Pretrain on a large text, then fine-tune on a small one; observe how few steps the fine-tune needs (8.2 Fine-tuning and PEFT).

Where this shows up later

• The same code, scaled, is every open LLM — compare with Llama's model file after 7.6 Decoder-only: GPT and autoregressive generation.
• The training loop's hyperparameters are governed by scaling laws — 7.9 Scaling laws and compute-optimal training; its inference loop by the KV cache — 7.10 Efficient inference: KV cache, GQA, FlashAttention, speculative decoding, quantization.
• Fine-tuning this model is the smallest possible version of 8.2 Fine-tuning and PEFT; sampling choices are 8.5 Decoding and sampling.