Project: character-level language model

What you will build

Four stages, each a few dozen lines of PyTorch, each runnable on a laptop CPU:

1. Bigram model from counts on a list of ~32k names (the makemore dataset) — an n-gram model (6.2 Classical NLP and n-gram language models).
2. The same bigram model as a neural network — a one-hot input, one linear layer, softmax, trained with negative log-likelihood. Its loss should match the count model's.
3. An MLP language model in the style of Bengio et al. (2003): embed the previous $k$ characters, concatenate, hidden layer, softmax over the alphabet (6.3 Word embeddings, 4.1 From perceptron to MLP; universal approximation).
4. A tiny attention seq2seq model that normalizes dates ("March 5, 2024" → "2024-03-05") or reverses strings, with greedy and beam decoding (6.6 Seq2seq, encoder–decoder and beam search, 6.7 Attention is born).

Follow Karpathy's first three makemore videos for stages 1–3; the lessons below tell you what to look for and what numbers to expect.

Stage 1 — counts

Add a boundary token "." to each name, count every character bigram into a $27\times27$ matrix $N$, and normalize rows to get $P(c_{t}\mid c_{t-1})$. Evaluate with the average negative log-likelihood over all bigrams in the data,

Expected: NLL ≈ 2.45 nats (perplexity ≈ 11.6 — the model is as uncertain as a uniform choice among ~12 of the 27 characters). Add 1 to every count before normalizing (add-one smoothing) so unseen bigrams do not give infinite loss (6.2 Classical NLP and n-gram language models). Sample names with torch.multinomial; they will look vaguely name-like and mostly silly.

Stage 2 — the same model, learned

One-hot encode the previous character ($\mathbb{R}^{27}$), multiply by a weight matrix $\mathbf{W}\in\mathbb{R}^{27\times27}$, take a softmax over the result, and minimize the NLL with gradient descent. Because a one-hot vector selects a row of $\mathbf{W}$, the logits for "previous character $i$" are just row $i$: the network is a table of log-counts. After training, the loss converges to the same ≈2.45 — and now you have a procedure that scales to models for which counting is impossible. Add an $\ell_2$ penalty on $\mathbf{W}$ and notice it behaves exactly like add-$k$ smoothing (3.6 Regularization and generalization).

import torch, torch.nn.functional as F
xs, ys = ...                      # tensors of previous-char and next-char indices
W = torch.randn(27, 27, requires_grad=True)
for step in range(200):
    logits = F.one_hot(xs, 27).float() @ W          # (M, 27)
    loss = F.cross_entropy(logits, ys) + 0.01 * (W**2).mean()
    W.grad = None; loss.backward()
    W.data -= 50 * W.grad                            # plain gradient descent
print(loss.item())                                   # -> about 2.45

Stage 3 — an MLP with embeddings

Use a context of the previous $k=3$ characters, each mapped through an embedding table $\mathbf{E}\in\mathbb{R}^{27\times d}$ (start with $d=10$), concatenated into a $3d$-vector, passed through a hidden layer ($\tanh$, 200 units) and a softmax over 27 characters. Split the names 80/10/10 into train/dev/test and use mini-batches (4.5 Optimizers and learning-rate schedules). Watch for: the initial loss should be about $\log 27 = 3.30$ if the output layer is initialized small (a much larger initial loss means confidently wrong initial predictions — fix the initialization, 4.6 Initialization and normalization); dead or saturated $\tanh$ units; and the learning-rate sweep trick (increase the LR exponentially over one epoch and plot the loss). Expected dev NLL ≈ 2.1–2.2 — a real improvement over bigrams, because the model shares statistics across contexts through the embedding. Plot the 2-D embedding (set $d=2$ for the plot): vowels cluster together. Then sample at temperatures 0.7, 1.0 and 1.3 by dividing the logits by $T$ before the softmax (8.5 Decoding and sampling) and describe the difference.

Stage 4 — attention seq2seq

Generate synthetic pairs: random dates rendered in several human formats as source, ISO format as target (or simply strings and their reverses). Build a one-layer GRU encoder (6.5 LSTM and GRU) and a GRU decoder with Luong dot-product attention over encoder states (6.7 Attention is born), trained with teacher forcing (6.6 Seq2seq, encoder–decoder and beam search). Decode greedily, then implement beam search with $B=3$ and length normalization. Expected: near-perfect accuracy on dates after a few minutes of training; a plotted attention matrix shows the decoder jumping to the month name when it emits the month digits. Measure how often beam search changes the output compared with greedy (on this easy task, rarely — explain why).

Deliverables

• A repository with four scripts or one notebook, fixed seeds, and a results table: bigram-counts NLL, bigram-NN NLL, MLP dev/test NLL and perplexity, seq2seq accuracy (greedy vs beam).
• Twenty sampled names per model and five attention heatmaps.
• A short write-up: why the learned bigram matches the counts, what the embedding learned, and what the attention plots show.

Stretch goals

Implement makemore parts 4–5 (manual backprop through the whole MLP, then a WaveNet-style hierarchical model); replace the GRU seq2seq with a two-layer bidirectional encoder; or swap the character alphabet for a 512-merge BPE vocabulary you train yourself (6.1 Text to numbers: tokenization) and compare perplexities per character so they remain comparable.

Where this shows up later

• The same data pipeline, loss and sampling loop scale directly to GPT in 7.8 Build GPT from scratch (nanoGPT walkthrough); initialization and activation-statistics debugging is 4.6 Initialization and normalization, 4.8 Training dynamics and debugging.
• Perplexity and held-out evaluation — 6.2 Classical NLP and n-gram language models, 8.7 Evaluating LLMs; temperature sampling — 8.5 Decoding and sampling; attention heatmaps vs real interpretability — 8.11 Interpretability: what is happening inside.