Text to numbers: tokenization

Models can't read

A neural network consumes vectors of numbers. To feed it text we need a vocabulary $V$ of symbols, a rule that splits a string into a sequence of those symbols (tokens), and an integer id for each. The id then selects a row of an embedding matrix (6.3 Word embeddings, 7.4 Inside the block: residuals, LayerNorm, FFN and embeddings) and the model never sees the characters again. Three design choices sit on a triangle:

Subword tokenizers keep frequent words whole ("the", "machine") and split rare ones into meaningful pieces ("tokeniz" + "ation"), so there are no unknown words and the vocabulary stays manageable.

Byte-pair encoding, step by step

BPE (Sennrich et al., 2016, adapting a 1994 compression algorithm) learns a vocabulary from data:

1. Start with a base vocabulary of characters (or bytes) and represent every word as a sequence of them, with an end-of-word marker.
2. Count all adjacent symbol pairs across the corpus (weighted by word frequency).
3. Merge the most frequent pair into a new symbol; add it to the vocabulary and record the merge rule.
4. Repeat until the vocabulary reaches the target size. The number of merges is the only hyperparameter.

Worked example. Corpus word counts: low ×5, lower ×2, newest ×6, widest ×3. Write each as characters plus the marker ⟨/w⟩: l o w ⟨/w⟩, l o w e r ⟨/w⟩, n e w e s t ⟨/w⟩, w i d e s t ⟨/w⟩. Pair counts: (e,s) = 6+3 = 9, (s,t) = 9, (t,⟨/w⟩) = 9, (w,e) = 2+6 = 8, (l,o) = 7, (o,w) = 7, (n,e) = 6, (e,w) = 6, (w,⟨/w⟩) = 5, and so on.

• Merge 1: (e, s) → es [9]. Now newest = n e w es t ⟨/w⟩.
• Merge 2: (es, t) → est [9].
• Merge 3: (est, ⟨/w⟩) → est⟨/w⟩ [9].
• Merge 4: (l, o) → lo [7]. Merge 5: (lo, w) → low [7].
• Merge 6: (n, e) → ne; Merge 7: (ne, w) → new; Merge 8: (new, est⟨/w⟩) → newest⟨/w⟩ [6] …

To tokenize an unseen word, apply the merge rules in the order they were learned: lowest → l o w e s t ⟨/w⟩ → (merge 1) l o w es t → (2) l o w est → (3) l o w est⟨/w⟩ → (4) lo w est⟨/w⟩ → (5) low est⟨/w⟩. Result: ["low", "est⟨/w⟩"] — a word never seen in training, represented by two meaningful pieces. The compression is real: 4 words × ~6 characters became a handful of tokens.

Intuition. BPE is a greedy compression algorithm. Frequent strings get short codes (one token); rare strings get spelled out. That is why common English words are one token and a rare Finnish word may be eight.

The modern variants

Byte-level BPE (GPT-2 and every OpenAI model since, Llama 3): the base vocabulary is the 256 byte values, so anything — emoji, code, any script — can be tokenized with no unknown token. A regular-expression pre-tokenizer first splits text into chunks (words, numbers, punctuation, leading spaces) so merges never cross a space or mix letters with digits; GPT-4's pattern caps digit runs at three, and Llama 3 copies that choice. Vocabulary sizes grew from GPT-2's 50,257 to ~100k (GPT-4's cl100k_base), ~128k (Llama 3) and ~200k (o200k_base): bigger vocabularies mean fewer tokens per text (cheaper, longer effective context) at the cost of a larger embedding and output layer (7.4 Inside the block: residuals, LayerNorm, FFN and embeddings).

WordPiece (BERT): same merge loop, but it merges the pair that maximizes the likelihood gain $\frac{\text{count}(ab)}{\text{count}(a)\,\text{count}(b)}$ rather than raw frequency, and marks word-internal pieces with ## ("playing" → play, ##ing).

Unigram / SentencePiece (T5, Llama 1–2, many multilingual models): start from a large candidate vocabulary and prune it, removing the tokens whose deletion hurts the corpus likelihood least under a unigram model $P(\text{text})=\prod_i P(t_i)$. Segmentation is then the most probable split (Viterbi), which also allows sampling alternative segmentations as regularization. SentencePiece treats the input as a raw stream and encodes spaces as ▁, so it is language-agnostic.

All tokenizers add special tokens: ⟨|endoftext|⟩, BERT's [CLS]/[SEP]/[MASK], and the chat-template tokens that mark system/user/assistant turns (8.2 Fine-tuning and PEFT).

Why tokenization explains so many LLM failures

• Arithmetic. "1234567" may be split as 123 456 7; the model must learn place value across arbitrary chunk boundaries. Digit-grouping rules and tool use (10.2 Tool use and function calling) are the fixes.
• Spelling and counting letters. The model sees straw berry, not letters, so "how many r's?" is genuinely hard; reversing a word is harder still.
• Multilingual cost and fairness. English-centric vocabularies spend 2–5× more tokens on Hindi, Burmese or Amharic text than on English, so the same prompt costs more and fits less in the context window. Newer vocabularies (o200k, Gemma's 256k) narrow the gap.
• Code and whitespace. GPT-2 spent one token per space; indentation ate the context. GPT-4-era tokenizers merge runs of spaces.
• Glitch tokens. Rare tokens that appeared in the tokenizer's training text but almost never in the model's training data (the famous "SolidGoldMagikarp") have untrained embeddings and provoke bizarre behaviour.
• Trailing spaces and capitalization. "Hello" and " Hello" and "hello" are different tokens; a prompt ending in a space can push the model into an unlikely region.

Rule of thumb: one token ≈ ¾ of an English word ≈ 4 characters; 100k tokens of context is roughly 75k words. You will budget everything — prompts, retrieved chunks, memory — in tokens (9.4 Chunking and ingestion, 9.6 Generation side: context assembly, citations, structured outputs and tool calling).

Where this shows up later

• The token ids index the embedding table — 6.3 Word embeddings, 7.4 Inside the block: residuals, LayerNorm, FFN and embeddings; vocabulary size enters every parameter count — 7.4 Inside the block: residuals, LayerNorm, FFN and embeddings.
• Byte-level BPE is what you build in Karpathy's tokenizer video and what nanoGPT uses — 7.8 Build GPT from scratch (nanoGPT walkthrough).
• Context windows and costs are measured in tokens — 9.4 Chunking and ingestion, 9.6 Generation side: context assembly, citations, structured outputs and tool calling, 9.9 Production: latency, cost, caching, guardrails, security and privacy; tokenizer-free and byte-level architectures are an active research direction — 11.6 Frontier 2026: where the field is heading.