The LLM lifecycle and model landscape

Three stages, one model

A modern assistant is the same transformer (7.6 Decoder-only: GPT and autoregressive generation) trained in three stages. Pretraining teaches it to predict the next token on trillions of tokens of text and code; the result is a base model — a superb autocomplete with broad knowledge but no notion of being helpful. Supervised fine-tuning (SFT) continues training on a much smaller set of (prompt, ideal response) pairs so the model learns the format of being an assistant. Preference optimization and RL (RLHF, DPO, RL with verifiable rewards — 8.3 Alignment: RLHF, PPO, DPO, GRPO and Constitutional AI) then shape behavior: helpfulness, honesty, refusals, and — for reasoning models — the habit of thinking before answering (8.6 Reasoning models and test-time compute).

Intuition. Andrej Karpathy's analogy: pretraining is reading every textbook; SFT is studying worked examples; RL is doing practice problems and learning from whether you got them right.

Where the tokens come from

The bulk of pretraining data is filtered web crawl (Common Crawl snapshots), plus code repositories, books, Wikipedia, scientific papers, and increasingly synthetic data written by other models. Turning a crawl into training data is a pipeline of its own:

1. Text extraction from HTML (boilerplate removal).
2. Language identification and quality filtering — heuristics (document length, symbol ratios, repeated lines) and learned classifiers that score "educational value" or similarity to trusted sources.
3. Deduplication at document and near-document level (MinHash/LSH on shingles) — duplicates waste compute and increase memorization.
4. Safety and PII filtering, and decontamination: removing benchmark test sets so evaluations stay meaningful (8.7 Evaluating LLMs).
5. Mixing: choosing proportions of web/code/math/multilingual data, often with a final annealing phase that up-weights the highest-quality data.

Open pipelines such as FineWeb (about 15 trillion tokens of filtered Common Crawl) and fully open efforts like OLMo document every step; most frontier labs publish only summaries. Two empirical facts drive the field: data quality filtering buys as much as several times more raw data, and when data is scarce, repeating it for up to roughly four epochs costs little — beyond that, returns fall off.

Tokens, compute and money

From 7.9 Scaling laws and compute-optimal training: training FLOPs $C \approx 6ND$ for $N$ parameters and $D$ tokens. Chinchilla's compute-optimal rule is $D \approx 20N$, but production models are over-trained far beyond it because inference cost depends on $N$, not $D$: Llama 3's 8B model was trained on about 15 trillion tokens — nearly 2,000 tokens per parameter.

Math check. 8B parameters × 15T tokens: $C \approx 6 \times 8\times10^{9} \times 1.5\times10^{13} = 7.2\times10^{23}$ FLOPs. An H100 sustaining about $4\times10^{14}$ FLOP/s in bf16 (roughly 40% utilization) needs $7.2\times10^{23}/4\times10^{14} = 1.8\times10^{9}$ GPU-seconds, i.e. about half a million GPU-hours before counting failures and restarts. The Llama 3 report gives the actual figures — it is on the order of a million GPU-hours, consistent with this estimate.

At cloud prices of a few dollars per GPU-hour, that is millions of dollars for a small model; frontier runs use tens of thousands of accelerators for months. This is why almost nobody pretrains from scratch and why fine-tuning (8.2 Fine-tuning and PEFT) and retrieval (9.1 Why RAG, and the canonical pipeline) matter so much.

Base vs instruct vs reasoning

The chat template matters in practice: an instruct model expects its conversation serialized with the exact special tokens it was trained on; feeding it raw text degrades it. Libraries apply the template for you, but when you fine-tune (8.2 Fine-tuning and PEFT) you must match it.

The landscape in late 2026

Two tiers coexist. Frontier closed models are served through APIs by a handful of labs — Anthropic's Claude, OpenAI's GPT and o-series, Google DeepMind's Gemini families — and lead on the hardest reasoning, agentic and multimodal tasks. Open-weight models you can download and run include Meta's Llama, Alibaba's Qwen, DeepSeek, Mistral, Google's Gemma, plus families such as Microsoft's Phi, NVIDIA's Nemotron and the fully open OLMo. The capability gap between the best open and closed models has narrowed to months on many benchmarks, and open reasoning models exist since DeepSeek-R1 (January 2025).

A few distinctions worth keeping straight:

• Open weights ≠ open source. Most releases publish weights under a license (Apache-2.0 or MIT for some Qwen, DeepSeek and Mistral models; custom community licenses for Llama and Gemma) but not the training data or full code. OLMo-style projects release everything.
• Model families ship in sizes (e.g. 1B–400B+) and variants (base, instruct, reasoning, vision, long-context). Smaller models are distilled from or trained alongside bigger siblings.
• Trends that are settled: natively multimodal inputs, context windows of hundreds of thousands of tokens, reasoning modes with adjustable effort, tool use as a first-class capability (10.2 Tool use and function calling), and price tiers that differ by 50–100× between flagship and small models.

Exact version numbers and leaderboard positions change every few months; the way to stay current is a leaderboard (e.g. LMArena, Artificial Analysis) plus your own evaluation (8.7 Evaluating LLMs).

Choosing a model

Decide in this order:

1. Capability on your task — measured on your own eval set, not a public benchmark.
2. Cost — priced per million tokens, with output tokens typically 3–5× input and cached input often much cheaper. Estimate tokens per request × requests per day.
3. Latency — time to first token and tokens per second; reasoning modes add seconds to minutes.
4. Context window and modality — do you need long documents, images, audio?
5. Privacy and deployment — API with data-retention terms vs self-hosting open weights (needs GPUs and an inference server such as vLLM, 7.10 Efficient inference: KV cache, GQA, FlashAttention, speculative decoding, quantization).
6. License and ecosystem — commercial use, fine-tuning rights, tooling support.

A practical workflow: establish a ceiling with the strongest model you can afford, build the eval, then test cheaper and smaller models (and routing between them) until quality drops below what you need.

Reading a model card or technical report

Look for: training data description and cutoff, parameter count and architecture, tokens trained, post-training method, evaluation tables (with the exact prompting setup), safety evaluations, known limitations, license, and intended use. Treat missing information as information.

Where this shows up later

• Fine-tuning and PEFT build on the SFT stage — 8.2 Fine-tuning and PEFT; preference optimization and RL on verifiable rewards are the third stage — 8.3 Alignment: RLHF, PPO, DPO, GRPO and Constitutional AI, 8.6 Reasoning models and test-time compute.
• Benchmarks and model cards only mean something once you can evaluate — 8.7 Evaluating LLMs.
• Retrieval (9.1 Why RAG, and the canonical pipeline) and agents (10.1 What is an agent? Loops, autonomy and workflows vs agents) are built around model choice, cost and context limits introduced here.