Scaling laws and compute-optimal training

Loss is predictable

In 2020 Kaplan et al. trained hundreds of decoder models and found that test loss follows smooth power laws in each of three resources, as long as the other two are not bottlenecks:

with $N$ non-embedding parameters, $D$ training tokens, $C$ compute, and exponents around $\alpha_N\approx 0.076$, $\alpha_D\approx 0.095$, $\alpha_C\approx 0.05$ — straight lines on log–log plots spanning seven orders of magnitude. Architecture details (depth vs width, number of heads) mattered far less than scale. Two consequences changed how labs plan: a bigger model is more sample-efficient (it reaches a given loss with fewer tokens), and you can extrapolate the loss of a run that costs millions from runs that cost thousands.

Counting compute: why FLOPs ≈ 6ND

Every weight in a matrix multiplication does one multiply and one add per token in the forward pass: $2N$ FLOPs per token for $N$ parameters (7.4 Inside the block: residuals, LayerNorm, FFN and embeddings). The backward pass computes two gradients per weight — one with respect to the layer input (to keep propagating, 4.4 Backpropagation, fully derived) and one with respect to the weight itself — so it costs about $4N$. Total: $6N$ FLOPs per token, or

Attention scores add roughly $12\,L\,d\,n$ per token for context length $n$, negligible when $n\ll 6d$ and significant for very long contexts (7.12 Beyond vanilla attention: state space models, long context and hybrids). Example: GPT-3, $N=175$B, $D=300$B → $C\approx 3.15\times 10^{23}$ FLOPs. On an H100 sustaining an effective 400 TFLOP/s (about 40% of its bf16 peak), that is $7.9\times 10^{8}$ seconds ≈ 9,100 GPU-days — three months on 100 GPUs.

Chinchilla: the right size for a budget

Kaplan's fits suggested spending extra compute mostly on parameters, and labs built models like Gopher (280B parameters, 300B tokens). Hoffmann et al. (2022) redid the experiment with learning-rate schedules matched to each run's length and fitted a joint law:

$E$ is the irreducible entropy of text (1.9 Information theory: entropy, cross-entropy and KL divergence); the other two terms are the penalties for finite model size and finite data. Minimizing $L$ subject to $6ND = C$ gives $N_{\text{opt}}\propto C^{a}$ and $D_{\text{opt}}\propto C^{b}$ with $a\approx b\approx 0.5$: parameters and tokens should grow together, and the optimal ratio is roughly

Gopher, at about one token per parameter, was badly under-trained. Chinchilla, a 70B model trained on 1.4T tokens with about the same compute, beat it across the board.

Worked example. Budget $C = 10^{23}$ FLOPs. With $D = 20N$: $6N\cdot 20N = 120N^2 = 10^{23}$, so $N = \sqrt{8.3\times 10^{20}}\approx 2.9\times 10^{10}$ — a 29B-parameter model trained on about 580B tokens. For $C = 10^{21}$: $N\approx 2.9$B and $D\approx 58$B tokens. The llm-calculator demo does this for any budget.

Inference changes the answer

Chinchilla optimizes training loss per training FLOP. But a deployed model is run billions of times, and inference cost scales with $N$, not $D$. If you expect heavy usage, it pays to train a smaller model on far more tokens than Chinchilla suggests — losing a little quality per training FLOP but gaining it back many times over at inference. Llama 3 8B was trained on 15T tokens, about 1,900 tokens per parameter (≈95× the Chinchilla ratio); such "over-trained" small models are now the norm, and their loss keeps falling with more data, just more slowly.

Data: the binding constraint

Tokens are not free. Estimates of the stock of high-quality public text run to a few hundred trillion tokens, and frontier training sets already use tens of trillions. Muennighoff et al. (2023) showed that repeating data for up to about four epochs costs almost nothing, after which returns diminish fast; beyond that, labs lean on filtering and deduplication (quality beats quantity), code and multilingual data, and synthetic data generated or rewritten by models — with care, since training on model outputs can narrow the distribution.

Emergence, or metric artifacts?

Wei et al. (2022) catalogued abilities (arithmetic, multi-step reasoning) that appear "suddenly" at some scale. Schaeffer et al. (2023) showed much of the suddenness comes from discontinuous metrics: exact-match accuracy jumps from 0 to 1 while the underlying per-token log-likelihood improves smoothly. The current view is that capability under smooth metrics is predictable, but thresholds on discrete tasks are real for users — a model either produces valid JSON for a tool call or it does not (10.2 Tool use and function calling).

Scaling in 2026

Pretraining scale still helps, but each 10× of compute buys less than it did, and the frontier added two new axes: post-training with reinforcement learning on verifiable tasks (8.3 Alignment: RLHF, PPO, DPO, GRPO and Constitutional AI, 8.6 Reasoning models and test-time compute), and test-time compute — letting the model think longer at inference, which has its own scaling curves (8.6 Reasoning models and test-time compute). Distillation moves frontier capability into small models, and mixture-of-experts (7.11 Mixture of Experts) decouples parameters from per-token compute, so "parameters" alone no longer predicts cost or quality.

Where this shows up later

• Parameter counting (7.4 Inside the block: residuals, LayerNorm, FFN and embeddings) and the 6ND rule are the inputs to every budgeting decision in this phase; the hardware side is 4.12 Compute, hardware and scale basics.
• Over-training small models for cheap inference connects to the KV-cache and quantization economics of 7.10 Efficient inference: KV cache, GQA, FlashAttention, speculative decoding, quantization.
• Test-time compute and RL post-training are the new scaling axes — 8.6 Reasoning models and test-time compute, 8.3 Alignment: RLHF, PPO, DPO, GRPO and Constitutional AI; deciding whether scale "emerged" a capability is an evaluation question — 8.7 Evaluating LLMs.