Training dynamics and debugging

The recipe

Karpathy's A Recipe for Training Neural Networks is the closest thing the field has to a checklist. Its order matters because each step makes the next step's failures interpretable:

1. Become one with the data. Look at hundreds of examples. Find duplicates, label noise, class imbalance, corrupted files, leakage between splits (3.13 Feature engineering, data quality and data-centric ML). Compute the trivial baselines: majority class, mean prediction.
2. Set up the end-to-end skeleton with dumb baselines. Fixed seed, tiny model, no augmentation, evaluate on the full validation set, plot everything. Verify the loss at initialization, make the model predict the label mean, overfit a single batch, and confirm that adding data decreases training loss.
3. Overfit. Scale the model until it fits the training set well. If it cannot overfit, there is a bug or a capacity problem — not a regularization problem.
4. Regularize (4.7 Regularization in deep nets): more data, augmentation, weight decay, dropout, early stopping — in that order.
5. Tune (4.5 Optimizers and learning-rate schedules): learning rate first, random search over the rest.
6. Squeeze: ensembles, longer training, averaging checkpoints.

Checks that take five minutes and save days

Expected initial loss. A $K$-way classifier with calibrated initialization predicts roughly uniform probabilities, so its cross-entropy should start near $\ln K$: $2.303$ for 10 classes, $\approx10.8$ for a 50k-token vocabulary. If the initial loss is far above that (say 7 for CIFAR-10), the logits are too large — the output layer's initialization is wrong, or there is a missing normalization — and the first hundred steps will be wasted undoing the confident randomness (4.6 Initialization and normalization). For regression with standardized targets, the initial MSE should be $\approx1$.

Overfit one batch. Train on a single batch of 8–32 examples for a few hundred steps. The loss must go to ~0 (or the exact minimum, e.g. $\ln 1 = 0$). If it does not: wrong labels alignment (shuffled inputs and targets), a loss that ignores half the outputs, gradients not flowing (a detach, a no_grad, an in-place op), zero_grad missing, the optimizer built before the model was moved to the GPU, or a learning rate so small nothing moves.

Input-independence baseline. Zero out the inputs: the model should do worse than with real inputs. If it does just as well, the inputs are not reaching the loss.

Visualize what the model sees. Decode the exact tensors fed to forward() — after augmentation, normalization and collation — back into images or text. Channel-order and normalization bugs are visible instantly and invisible otherwise.

Reading loss curves

Always plot on a log-step or log-loss axis when losses span orders of magnitude, and smooth with an exponential moving average rather than squinting at raw noise.

Statistics to log

• Gradient norm (global, pre-clipping) every step. Healthy: slowly decreasing, no spikes. Spikes precede loss spikes by a step.
• Update-to-weight ratio per layer, $\|\Delta\theta\|/\|\theta\|$. Healthy: around $10^{-3}$ per step. Much larger means the LR is too high for that layer; much smaller means it is barely learning (common for the first or last layer).
• Activation statistics per layer: mean, standard deviation, fraction saturated (tanh/sigmoid near ±1) or dead (ReLU at 0). Standard deviation should stay near 1 through depth at initialization (4.6 Initialization and normalization).
• Weight norms and, for BatchNorm, running statistics; learning rate actually applied (schedulers are often wired wrong); throughput (examples/s) to catch data-loader bottlenecks (4.9 PyTorch II: real training pipelines).

Numerical stability and mixed precision

Modern training runs in mixed precision: matmuls in 16-bit, accumulations and master weights in fp32. The two 16-bit formats behave very differently (4.12 Compute, hardware and scale basics):

• fp16 has 5 exponent bits: maximum $65{,}504$, smallest normal $\approx6\times10^{-5}$. Activations overflow to inf (→ NaN after any further op) and small gradients underflow to 0. Hence loss scaling: multiply the loss by a factor $S$ (e.g. $2^{16}$) before backward so gradients stay representable, divide by $S$ before the optimizer step, and shrink $S$ when an inf/NaN appears (torch.cuda.amp.GradScaler automates this).
• bf16 keeps fp32's 8 exponent bits with only 7 mantissa bits: same range as fp32, less precision. No loss scaling needed; the default on Ampere and newer GPUs and on TPUs.

Other NaN sources: $\log 0$ (use with-logits losses, 4.3 Loss functions catalog), $\sqrt{0}$ and division by zero (add $\epsilon$ in norms and Adam), exploding gradients (clip), and softmax or variance computed in 16-bit (force fp32 for reductions; attention softmax is computed in fp32 inside FlashAttention, 7.10 Efficient inference: KV cache, GQA, FlashAttention, speculative decoding, quantization). torch.autograd.set_detect_anomaly(True) finds the op that first produced a NaN at the cost of speed.

Reproducibility

Seed everything (torch, numpy, Python's random, CUDA), set torch.backends.cudnn.deterministic=True when exact repeatability matters (slower), and seed data-loader workers. Log the git commit, the full config and the library versions with every run; save checkpoints that include the optimizer, scheduler and RNG states so a resumed run continues the same trajectory (4.9 PyTorch II: real training pipelines). Expect run-to-run variance anyway and report mean ± spread over seeds (1.8 Statistics essentials).

A debugging checklist

• Data: labels aligned with inputs after shuffling? normalization applied once? leakage between splits? augmentation only on train?
• Model: train()/eval() set correctly? logits (not probabilities) to the loss? shapes broadcast as intended (a (B,1) vs (B,) target silently broadcasts into a (B,B) loss)?
• Optimization: zero_grad each step? scheduler stepped at the right granularity? gradient norm finite and non-zero? LR within an order of magnitude of the range-test value?
• Numerics: bf16 if available; fp16 only with a scaler; reductions in fp32.
• Evaluation: same preprocessing as training; metrics computed over the full set, not averaged per-batch when batch sizes differ.

Diagnostics in code

import torch, math

# expected initial loss for K classes
K = 10; print("expect ≈", math.log(K))

# global gradient norm (pre-clip) and update/weight ratios after one step
loss.backward()
gnorm = torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=float("inf"))
before = [p.detach().clone() for p in model.parameters()]
optimizer.step()
ratios = [((p - b).norm() / (b.norm() + 1e-12)).item() for p, b in zip(model.parameters(), before)]
print(f"grad norm {gnorm:.3f}; update/weight ratios:", [f"{r:.1e}" for r in ratios])   # want ~1e-3

# activation statistics via forward hooks
def stats_hook(name):
    def hook(mod, inp, out):
        o = out.detach().float()
        print(f"{name:20s} mean {o.mean():+.3f} std {o.std():.3f} dead {(o <= 0).float().mean():.2f}")
    return hook
for name, mod in model.named_modules():
    if isinstance(mod, torch.nn.ReLU):
        mod.register_forward_hook(stats_hook(name))

Where this shows up later

• The training template with AMP, clipping, logging and checkpoints — 4.9 PyTorch II: real training pipelines; the formats and memory numbers behind mixed precision — 4.12 Compute, hardware and scale basics.
• Fine-tuning diagnostics for vision backbones — 5.4 Transfer learning and training CNNs in practice; watching activation and gradient statistics in a GPT training run — 7.8 Build GPT from scratch (nanoGPT walkthrough); loss spikes and their fixes at LLM scale — 8.1 The LLM lifecycle and model landscape.
• Evaluation hygiene (held-out sets, seeds, variance) — 3.5 Evaluating models: metrics, validation and the bias–variance trade-off, 8.7 Evaluating LLMs; the same "overfit one example first" discipline for agents and RAG pipelines — 9.8 Evaluating and observing LLM apps, 10.9 Evaluating agents.