Regularization in deep nets

Capacity is not the whole story

Classical learning theory (3.5 Evaluating models: metrics, validation and the bias–variance trade-off, 3.6 Regularization and generalization) says generalization is governed by capacity: more parameters, more overfitting, unless you regularize. Zhang et al. (2017) showed that standard CNNs reach zero training error on ImageNet images with randomly shuffled labels — the capacity to memorize anything is already there — yet the same networks generalize well on real labels. So the architecture, the data and the optimizer together carry an implicit regularization that classical counts miss. Explicit regularizers still matter, especially with small data, but they work alongside this implicit bias rather than being the whole explanation.

Dropout

Training rule. For each hidden unit and each forward pass, draw an independent mask $m_i\sim\mathrm{Bernoulli}(1-p)$ and multiply the activation by it: $\tilde h_i = m_i h_i$. With $p=0.5$ half the units vanish on each step, so no unit can rely on a specific partner being present (co-adaptation is prevented) and the network learns redundant, individually useful features.

The scaling derivation. At test time we want a deterministic network whose activations match what downstream layers saw on average during training. Since $\mathbb{E}[m_i]=1-p$,

The original formulation therefore multiplies activations by $(1-p)$ at inference. Inverted dropout moves the correction to training time instead:

so inference is simply the identity — no special-casing, and the layer disappears entirely at test time. This is what nn.Dropout implements and why model.eval() is required.

Worked example. $\mathbf{h}=(2,\,-1,\,4,\,0.5)$, $p=0.5$, sampled mask $(1,0,1,1)$: inverted dropout outputs $(4,\,0,\,8,\,1)$. Average over all 16 masks and you recover $(2,-1,4,0.5)$ exactly.

Interpretations. (i) An implicit ensemble of $2^{n}$ sub-networks sharing weights, averaged approximately at test time; (ii) multiplicative noise injection, which for linear models is equivalent to an adaptive L2 penalty; (iii) a stochastic regularizer whose expected value is GELU-like (4.2 Activation functions). Practice: $p=0.5$ in the fully-connected layers of classical CNNs, $0.1$–$0.2$ in transformers for fine-tuning and small-data training, 0 in most LLM pretraining since 2022 — with enough data, dropout only slows learning. Place it after the activation, avoid stacking it right before BatchNorm (variance shift), and prefer stochastic depth / DropPath — dropping an entire residual branch with probability increasing with depth — in ViTs and deep ResNets (5.7 Vision Transformers). Spatial dropout drops whole channels in CNNs because adjacent pixels are too correlated for element-wise masks to matter.

Weight decay

Shrinking the weights toward zero every step, $\theta\leftarrow(1-\eta\lambda)\theta-\eta\hat{\mathbf{g}}$, is weight decay; for SGD it is identical to adding $\frac{\lambda}{2}\|\theta\|^2$ to the loss (L2 regularization, the ridge penalty of 3.6 Regularization and generalization), and for Adam it must be decoupled (AdamW, 4.5 Optimizers and learning-rate schedules). Two modern subtleties. First, layers followed by a normalization are scale-invariant: multiplying their weights by any constant leaves the output unchanged, so weight decay cannot "shrink the function" — instead it shrinks the weight norm, which increases the effective learning rate of that layer (the gradient scales as $1/\|\theta\|$). Weight decay on normalized networks is partly a learning-rate controller. Second, do not decay biases, LayerNorm/BatchNorm gains, or (usually) embeddings; the GPT-3 recipe decays only the 2-D weight matrices. Typical values: $10^{-4}$ for SGD on CNNs, $0.01$–$0.1$ for AdamW.

Early stopping

Monitor the validation metric, keep the best checkpoint, stop after a patience of $k$ evaluations without improvement. For linear models, stopping gradient descent after $T$ steps is equivalent to L2 regularization with $\lambda\approx 1/(\eta T)$ — training time is capacity. In practice the "best checkpoint" rule costs nothing and should always be on; the caveat is that with the double-descent dynamics below, the validation curve can go up and then down again, so do not stop too eagerly on noisy curves.

Data augmentation

Augmentation encodes invariances you know the task has: a cat flipped horizontally, cropped, slightly rotated or recolored is still a cat. Each epoch sees different variants, so the effective dataset is far larger and the model cannot memorize pixel patterns. In vision it is the most powerful regularizer available — RandAugment/AutoAugment policies, random resized crops and flips are standard (5.4 Transfer learning and training CNNs in practice). For text: back-translation, synonym replacement, token dropout; for audio: SpecAugment (masking time and frequency bands). mixup blends pairs of examples and labels,

encouraging linear behaviour between training points and improving calibration; CutMix pastes a patch of one image onto another and mixes labels by area. Label smoothing (4.3 Loss functions catalog) is augmentation of the targets rather than the inputs. The common thread: never show the network exactly the same (input, target) pair twice.

Implicit regularization

SGD's noise prefers flat minima, where the loss changes slowly under weight perturbation; such solutions tend to generalize better than sharp ones. Small batches and large learning rates increase this noise (4.5 Optimizers and learning-rate schedules). Architecture is a prior too: convolution's weight sharing says "the same feature matters everywhere" (5.1 Convolution: the operation and its math), attention says "relevance is content-based" (7.2 Scaled dot-product and multi-head attention, fully derived). And gradient descent itself, started from a small initialization, tends toward minimum-norm, low-complexity solutions among the many that fit the data.

Memorization versus generalization: double descent and grokking

Classical theory predicts test error that falls, then rises with capacity (the U-curve). Belkin et al. and Nakkiran et al. showed that beyond the interpolation threshold — where the model can just barely fit all training data — test error comes back down as capacity grows further: double descent. The peak occurs where the model is forced to fit the data with no slack; more parameters give it many ways to fit and SGD picks a smoother one. The same peak appears when varying training epochs or data size, which is why adding a little data can occasionally hurt. Grokking (Power et al., 2022) is the time-domain cousin: on small algorithmic datasets a network memorizes the training set quickly, sits at chance-level test accuracy for thousands of steps, and then abruptly generalizes — with weight decay as the ingredient that eventually favours the simpler, general circuit over the memorized lookup table. Together these results explain the modern instinct that scale plus data is a regularizer in its own right (7.9 Scaling laws and compute-optimal training).

A practical recipe

1. More or better data first, then augmentation that matches the task's invariances.
2. Weight decay (decoupled, on matrices only): 0.01–0.1 for AdamW.
3. Early stopping on the metric you care about, always keeping the best checkpoint.
4. Dropout 0.1–0.5 only when the dataset is small relative to the model; stochastic depth for deep residual nets.
5. Label smoothing / mixup for classification when calibration matters.
6. If still overfitting: make the model smaller or — often better — pretrain on more data and fine-tune (5.4 Transfer learning and training CNNs in practice, 8.2 Fine-tuning and PEFT), where a low LoRA rank acts as yet another regularizer.

Where this shows up later

• Reading the train/validation gap in loss curves — 4.8 Training dynamics and debugging; weight decay as an optimizer setting — 4.5 Optimizers and learning-rate schedules.
• Augmentation policies and fine-tuning recipes for vision — 5.4 Transfer learning and training CNNs in practice; stochastic depth in ViTs — 5.7 Vision Transformers; dropout settings in transformer blocks — 7.4 Inside the block: residuals, LayerNorm, FFN and embeddings.
• Why LLM pretraining uses no dropout but heavy weight decay and why scaling data beats regularizing — 7.9 Scaling laws and compute-optimal training, 8.1 The LLM lifecycle and model landscape; regularizing fine-tunes with LoRA rank and early stopping — 8.2 Fine-tuning and PEFT; double descent in the modern-view discussion of 3.6 Regularization and generalization.