Regularization and generalization

What regularization is

Any modification to a learning algorithm intended to reduce its test error but not its training error is regularization. The classical form adds a penalty on the parameters to the empirical risk (3.1 What is learning? ERM, generalization and the ML workflow):

with $\lambda\ge 0$ controlling how much you care about small weights versus fitting the data. $\Omega$ encodes a belief about what good solutions look like: $\|\mathbf{w}\|_2^2$ says "small", $\|\mathbf{w}\|_1$ says "small and mostly zero". We will see the same objective through three lenses — a constraint, a shrinkage operator and a Bayesian prior — and then meet regularizers that are not penalties at all.

Ridge regression: the closed form and what it shrinks

Add the squared $\ell_2$ norm to least squares (3.2 Linear regression; the intercept is left unpenalized and features are standardized first):

The eigenvalues of $\mathbf{X}^\top\mathbf{X}$ are $\sigma_j^2\ge 0$ (squared singular values of $\mathbf{X}$, 1.3 Eigen-things, SVD and low rank); adding $\lambda\mathbf{I}$ lifts every one to $\sigma_j^2+\lambda>0$, so the matrix is always invertible — ridge is the fix for collinear or more-features-than-rows design matrices. The SVD $\mathbf{X}=\mathbf{U}\boldsymbol{\Sigma}\mathbf{V}^\top$ shows exactly what changes. Least squares projects $\mathbf{y}$ onto the column space: $\hat{\mathbf{y}}_{\text{OLS}}=\sum_j \mathbf{u}_j\mathbf{u}_j^\top\mathbf{y}$. Ridge gives

Each direction is kept with a shrinkage factor $\sigma_j^2/(\sigma_j^2+\lambda)$: near 1 for directions in which the data vary a lot (well-determined), near 0 for directions with tiny $\sigma_j$, where the least-squares solution was mostly fitting noise and had huge variance. Ridge trades a little bias for a large reduction in variance (3.5 Evaluating models: metrics, validation and the bias–variance trade-off) precisely where variance was the problem. In the special case of orthonormal columns every coefficient is shrunk by the same factor, $\mathbf{w}_\lambda = \mathbf{w}_{\text{OLS}}/(1+\lambda)$.

Lasso: why $\ell_1$ gives zeros

Replace the penalty by $\lambda\|\mathbf{w}\|_1 = \lambda\sum_j\lvert w_j\rvert$. There is no closed form in general (the $\ell_1$ norm is not differentiable at zero; coordinate descent or LARS solve it), but with orthonormal columns the problem separates per coordinate and the solution is soft-thresholding:

where $\hat w_j$ is the least-squares coefficient. Ridge divides every coefficient; lasso subtracts a constant and sets small ones to exactly zero. Worked example: $\hat{\mathbf{w}} = (3,\ 0.8,\ -0.2)$ with $\lambda=1$. Ridge: $(1.5,\ 0.4,\ -0.1)$. Lasso (threshold $0.5$): $(2.5,\ 0.3,\ 0)$ — the third feature is dropped. Lasso therefore performs feature selection, which is why it is used when you suspect most features are irrelevant; ridge is preferred when many features matter a little, and the elastic net $\lambda_1\|\mathbf{w}\|_1+\lambda_2\|\mathbf{w}\|_2^2$ mixes the two and handles groups of correlated features gracefully.

The geometric reason is the one from 1.1 Vectors, dot products and norms: the penalized problem is equivalent (by Lagrange duality, 1.10 Optimization: from gradient descent to Adam) to minimizing the loss subject to $\Omega(\mathbf{w})\le t$. The loss's elliptical contours expand until they first touch the constraint region. The $\ell_2$ ball is round, so the touching point is generic — no coordinate is special. The $\ell_1$ ball is a diamond whose corners sit on the axes, and an ellipse is far more likely to hit a corner than a flat face; at a corner some coordinates are exactly zero. In high dimensions the $\ell_1$ ball is almost all corners.

The Bayesian view: regularizers are priors

Return to the probabilistic model of 3.2 Linear regression, $y_i\sim\mathcal{N}(\mathbf{w}^\top\mathbf{x}_i,\sigma^2)$, and add a prior belief that weights are small: $\mathbf{w}\sim\mathcal{N}(\mathbf{0},\tau^2\mathbf{I})$. The log posterior (1.7 Probability II: Bayes, independence, MLE and MAP) is

Maximizing it (the MAP estimate) is minimizing $\|\mathbf{X}\mathbf{w}-\mathbf{y}\|^2 + \frac{\sigma^2}{\tau^2}\|\mathbf{w}\|^2$: ridge with $\lambda=\sigma^2/\tau^2$. Noisy data (large $\sigma^2$) or a strong prior (small $\tau^2$) mean more shrinkage, which is exactly right. A Laplace prior $p(w_j)\propto e^{-\lvert w_j\rvert/b}$, sharply peaked at zero, gives lasso with $\lambda = 2\sigma^2/b$. Every penalty is a prior and every prior is a penalty; when you meet weight decay in a transformer (4.7 Regularization in deep nets) or the KL penalty toward a reference policy in RLHF (8.3 Alignment: RLHF, PPO, DPO, GRPO and Constitutional AI), you are looking at a prior saying "stay close to what we already believed".

Regularizers that are not penalties

Early stopping. Run gradient descent on least squares from $\mathbf{w}=\mathbf{0}$ and stop after $t$ steps. In the eigenbasis of $\mathbf{X}^\top\mathbf{X}$ (eigenvalues $s_j$) the $j$-th component of the iterate is $w_{\text{OLS},j}\big[1-(1-\eta s_j)^t\big]$: high-curvature directions converge in a few steps, low-curvature ones (small $s_j$, the noisy directions) barely move. That is the same ordering of shrinkage as ridge, and matching the factors gives the correspondence $\lambda\approx 1/(\eta t)$. Stopping when validation error turns up is the most used regularizer in deep learning (4.7 Regularization in deep nets).

Data. More relevant data lowers variance directly (3.5 Evaluating models: metrics, validation and the bias–variance trade-off). Augmentation manufactures more data by applying transformations the label should be invariant to — flips, crops and color shifts for images (5.4 Transfer learning and training CNNs in practice), synonym swaps for text — and is the strongest regularizer in computer vision. Noise injection on inputs is, to second order, equivalent to an $\ell_2$ penalty on the weights (Bishop, 1995), and dropout (4.7 Regularization in deep nets) is noise injection on hidden units. Simpler models and fewer features are regularization by hypothesis-space design (3.1 What is learning? ERM, generalization and the ML workflow).

Complexity versus data, and double descent

The classical story: as capacity grows, training error falls monotonically while test error follows a U — the sweet spot sits before the model can interpolate (fit every training point). The interactive demo shows this U: raise the polynomial degree on noisy data and watch test error fall, then explode. The modern story adds a chapter. If you keep increasing capacity past the interpolation threshold (parameters $\approx$ data points), test error often falls again — double descent (Belkin et al., 2019; Nakkiran et al., 2019). At the threshold there is exactly one interpolating solution and it must contort wildly to hit every noisy point, so variance peaks; beyond it, infinitely many interpolants exist and gradient descent from small initialization finds the minimum-norm one, which is smooth — an implicit regularization with no penalty term. This is why heavily over-parameterized networks generalize, and it reframes the practical recipe: use a model far larger than the data would classically justify, then regularize (weight decay, augmentation, early stopping) and tune by validation.

Choosing λ

Sweep $\lambda$ on a logarithmic grid (e.g., $10^{-4}$ to $10^{2}$), cross-validate (3.5 Evaluating models: metrics, validation and the bias–variance trade-off), and plot the regularization path — coefficients versus $\lambda$ — to see which features survive. Pick the minimum validation error or, if you value simplicity, the one-standard-error rule: the largest $\lambda$ whose error is within one standard error of the best. Always standardize features first (the penalty treats all coordinates alike) and never penalize the intercept.

from sklearn.linear_model import RidgeCV, LassoCV
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
import numpy as np

alphas = np.logspace(-4, 2, 30)
ridge = make_pipeline(StandardScaler(), RidgeCV(alphas=alphas, cv=5)).fit(X, y)
lasso = make_pipeline(StandardScaler(), LassoCV(alphas=alphas, cv=5)).fit(X, y)
ridge[-1].alpha_, lasso[-1].alpha_, (lasso[-1].coef_ != 0).sum()   # chosen λ's and surviving features

Where this shows up later

• Weight decay, dropout, augmentation and early stopping in deep nets are these ideas at scale — 4.7 Regularization in deep nets; AdamW's decoupled weight decay is ridge done right — 4.5 Optimizers and learning-rate schedules.
• Double descent and grokking are the modern generalization puzzles — 4.7 Regularization in deep nets, 7.9 Scaling laws and compute-optimal training; "bigger model plus regularization" is the LLM recipe — 8.1 The LLM lifecycle and model landscape.
• The KL penalty in RLHF and DPO is a prior toward the reference model — 8.3 Alignment: RLHF, PPO, DPO, GRPO and Constitutional AI; LoRA's low rank is a structural regularizer — 8.2 Fine-tuning and PEFT.
• Sparsity from $\ell_1$ returns in sparse autoencoders for interpretability — 8.11 Interpretability: what is happening inside.