Loss functions catalog

Losses are negative log-likelihoods

Almost every loss in deep learning comes from one recipe (1.7 Probability II: Bayes, independence, MLE and MAP): assume a probabilistic model $p(y\mid\mathbf{x};\theta)$ for the output, and minimize the negative log-likelihood of the data,

The network outputs the parameters of that distribution (a mean, a probability, a vector of class probabilities), and the choice of distribution is the choice of loss. Seeing losses this way tells you which loss belongs to which problem and why some losses are robust and others are not.

Regression losses

• Mean squared error. Assume $y = f_\theta(\mathbf{x}) + \epsilon$ with $\epsilon\sim\mathcal{N}(0,\sigma^2)$. Then $-\log p(y\mid\mathbf{x}) = \frac{(y-f_\theta(\mathbf{x}))^2}{2\sigma^2} + \text{const}$, so maximizing likelihood is minimizing $\mathrm{MSE} = \frac1n\sum (y_i-\hat y_i)^2$ (3.2 Linear regression). Its gradient $2(\hat y - y)$ grows with the error, so a single outlier can dominate a batch.
• Mean absolute error. Assume Laplace noise, $p(\epsilon)\propto e^{-|\epsilon|/b}$, and you get $\mathrm{MAE}=\frac1n\sum|y_i-\hat y_i|$: constant-magnitude gradient, robust to outliers, but non-smooth at zero and it estimates the median rather than the mean.
• Huber loss interpolates: $\frac12 r^2$ for $|r|\le\delta$ and $\delta(|r|-\frac{\delta}{2})$ beyond, where $r=y-\hat y$. Quadratic near the target (smooth, fast convergence), linear far away (robust). Default in many regression and RL value-function heads (4.11 Reinforcement learning essentials).

Classification losses

Binary cross-entropy. Model $y\in\{0,1\}$ as Bernoulli with $\hat p = \sigma(z)$ where $z$ is the logit. The NLL is

Categorical cross-entropy. For $K$ classes with one-hot target $\mathbf{y}$ and softmax probabilities $\mathbf{p}=\mathrm{softmax}(\mathbf{z})$,

the negative log-probability assigned to the correct class. This is also the cross-entropy $H(\mathbf{y},\mathbf{p})$ of 1.9 Information theory: entropy, cross-entropy and KL divergence between the empirical distribution and the model, and for language models it is the per-token loss whose exponential is the perplexity (6.2 Classical NLP and n-gram language models, 7.6 Decoder-only: GPT and autoregressive generation). Its gradient with respect to the logits is startlingly simple — derived in 4.4 Backpropagation, fully derived:

Predicted minus actual: the network is pushed to raise the correct logit and lower the others in proportion to how much probability it wrongly assigned them.

Numerical stability: never exponentiate without the trick

Softmax involves $e^{z_k}$, and $e^{89}$ already overflows float32. Because softmax is shift-invariant (4.2 Activation functions), subtract the maximum logit $m=\max_j z_j$ first. The log-sum-exp trick:

where every exponent is now $\le 0$. Then $\log p_k = z_k - \log\sum_j e^{z_j}$ and $\mathrm{CE}=-z_y+\mathrm{LSE}(\mathbf{z})$ with no explicit probabilities. For the binary case the stable form is

This is exactly what nn.CrossEntropyLoss and nn.BCEWithLogitsLoss compute. Rule: the model outputs logits; the loss owns the softmax/sigmoid. Computing log(softmax(z)) yourself gives $\log 0 = -\infty$ as soon as one class underflows.

Worked example. Three classes, logits $\mathbf{z}=(2.0,\,1.0,\,0.1)$, correct class 0. Exponentials: $7.389,\ 2.718,\ 1.105$; sum $11.212$; probabilities $\mathbf{p}=(0.659,\,0.242,\,0.099)$. Loss $=-\ln 0.659 = 0.417$. Gradient $\mathbf{p}-\mathbf{y} = (-0.341,\ 0.242,\ 0.099)$: the correct logit is pushed up, the others down. With the stable route: $m=2$, $\mathrm{LSE}=2+\ln(1+0.368+0.150)=2+0.417=2.417$, and $\mathrm{CE}=-2.0+2.417=0.417$. ✓

Modifiers for real data

• Class weights. Multiply each example's loss by a weight $w_{y}$ (often inverse class frequency) to stop a 99:1 dataset from being solved by predicting the majority (3.13 Feature engineering, data quality and data-centric ML).
• Label smoothing. Replace the one-hot target by $y_k^{\text{LS}} = (1-\varepsilon)y_k + \varepsilon/K$. The optimum is no longer a logit of $+\infty$ for the correct class, so the model stays calibrated and less over-confident; $\varepsilon=0.1$ is standard in vision and machine translation. In the example above, $\varepsilon=0.1$ gives smoothed targets $(0.933,0.033,0.033)$ and a loss of $0.514$ instead of $0.417$: the model is charged for the probability it refused to spread.
• Focal loss $-(1-p_y)^{\gamma}\log p_y$ down-weights examples the model already gets right ($p_y$ near 1), focusing on hard ones; $\gamma=2$ fixed the extreme foreground/background imbalance in one-stage detectors (5.5 Object detection).
• Hinge loss $\max(0,\,1-y\,z)$ with $y\in\{-1,+1\}$ wants a margin, not a probability — the SVM loss of 3.10 Support vector machines and kernels; rarely used in deep nets today.

Distribution-matching and metric losses

• KL divergence $D_{\mathrm{KL}}(p\,\|\,q)=\sum_k p_k\log\frac{p_k}{q_k}$ (1.9 Information theory: entropy, cross-entropy and KL divergence) matches one distribution to another: a student to a teacher in knowledge distillation (soft targets at temperature $T$), the approximate posterior to the prior in a VAE (8.8 Generative models I: VAEs and GANs), and the fine-tuned policy to its reference model in RLHF (8.3 Alignment: RLHF, PPO, DPO, GRPO and Constitutional AI). Minimizing $D_{\mathrm{KL}}(\text{data}\,\|\,\text{model})$ is minimizing cross-entropy, because the data entropy is constant.
• Contrastive losses learn embeddings rather than labels. The InfoNCE loss for a batch of matched pairs with similarities $s_{ij}$ and temperature $\tau$,

is just cross-entropy where the "classes" are the other items in the batch and the correct class is the true partner. It trains SimCLR (5.8 Self-supervised and contrastive learning), CLIP (8.10 Multimodal models) and every sentence-embedding model behind RAG (9.2 Embeddings deep dive). Triplet loss $\max(0,\ d(a,p)-d(a,n)+m)$ is the older margin-based cousin.

• Dice / IoU losses optimize overlap directly for segmentation masks where cross-entropy is swamped by background pixels (5.6 Segmentation).

The catalog in one table

Losses vs metrics

The loss must be differentiable and well-behaved; the metric you care about (accuracy, F1, BLEU, pass@1) usually is not. Cross-entropy is a smooth surrogate for accuracy; it keeps improving the margin after accuracy saturates, which is why validation loss can rise while accuracy is flat (over-confidence on a few errors). Report metrics, optimize losses, and check that the two agree on your validation set (3.5 Evaluating models: metrics, validation and the bias–variance trade-off).

Where this shows up later

• The gradient $\mathbf{p}-\mathbf{y}$ is the first step of the backprop derivation — 4.4 Backpropagation, fully derived; loss-curve pathologies and NaNs from unstable losses — 4.8 Training dynamics and debugging.
• Focal and Dice losses in detection and segmentation — 5.5 Object detection, 5.6 Segmentation; InfoNCE in self-supervised vision and CLIP — 5.8 Self-supervised and contrastive learning, 8.10 Multimodal models.
• Per-token cross-entropy as the language-modeling objective and perplexity — 6.2 Classical NLP and n-gram language models, 7.6 Decoder-only: GPT and autoregressive generation; KL penalties and preference losses in alignment — 8.3 Alignment: RLHF, PPO, DPO, GRPO and Constitutional AI; the ELBO's KL term — 8.8 Generative models I: VAEs and GANs.