Probability II: Bayes, independence, MLE and MAP

Joint, marginal, conditional

For two random variables, the joint $p(x,y)$ assigns probability to pairs. The sum rule recovers a marginal by summing out what you do not care about, $p(x)=\sum_y p(x,y)$, and the product rule defines the conditional: $p(x,y)=p(y\mid x)\,p(x)$, so $p(y\mid x)=p(x,y)/p(x)$ — the probability of $y$ in the world where $x$ is known. Applying the product rule repeatedly gives the chain rule of probability,

which is exactly how a language model scores a sentence: one conditional per token, each given everything before it (6.2 Classical NLP and n-gram language models, 7.6 Decoder-only: GPT and autoregressive generation).

Independence

$X$ and $Y$ are independent when $p(x,y)=p(x)\,p(y)$, equivalently $p(y\mid x)=p(y)$: learning $x$ tells you nothing about $y$. They are conditionally independent given $Z$ when $p(x,y\mid z)=p(x\mid z)\,p(y\mid z)$. Two assumptions of this kind underlie much of ML: that training examples are independent and identically distributed (i.i.d., 3.1 What is learning? ERM, generalization and the ML workflow), and naive Bayes' assumption that features are independent given the class (3.7 Naive Bayes and generative vs discriminative models). Independence turns products into sums in log space, which is why log-likelihoods of datasets are sums over examples.

Bayes' theorem

Writing the product rule both ways, $p(\theta\mid D)\,p(D)=p(D\mid\theta)\,p(\theta)$, and dividing:

The posterior is proportional to likelihood times prior; the evidence is just the normalizer. Worked example. A disease has prevalence 1%. A test has sensitivity 99% (positive if sick) and specificity 95% (negative if healthy). You test positive. Then

Only a one-in-six chance, despite a "99% accurate" test, because the 5% false-positive rate acts on the 99% of people who are healthy. Ignoring the prior is the base-rate fallacy; the demo above lets you move all three numbers and watch the posterior respond.

Intuition. Bayes' theorem is bookkeeping for evidence: start from how plausible each hypothesis was, multiply by how well it predicts what you saw, renormalize.

Likelihood is not probability

Fix the data $D$ and read $p(D\mid\theta)$ as a function of the parameters: that is the likelihood $\mathcal{L}(\theta)$. It need not integrate to one over $\theta$; it ranks parameter values by how well they explain the data. For i.i.d. data the likelihood is a product and the log-likelihood is a sum, $\ell(\theta)=\sum_{i=1}^{n}\log p(x_i\mid\theta)$, which is numerically stable and easy to differentiate.

Maximum likelihood estimation

Bernoulli. Observe $k$ heads in $n$ flips. $\ell(p)=k\log p+(n-k)\log(1-p)$; setting $\ell'(p)=\dfrac{k}{p}-\dfrac{n-k}{1-p}=0$ gives $\hat p_{\text{MLE}}=k/n$. The sample frequency — reassuring, and derived rather than assumed.

Gaussian. For $x_1,\dots,x_n\sim\mathcal{N}(\mu,\sigma^2)$,

Differentiating in $\mu$: $\frac{1}{\sigma^2}\sum_i(x_i-\mu)=0\Rightarrow\hat\mu=\bar x$, the sample mean. Differentiating in $\sigma^2$: $-\frac{n}{2\sigma^2}+\frac{1}{2\sigma^4}\sum_i(x_i-\mu)^2=0\Rightarrow\hat\sigma^2=\frac1n\sum_i(x_i-\bar x)^2$ — note the $1/n$, which makes this estimator slightly biased; 1.8 Statistics essentials explains the $n-1$ fix.

MAP: adding a prior

Maximum a posteriori estimation maximizes the posterior instead: $\theta_{\text{MAP}}=\arg\max_\theta\big[\log p(D\mid\theta)+\log p(\theta)\big]$ (the evidence does not depend on $\theta$). With a $\mathrm{Beta}(\alpha,\beta)$ prior on the coin, $\hat p_{\text{MAP}}=\dfrac{k+\alpha-1}{n+\alpha+\beta-2}$: pseudo-counts that pull the estimate toward the prior when data are scarce — the same mathematics as Laplace smoothing in naive Bayes (3.7 Naive Bayes and generative vs discriminative models). With a Gaussian prior on weights, $p(\mathbf{w})\propto\exp(-\|\mathbf{w}\|^2/2\tau^2)$, the log-prior is $-\|\mathbf{w}\|^2/2\tau^2+\text{const}$: MAP adds an $\ell_2$ penalty to the loss. Weight decay is a Gaussian prior; a Laplace prior gives $\ell_1$ (3.6 Regularization and generalization). The prior's influence shrinks as $n$ grows, so MAP and MLE agree in the large-data limit.

Negative log-likelihood is the loss

Suppose a model predicts $\hat y=f_\theta(\mathbf{x})$ and we assume a distribution of the observed $y$ around it. Minimizing $-\log p(y\mid\mathbf{x},\theta)$ summed over the data gives:

So "choose a loss" really means "choose a noise model", and the pairing is not optional: squared error assumes Gaussian residuals, cross-entropy assumes a categorical output (1.9 Information theory: entropy, cross-entropy and KL divergence shows the same table from the information-theory side).

The multivariate Gaussian

For $\mathbf{x}\in\mathbb{R}^d$ with mean $\boldsymbol{\mu}$ and covariance $\boldsymbol{\Sigma}$ (symmetric positive definite), the density is the one quoted in 1.6 Probability I: random variables and distributions. The quadratic form $(\mathbf{x}-\boldsymbol{\mu})^\top\boldsymbol{\Sigma}^{-1}(\mathbf{x}-\boldsymbol{\mu})$ is the squared Mahalanobis distance: Euclidean distance after whitening by the covariance. Writing $\boldsymbol{\Sigma}=\mathbf{Q}\boldsymbol{\Lambda}\mathbf{Q}^\top$ (1.3 Eigen-things, SVD and low rank), the level sets are ellipsoids with axes $\mathbf{q}_i$ and semi-axes $\sqrt{\lambda_i}$; $\boldsymbol{\Sigma}^{-1}$ is the precision matrix, whose zeros encode conditional independences. To sample, take $\mathbf{z}\sim\mathcal{N}(\mathbf{0},\mathbf{I})$ and set $\mathbf{x}=\boldsymbol{\mu}+\mathbf{L}\mathbf{z}$ with $\mathbf{L}\mathbf{L}^\top=\boldsymbol{\Sigma}$ (a Cholesky factor) — the reparameterization that VAEs rely on (8.8 Generative models I: VAEs and GANs). The MLE of $\boldsymbol{\Sigma}$ is the sample covariance $\frac1n\sum_i(\mathbf{x}_i-\bar{\mathbf{x}})(\mathbf{x}_i-\bar{\mathbf{x}})^\top$, the matrix PCA diagonalizes (3.12 Dimensionality reduction: PCA, t-SNE and UMAP).

Where this shows up later

• Estimator bias, variance and intervals — 1.8 Statistics essentials; cross-entropy as KL plus entropy — 1.9 Information theory: entropy, cross-entropy and KL divergence.
• Naive Bayes and generative classifiers — 3.7 Naive Bayes and generative vs discriminative models; logistic regression as Bernoulli MLE — 3.3 Logistic regression and softmax classification; weight decay as a Gaussian prior — 3.6 Regularization and generalization; the loss catalog — 4.3 Loss functions catalog.
• The chain rule of probability as the language-model objective — 6.2 Classical NLP and n-gram language models, 7.6 Decoder-only: GPT and autoregressive generation; ELBO and the reparameterization trick — 8.8 Generative models I: VAEs and GANs; Bradley–Terry preference likelihoods in RLHF — 8.3 Alignment: RLHF, PPO, DPO, GRPO and Constitutional AI.