1.6 Probability I: random variables and distributions

A classifier does not output an answer, it outputs a distribution over answers; a language model is a distribution over the next token (7.6 Decoder-only: GPT and autoregressive generation); a diffusion model adds and removes Gaussian noise (8.9 Generative models II: Diffusion and flow matching). Probability is the language in which all of that is written. This lesson builds the vocabulary — random variables, PMFs and PDFs, expectation and variance, the standard distributions — and spends extra time on the Gaussian, the distribution that keeps reappearing for reasons you will finally understand.

You'll be able to

State the probability axioms and distinguish discrete (PMF) from continuous (PDF) random variables
Compute expectations, variances and covariances and use their linearity rules
Recognize Bernoulli, Binomial, Categorical, Uniform, Gaussian, Exponential and Poisson distributions and where each appears in ML
Work with the Gaussian density, standardization and the multivariate Gaussian's covariance geometry

Key ideas

Axioms: P ≥ 0, P(Ω) = 1, additivity for disjoint events; everything else (complements, unions) follows.
Discrete variables have PMFs that sum to 1; continuous variables have densities that integrate to 1 and can exceed 1 pointwise; the CDF covers both.
E[X] is a weighted average and is always linear; Var(X) = E[X²] − μ², Var(aX + b) = a²Var(X); variances of independent variables add.

Show 4 more key ideas

Bernoulli/Categorical are classifier output distributions; the Gaussian is the default noise model and prior.
Gaussian density ∝ exp(−½((x − μ)/σ)²): a squared distance in the exponent, which is why Gaussian noise ⇒ squared-error loss.
Multivariate Gaussian level sets are ellipsoids with axes along the covariance's eigenvectors.
Gaussians are everywhere because of the CLT, maximum entropy and closure under sums, linear maps and conditioning.

Suggested path through this lesson

Why probability is the language of learning

Data are noisy, labels are ambiguous, and the future is uncertain. A model that commits to one answer cannot say how sure it is, cannot be calibrated and cannot be combined with other evidence. So models output distributions: logistic regression gives $P(y=1\mid\mathbf{x})$ (3.3 Logistic regression and softmax classification), a softmax layer gives a distribution over classes (4.3 Loss functions catalog), and GPT gives a distribution over 100,000 possible next tokens (7.6 Decoder-only: GPT and autoregressive generation). Even training is probabilistic: minimizing a loss turns out to be maximizing the probability the model assigns to the data (1.7 Probability II: Bayes, independence, MLE and MAP).

Sample spaces, events and the axioms

A sample space $\Omega$ is the set of all outcomes of an experiment; an event is a subset of $\Omega$ . A probability $P$ assigns each event a number such that $P(A)\ge0$ , $P(\Omega)=1$ , and $P(A\cup B)=P(A)+P(B)$ when $A$ and $B$ cannot both happen. Everything else follows: $P(A^c)=1-P(A)$ and, for overlapping events, $P(A\cup B)=P(A)+P(B)-P(A\cap B)$ .

Random variables: PMF, PDF and CDF

A random variable $X$ attaches a number to each outcome. If $X$ takes countably many values, its probability mass function $p(x)=P(X=x)$ satisfies $\sum_x p(x)=1$ . If $X$ is continuous, single values have probability zero and we use a probability density $p(x)$ with

P(a\le X\le b)=\int_a^b p(x)\,dx,\qquad\int_{-\infty}^{\infty}p(x)\,dx=1 .

A density is not a probability — it can exceed 1 — only its integrals are. The cumulative distribution function $F(x)=P(X\le x)$ works for both kinds and is what scipy.stats calls cdf.

Expectation, variance and covariance

The expectation is the probability-weighted average, $\mathbb{E}[X]=\sum_x x\,p(x)$ or $\int x\,p(x)\,dx$ , and for any function $g$ the law of the unconscious statistician gives $\mathbb{E}[g(X)]=\sum_x g(x)p(x)$ — no need to find the distribution of $g(X)$ . Expectation is linear: $\mathbb{E}[aX+bY]=a\mathbb{E}[X]+b\mathbb{E}[Y]$ , always, whether or not $X$ and $Y$ are independent. The variance measures spread,

\operatorname{Var}(X)=\mathbb{E}\big[(X-\mu)^2\big]=\mathbb{E}[X^2]-\mu^2,\qquad \operatorname{Var}(aX+b)=a^2\operatorname{Var}(X),

and the standard deviation $\sigma=\sqrt{\operatorname{Var}(X)}$ has the units of $X$ . For two variables the covariance $\operatorname{Cov}(X,Y)=\mathbb{E}[(X-\mu_X)(Y-\mu_Y)]$ measures linear co-movement and the correlation $\rho=\operatorname{Cov}(X,Y)/(\sigma_X\sigma_Y)\in[-1,1]$ is its scale-free version — the Cauchy–Schwarz inequality of 1.1 Vectors, dot products and norms in probabilistic clothing. For independent variables $\operatorname{Var}(X+Y)=\operatorname{Var}(X)+\operatorname{Var}(Y)$ ; in general you add $2\operatorname{Cov}(X,Y)$ .

Intuition. Expectation is the center of mass of the distribution; variance is its moment of inertia. Linearity of expectation is the single most-used fact in all of probability.

The distributions you will meet

distribution	parameters	PMF or PDF	mean, variance	where in ML
Bernoulli	$p$	$p^x(1-p)^{1-x}$ , $x\in\{0,1\}$	$p$ , $p(1-p)$	binary labels, logistic regression output
Binomial	$n,p$	$\binom{n}{x}p^x(1-p)^{n-x}$	$np$ , $np(1-p)$	number of correct answers in $n$ trials
Categorical	$\boldsymbol{\pi}$	$\pi_k$ for class $k$	—	softmax outputs, next-token distribution
Uniform	$a,b$	$1/(b-a)$ on $[a,b]$	$(a+b)/2$ , $(b-a)^2/12$	random initialization, sampling
Gaussian	$\mu,\sigma^2$	$\dfrac{1}{\sigma\sqrt{2\pi}}\exp\!\Big(-\dfrac{(x-\mu)^2}{2\sigma^2}\Big)$	$\mu$ , $\sigma^2$	noise models, priors, diffusion, initialization
Exponential	$\lambda$	$\lambda e^{-\lambda x}$ , $x\ge0$	$1/\lambda$ , $1/\lambda^2$	waiting times, survival
Poisson	$\lambda$	$\lambda^x e^{-\lambda}/x!$	$\lambda$ , $\lambda$	counts per interval

Bernoulli and Categorical are the output distributions of almost every classifier; the Gaussian is the default model of noise and the default prior over weights.

The Gaussian, up close

Write the density as $p(x)\propto\exp\!\big(-\tfrac12 z^2\big)$ with $z=(x-\mu)/\sigma$ : the exponent is a negative squared distance from the mean in units of $\sigma$ . That is why assuming Gaussian noise leads to squared-error losses (3.2 Linear regression, 1.7 Probability II: Bayes, independence, MLE and MAP). Standardizing, $Z=(X-\mu)/\sigma\sim\mathcal{N}(0,1)$ , lets one table (or norm.cdf) answer every Gaussian question; roughly 68%, 95% and 99.7% of the mass lies within 1, 2 and 3 standard deviations. Sums of independent Gaussians are Gaussian, with means and variances adding. In $d$ dimensions,

p(\mathbf{x})=\frac{1}{(2\pi)^{d/2}|\boldsymbol{\Sigma}|^{1/2}}\exp\!\Big(-\tfrac12(\mathbf{x}-\boldsymbol{\mu})^\top\boldsymbol{\Sigma}^{-1}(\mathbf{x}-\boldsymbol{\mu})\Big),

the level sets are ellipsoids whose axes are the eigenvectors of the covariance $\boldsymbol{\Sigma}$ and whose semi-axis lengths are the square roots of its eigenvalues (1.3 Eigen-things, SVD and low rank); the determinant in the normalizer is the volume factor of 1.2 Matrices as transformations. The full treatment is in 1.7 Probability II: Bayes, independence, MLE and MAP.

Why Gaussians are everywhere

Three reasons. The central limit theorem: a sum of many small independent effects is approximately Gaussian whatever the effects' own distributions, so measurement noise usually is (1.8 Statistics essentials). Maximum entropy: among all distributions with a given mean and variance, the Gaussian assumes the least (1.9 Information theory: entropy, cross-entropy and KL divergence). Convenience: Gaussians stay Gaussian under sums, linear maps, conditioning and marginalizing, which keeps the math closed-form.

Worked example

A biased coin with $P(\text{heads})=0.3$ is $X\sim\mathrm{Bernoulli}(0.3)$ : $\mathbb{E}[X]=0.3$ and $\operatorname{Var}(X)=0.3\cdot0.7=0.21$ . Ten flips give $S\sim\mathrm{Binomial}(10,0.3)$ with mean $3$ and variance $2.1$ ; the chance of no heads at all is $0.7^{10}\approx0.028$ . For heights $X\sim\mathcal{N}(170,7^2)$ in centimetres, $P(X>184)=P(Z>2)\approx0.0228$ — about one person in 44.

from scipy import stats
print(stats.binom(10, 0.3).pmf(0))      # 0.0282
print(stats.norm(170, 7).sf(184))       # 0.0228  (sf = 1 - cdf)

Where this shows up later

Conditioning, Bayes and maximum likelihood — 1.7 Probability II: Bayes, independence, MLE and MAP; estimators and the CLT — 1.8 Statistics essentials; entropy of distributions — 1.9 Information theory: entropy, cross-entropy and KL divergence.
Bernoulli and Categorical outputs — 3.3 Logistic regression and softmax classification, 4.3 Loss functions catalog; Gaussian noise and squared error — 3.2 Linear regression; Gaussian naive Bayes and mixtures — 3.7 Naive Bayes and generative vs discriminative models, 3.11 Clustering: k-means, GMMs/EM, DBSCAN and hierarchical.
Gaussian initialization — 4.6 Initialization and normalization; VAEs and diffusion are built from Gaussians — 8.8 Generative models I: VAEs and GANs, 8.9 Generative models II: Diffusion and flow matching; sampling from the next-token Categorical — 8.5 Decoding and sampling.

Common pitfalls (4)

Reading a density value as a probability — p(x) = 3 is fine; only integrals are probabilities.
Assuming Var(X + Y) = Var(X) + Var(Y) without independence; the covariance term can be large.
Confusing standard deviation (units of X) with variance (squared units) when reporting results.
Thinking 'the data is Gaussian' is an observation; it is a modeling assumption that should be checked (plot a histogram or QQ plot).

Recap: all 7 key points

Axioms: P ≥ 0, P(Ω) = 1, additivity for disjoint events; everything else (complements, unions) follows.
Discrete variables have PMFs that sum to 1; continuous variables have densities that integrate to 1 and can exceed 1 pointwise; the CDF covers both.
E[X] is a weighted average and is always linear; Var(X) = E[X²] − μ², Var(aX + b) = a²Var(X); variances of independent variables add.
Bernoulli/Categorical are classifier output distributions; the Gaussian is the default noise model and prior.
Gaussian density ∝ exp(−½((x − μ)/σ)²): a squared distance in the exponent, which is why Gaussian noise ⇒ squared-error loss.
Multivariate Gaussian level sets are ellipsoids with axes along the covariance's eigenvectors.
Gaussians are everywhere because of the CLT, maximum entropy and closure under sums, linear maps and conditioning.

Watch

Lecture 1: Probability and Counting | Statistics 110

Joe Blitzstein (Harvard University) · 50 min · open on YouTube ↗

Sample spaces, events and the naive definition — a rigorous but friendly start to the course this lesson follows.

3 videos

Read

Books

Mathematics for Machine Learning — Sections 6.1–6.5 (probability space, random variables, sum and product rules, summary statistics, Gaussian distribution) ↗Deisenroth, Faisal & Ong · 2020Section 6.5 derives the marginal and conditional of a multivariate Gaussian — the closure properties used throughout the path.Probabilistic Machine Learning: An Introduction — Chapter 2 (Probability: univariate models) ↗Kevin Murphy · 2022The ML-flavored treatment of the same distributions, with the Gaussian as the running example.

Interactive

Seeing Theory — Basic Probability, Compound Probability, Probability Distributions ↗Daniel Kunin et al. (Brown University) · 2018Interactive visualizations of expectation, variance and the common distributions; play with the parameters.

Courses

Statistics 110: Probability (course site with the free textbook by Blitzstein & Hwang) ↗Joe Blitzstein · 2019Chapters 1–5 of the book match this lesson; the strategic practice problems are excellent.Khan Academy — Statistics and probability ↗Khan Academy · 2024Drill on random variables, expected value and the normal distribution if you need repetitions.

Try it live

Interactive demo

runs in your browser

Loading the demo…

Exercises

By hand: moments and probabilities
For a die, compute $\mathbb{E}[X]$ , $\mathbb{E}[X^2]$ and $\operatorname{Var}(X)$ . For $S$ the sum of two dice, find $P(S=7)$ and $\operatorname{Var}(S)$ using independence. For $X\sim\mathcal{N}(100,15^2)$ (IQ scores), find $P(X>130)$ and the value $x$ with $P(X<x)=0.9$ . Expected: $\operatorname{Var}(X)=35/12$ , $P(S=7)=1/6$ , $\operatorname{Var}(S)=35/6$ , $P(X>130)\approx0.0228$ , $x\approx119.2$ .
NumPy: simulate and compare
Draw 100,000 samples from Bernoulli(0.3), Binomial(10, 0.3) and $\mathcal{N}(170,7^2)$ with np.random.default_rng. Compare the sample mean and variance with the formulas, and overlay histograms with the PMF/PDF from scipy.stats. Expected: agreement to two decimals; the empirical $P(X>184)$ is near 0.0228.
Experiment: variance of sums and the CLT
Sample pairs $(X,Y)$ that are independent and pairs that are correlated (generate $Y=X+\text{noise}$ ), and compare $\operatorname{Var}(X+Y)$ with $\operatorname{Var}(X)+\operatorname{Var}(Y)$ . Then average $n\in\{1,2,10,100\}$ uniform random numbers and plot the histograms. Expected: the independence case matches; the averages look Gaussian by $n=10$ with variance shrinking like $1/n$ .

Quiz

Question 1 of 6

Which statement about a probability density p(x) is true?

Flashcards

Card 1 of 8

These cards are now in your spaced-repetition queue. Review all due cards →

Scored 80% or more and reviewed the cards? You're done here.