Statistics essentials

Population, sample, estimator

A population quantity $\theta$ (the true accuracy of a model on all possible inputs, the true mean of a distribution) is unknowable; you observe a sample of $n$ draws and compute an estimator $\hat\theta$ — a function of the data. Because the sample is random, so is $\hat\theta$: repeat the experiment and you get a different number. The distribution of $\hat\theta$ across hypothetical repetitions is its sampling distribution, and everything in this lesson is about describing it.

Bias and variance of an estimator

Two properties summarize an estimator. Its bias is $\mathbb{E}[\hat\theta]-\theta$ — is it right on average? Its variance is $\operatorname{Var}(\hat\theta)$ — how much does it wobble? They combine into the mean squared error:

Math check. Write $\hat\theta-\theta=(\hat\theta-\mathbb{E}\hat\theta)+(\mathbb{E}\hat\theta-\theta)$ and square. The cross term is $2(\mathbb{E}\hat\theta-\theta)\,\mathbb{E}[\hat\theta-\mathbb{E}\hat\theta]=0$. ✓ The same decomposition, applied to a model's predictions, is the bias–variance trade-off of 3.5 Evaluating models: metrics, validation and the bias–variance trade-off.

The sample mean $\bar x=\frac1n\sum_i x_i$ is unbiased ($\mathbb{E}\bar x=\mu$ by linearity) with variance $\sigma^2/n$ (independent variances add, then scale by $1/n^2$). The MLE of the variance from 1.7 Probability II: Bayes, independence, MLE and MAP, $\hat\sigma^2=\frac1n\sum_i(x_i-\bar x)^2$, is biased: because $\bar x$ was fit to the same data, deviations from it are systematically smaller than deviations from $\mu$. Precisely, $\sum_i(x_i-\bar x)^2=\sum_i(x_i-\mu)^2-n(\bar x-\mu)^2$; taking expectations gives $n\sigma^2-n\cdot\sigma^2/n=(n-1)\sigma^2$. Hence $\mathbb{E}[\hat\sigma^2]=\frac{n-1}{n}\sigma^2$, and dividing by $n-1$ instead (Bessel's correction, ddof=1) makes the estimator unbiased. One degree of freedom was spent estimating the mean.

Consistency, the law of large numbers and the CLT

An estimator is consistent if it converges to $\theta$ as $n\to\infty$. The law of large numbers says the sample mean does: $\bar x\to\mu$. The central limit theorem says how fast and in what shape: for i.i.d. data with finite variance,

whatever the distribution of the individual $x_i$ — coin flips, token losses, click-through indicators. The quantity $\sigma/\sqrt{n}$ (estimated by $s/\sqrt{n}$) is the standard error. The square root is the cruel part: to halve your error bars you need four times the data.

Confidence intervals

Because $\bar x$ is approximately Gaussian, the interval $\bar x\pm1.96\,s/\sqrt{n}$ captures $\mu$ in about 95% of repeated experiments: that is a 95% confidence interval. Its interpretation is a statement about the procedure, not about this particular interval — it is not "a 95% probability that $\mu$ lies here" (that would be a Bayesian credible interval, 1.7 Probability II: Bayes, independence, MLE and MAP). For a proportion such as accuracy, $x_i\in\{0,1\}$, $\sigma^2=p(1-p)$ and

Worked example. A model scores 80% on a 500-question benchmark: $\text{SE}=\sqrt{0.8\cdot0.2/500}=0.0179$, so the 95% interval is roughly $[76.5\%,\ 83.5\%]$. A competitor at 82% on the same 500 questions is well inside that interval. To decide between them you compare the difference: on independent test sets $\text{SE}_{\text{diff}}=\sqrt{\text{SE}_1^2+\text{SE}_2^2}\approx2.5\%$, so a 2-point gap is noise. On the same questions you should use the per-item differences (a paired comparison), whose standard error is often much smaller because both models get the easy questions right and the hard ones wrong together.

Hypothesis tests and p-values

A hypothesis test starts from a null hypothesis $H_0$ ("the two models have equal accuracy"), computes a test statistic, and reports a p-value: the probability, if $H_0$ were true, of a result at least as extreme as the one observed. A small p-value (conventionally below 0.05) is evidence against $H_0$. Three things a p-value is not: the probability that $H_0$ is true, the probability that the result is a fluke, or a measure of effect size — a tiny improvement becomes "significant" with enough data. Beware multiple comparisons: evaluate on 20 benchmarks and you expect one to look significant at the 0.05 level by chance alone. For paired classification results, McNemar's test and the paired bootstrap below are the right tools.

Intuition. A p-value answers "how surprising is my data if nothing is going on?" — not "is something going on?". Both questions matter; only the first is being answered.

The bootstrap and Monte Carlo

When the statistic is complicated (F1, nDCG, a median, a win rate), skip the formulas and bootstrap: resample the $n$ data points with replacement $B$ times (say $B=1000$), recompute the statistic each time, and read the 2.5th and 97.5th percentiles as a confidence interval. The sampling distribution is simulated instead of derived. For comparing two models, resample pairs of outcomes and bootstrap the difference — if the interval excludes zero, the gap is real at that confidence.

import numpy as np
rng = np.random.default_rng(0)
def bootstrap_ci(values, stat=np.mean, B=1000, alpha=0.05):
    n = len(values)
    boots = [stat(values[rng.integers(0, n, n)]) for _ in range(B)]
    return np.percentile(boots, [100 * alpha / 2, 100 * (1 - alpha / 2)])
correct = rng.random(500) < 0.8        # 500 right/wrong outcomes
print(bootstrap_ci(correct))           # roughly [0.765, 0.835]

The bootstrap is one instance of Monte Carlo estimation: approximate an expectation by an average over random samples. Its error shrinks like $1/\sqrt{N}$ in the number of samples — the CLT again — which is why variance-reduction tricks matter so much in reinforcement learning (4.11 Reinforcement learning essentials) and variational inference (8.8 Generative models I: VAEs and GANs).

Statistics in machine-learning evaluation

Three habits follow. Size your test set: for accuracy near 0.8 and a desired half-width of 1 percentage point you need $n\approx1.96^2\cdot0.16/0.01^2\approx6100$ examples; a 100-example eval set cannot resolve differences below about 8 points. Report variance across seeds: train with at least three random seeds and report mean ± standard deviation; many published "improvements" are within seed noise. Treat leaderboards statistically: differences smaller than the confidence interval are ties (8.7 Evaluating LLMs). Cross-validation (3.5 Evaluating models: metrics, validation and the bias–variance trade-off) is a resampling scheme that estimates both the mean and the variance of a model's performance, and the learning-rate sweeps of 4.8 Training dynamics and debugging are hypothesis tests you run informally — know what they can and cannot conclude.

Where this shows up later

• The bias–variance decomposition of a model's error — 3.5 Evaluating models: metrics, validation and the bias–variance trade-off; cross-validation and model selection — 3.5 Evaluating models: metrics, validation and the bias–variance trade-off, 3.6 Regularization and generalization.
• Benchmark error bars, LLM-as-judge noise and eval-set sizing — 8.7 Evaluating LLMs; agent evals with pass@k and compounding error — 10.9 Evaluating agents.
• Monte Carlo estimates and variance reduction — 4.11 Reinforcement learning essentials, 8.8 Generative models I: VAEs and GANs; retrieval metrics with confidence intervals — 9.8 Evaluating and observing LLM apps.