Evaluating LLMs

Why evaluation is the hardest part

Classical ML evaluation (3.5 Evaluating models: metrics, validation and the bias–variance trade-off) assumes a label per example and a metric that compares them. LLM outputs are free text; many answers are acceptable; the same question has a different "right" answer depending on audience and format. Meanwhile, public benchmarks saturate within a year or two and leak into pretraining corpora. The consequence is a rule you will hear from every experienced builder: your own eval set is the asset. Public numbers tell you which models to try; only your eval tells you which to ship.

Intrinsic quality: perplexity

For a held-out token sequence of length $N$,

the exponential of the average cross-entropy (1.9 Information theory: entropy, cross-entropy and KL divergence). Lower is better; it tracks pretraining progress closely and is the quantity scaling laws predict (7.9 Scaling laws and compute-optimal training). But it is only comparable between models with the same tokenizer on the same text, and after instruction tuning and RL it decouples from usefulness: aligned models are often worse language models of the web and far better assistants.

The benchmark landscape

The same benchmark gives different numbers under different prompts (zero- vs few-shot, answer letters vs text, chain-of-thought or not) and decoding settings (8.5 Decoding and sampling). Always read the footnotes of an eval table.

Human preference: arenas

Chatbot arenas (LMArena being the best known) show users two anonymous model responses and record a vote. Votes feed a Bradley–Terry model — the same one used to train reward models in 8.3 Alignment: RLHF, PPO, DPO, GRPO and Constitutional AI — producing Elo-like ratings with confidence intervals. Arenas measure what users prefer on the prompts they bring, which is valuable and also biased: longer, well-formatted, confident answers win more; the user population skews technical; and vendors can target the format. Style-controlled ratings partially correct for length and markdown.

LLM-as-judge

Grading with a strong model is cheap and scalable, and the MT-Bench study found that GPT-4-class judges agreed with human experts about as often as humans agreed with each other (roughly 80%). Known biases and their remedies:

• Position bias (preferring the first answer) — evaluate both orders and average or require agreement.
• Verbosity bias — instruct the judge explicitly, and report length alongside scores.
• Self-preference — a model rates its own style higher; use a different judge model, or several.
• Rubric drift — give a concrete rubric and reference answers; ask for a short justification before the score (CoT grading).
• Calibration — hand-label 50–100 items and measure judge–human agreement before trusting the judge at scale.

Pairwise comparison is more reliable than absolute scoring; absolute scores are more convenient for tracking over time. Many teams do both.

Contamination, saturation and noise

Contamination: test items appear in pretraining data (benchmarks are posted online). Detection uses n-gram overlap, canary strings, or checking whether the model can complete the test item verbatim; the structural fix is fresh, regularly refreshed test sets (LiveBench-style) and private evals.

Noise: treat a benchmark score as an estimate of a success probability $p$ from $n$ trials (1.8 Statistics essentials). The standard error is $\sqrt{p(1-p)/n}$.

Math check. MMLU has about 14,000 questions: at $p=0.85$, $\text{SE} \approx \sqrt{0.85\times0.15/14000} \approx 0.003$, so differences of 0.3 points are within noise. GPQA Diamond has 198 questions: at $p=0.7$, $\text{SE} \approx \sqrt{0.7\times0.3/198}\approx 0.033$ — a 95% interval about $\pm 6.5$ points wide. Two models "3 points apart" on it may be tied.

Add prompt-format variance and sampling variance, and the honest conclusion is that leaderboards resolve tiers, not ranks.

Building your own eval

1. Collect 50–200 real inputs (logs, user interviews, synthetic variations of real cases). Cover the hard and rare cases deliberately.
2. Define success per item: an exact answer, a checker (tests, regex, schema), or a rubric with examples of pass/fail.
3. Automate scoring: deterministic checks first, judges only where necessary — calibrated as above.
4. Run it on every change (prompt, model, retrieval, decoding) and keep the history; a dashboard of per-item results beats a single number.
5. Do error analysis by reading failures and grouping them — this, not the aggregate score, tells you what to fix next (Hamel Husain's "look at your data").
6. Refresh the set as the product changes; keep a held-out split that nobody tunes against.

Tooling: lm-evaluation-harness and HELM for public benchmarks; promptfoo, Inspect, Braintrust and provider eval suites for custom sets; tracing tools for production (9.8 Evaluating and observing LLM apps).

Where this shows up later

• RAG evaluation adds retrieval metrics and faithfulness — 9.8 Evaluating and observing LLM apps; agent evaluation adds trajectories and compounding error — 10.9 Evaluating agents.
• Fine-tuning and prompt changes are only real when the eval says so — 8.2 Fine-tuning and PEFT, 8.4 Prompting as programming.
• Statistics of estimates come from 1.8 Statistics essentials; classification metrics from 3.5 Evaluating models: metrics, validation and the bias–variance trade-off.