Evaluating and observing LLM apps

Evals are the product

A RAG system is a pile of choices — chunk size, embedding model, $k$, reranker, prompt, model, temperature — and every choice can be changed in a minute. Without a measurement you are changing things blind, and prompts "feel better" to whoever edited them last. Hamel Husain's framing has become the standard: Level 1 — unit tests and assertions (cheap, deterministic: did the JSON parse, are the citations valid, does the answer avoid banned phrases); Level 2 — human and model-based grading of outputs; Level 3 — A/B tests on real users. Most teams need Level 1 in continuous integration and Level 2 on every meaningful change, with Level 3 reserved for product decisions. The loop that produces good systems is: look at real traces → write down how they fail → turn the failures into evals → fix the biggest bucket → repeat.

Golden datasets

A golden set is a list of (question, reference answer, relevant chunk ids, metadata) — 50 items is a start, a few hundred is comfortable. Sources in order of value: real user questions from logs, with answers written or approved by experts; expert-written questions covering the domain's hard cases; synthetic questions generated by an LLM from sampled chunks, filtered by a second pass and spot-checked, because they skew easy and lexically close to the text. Cover the failure modes you already know (unanswerable questions, multi-hop, tables, ambiguous phrasing), version the set in git, add every production failure to it, and keep a held-out slice you never tune on (1.8 Statistics essentials, 3.5 Evaluating models: metrics, validation and the bias–variance trade-off).

Separate retrieval metrics from end-to-end metrics

Measure the retriever with recall@$k$, MRR and nDCG (9.5 Retrieval quality: hybrid search, reranking, query transforms and metrics) and the whole system with answer-level metrics. When an answer is wrong, the retrieval metrics tell you where: if the relevant chunk was not in the top-$k$, the fix is in 9.4 Chunking and ingestion and 9.5 Retrieval quality: hybrid search, reranking, query transforms and metrics; if it was there and the model ignored or contradicted it, the fix is in prompting, context size or the model (9.6 Generation side: context assembly, citations, structured outputs and tool calling).

RAGAS-style metrics, defined

The RAGAS framework (2023) popularized reference-free and reference-based metrics computed by an LLM:

• Faithfulness $= \dfrac{\#\text{ answer claims supported by the retrieved context}}{\#\text{ answer claims}}$. The judge splits the answer into atomic claims and checks each against the context. Reference-free; the core hallucination metric.
• Answer (response) relevancy: generate $n$ questions that the answer would be answering, embed them, and take the mean cosine similarity to the original question. Penalizes evasive, incomplete or rambling answers. Reference-free.
• Context precision@K $= \dfrac{\sum_{k=1}^{K} \text{precision@}k \cdot v_k}{\#\text{ relevant chunks in the top } K}$, where $v_k \in \{0,1\}$ says whether the chunk at rank $k$ is relevant (judged against the reference answer or labels). Rewards ranking relevant chunks high.
• Context recall $= \dfrac{\#\text{ reference-answer claims attributable to the retrieved context}}{\#\text{ reference-answer claims}}$. Did retrieval bring in what was needed? Requires a reference answer.
• Answer correctness: a weighted combination of factual overlap between answer and reference (an F1 over matched, missing and extra claims) and semantic similarity.

Math check. Answer: "The device resets by holding power for 10 s [1]. It then reboots twice [1]. Support is open 24/7." Context supports the first two claims, not the third ⇒ faithfulness $= 2/3$. If the reference answer has two claims (hold power 10 s; reboots twice) and both appear in the context ⇒ context recall $= 1$. Retrieval returned $[\text{irrelevant}, \text{relevant}, \text{relevant}]$: precision@2 $= 1/2$, precision@3 $= 2/3$, so context precision@3 $= (0 + \tfrac12 + \tfrac23)/2 \approx 0.58$ — the relevant chunks were there but ranked below a distractor.

Every one of these is itself an LLM judgment and inherits judge noise; treat them as directional signals with error bars, not ground truth.

LLM-as-judge: design and calibration

A judge prompt states the task, the inputs (question, context, answer, optional reference), a rubric, and asks for reasoning then a verdict. Prefer binary pass/fail per criterion (faithful? complete? correct format?) over 1–5 scales — binary labels are more consistent across judges and easier to act on. For comparing two systems use pairwise judgments and swap the order to cancel position bias; also watch for verbosity bias (longer answers win) and self-preference (a model favouring its own style). Calibrate: have humans label 50–100 items, measure agreement with the judge (percent agreement and Cohen's $\kappa$), fix the rubric until agreement is high, then trust the judge on thousands. Use a strong model as the judge even when a cheap model serves production (8.7 Evaluating LLMs).

You are grading a support assistant. Given the QUESTION, the SOURCES it was shown
and its ANSWER, decide:
1. FAITHFUL: every factual claim in ANSWER is supported by SOURCES (yes/no).
2. COMPLETE: ANSWER addresses every part of QUESTION that SOURCES can answer (yes/no).
3. CITED: every claim carries a correct source number (yes/no).
First list each claim with the source that supports it, or NONE. Then output JSON:
{"faithful": true|false, "complete": true|false, "cited": true|false, "notes": "..."}

Tracing, regression tests and observability

Instrument the pipeline with traces: one trace per request, with spans for query rewriting, retrieval (which chunks, which scores), reranking, prompt assembly (token counts), generation (model, latency, tokens, cost) and post-processing. Tools such as LangSmith, Langfuse and Arize Phoenix store traces, let you attach eval scores and human feedback, and turn production traces into datasets. Run the eval suite in CI whenever a prompt, model or index changes, and block merges on regressions. In production, sample traces for human review, collect thumbs-up/down as weak labels, and chart p50/p95 latency, cost per request and eval scores per version. Redact personal data before logging (9.9 Production: latency, cost, caching, guardrails, security and privacy).

Online experiments

A/B tests on real traffic measure what offline evals cannot: task completion, deflection of support tickets, follow-up rate, user ratings. They need enough traffic for statistical power (1.8 Statistics essentials) and guardrail metrics (error rate, cost) so a "better" variant does not quietly double spending.

The actual skill: error analysis

Open fifty traces. For each, write one line: what went wrong and at which stage. Cluster the lines; you will find three or four buckets (retrieval misses on acronyms, chunks cut mid-table, over-long answers, false refusals). Fix the largest bucket, re-run the evals, repeat. Teams that do this weekly ship systems that work; teams that only watch aggregate scores do not.

Where this shows up later

• Retrieval metrics — 9.5 Retrieval quality: hybrid search, reranking, query transforms and metrics; production monitoring, cost and privacy of logs — 9.9 Production: latency, cost, caching, guardrails, security and privacy.
• Agent evaluation extends this to multi-step trajectories — 10.9 Evaluating agents; LLM benchmarks and judge biases — 8.7 Evaluating LLMs; statistics of noisy measurements — 1.8 Statistics essentials.