Evaluating agents

Why agent evaluation is hard

A chatbot answer can be graded in isolation. An agent run is a trajectory: dozens of model calls and tool results, each depending on the last, in an environment that the agent itself changes. Two runs of the same task differ; a run can reach the right end state by a wrong path or the wrong end state by a reasonable one; grading requires inspecting the world (files, databases, web pages) rather than a string; and each trial costs many calls, so you cannot afford thousands of them. On top of that, frontier models sometimes recognize that they are being evaluated — Anthropic documented this "eval awareness" in 2026 — and environments are flaky enough that a measurable share of failures is infrastructure noise, not model error.

Vocabulary

Anthropic's Demystifying evals for AI agents (January 2026) fixes the terms most teams now use:

• Task — a prompt plus an environment plus success criteria. Trial — one attempt at a task. Transcript/trajectory — everything that happened in the trial. Outcome — the final state of the environment, which is usually what you grade.
• Graders: code-based (deterministic checks: tests pass, a row exists, a file matches, a regex holds — cheap, reliable, limited to what you can specify), model-based (an LLM scores against a rubric — flexible, needs calibration against humans, 9.8 Evaluating and observing LLM apps), and human (gold standard, expensive, used to calibrate the other two).
• Capability evals ask "can it do this at all?" and should be hard; regression evals ask "did we break anything?" and should be stable and near-saturated.

Grade end states, not paths, unless the path is the point: a grader that insists on one particular sequence of tool calls punishes valid alternatives and over-specifies behavior. Give partial credit through separate checks rather than one brittle pass/fail where possible.

pass@k versus pass^k

Both start from the probability $p$ that one trial succeeds.

pass@k is the probability that at least one of $k$ trials succeeds: $1-(1-p)^k$. It measures what you can achieve if you can recognize success — with a verifier you can run $k$ attempts and keep the winner (10.3 Reasoning and planning for agents). From $n$ samples of which $c$ are correct, the unbiased estimator (Chen et al., 2021) is

pass^k (introduced with τ-bench) is the probability that all $k$ trials succeed: $p^k$. It measures consistency — what a user experiences when they rely on the agent $k$ times without a verifier.

Math check. With $p = 0.7$ and $k = 3$: pass@3 $= 1 - 0.3^3 = 0.973$, while pass^3 $= 0.7^3 = 0.343$. Same agent, 97% or 34%, depending on whether someone can tell success from failure. Report both, and say which you mean.

Compounding error

If a task needs $n$ steps and each succeeds independently with probability $p$, the end-to-end success is

To finish a 20-step task 90% of the time you need $p = 0.9^{1/20} \approx 0.9947$ per step — an accuracy no prompt tweak delivers. The levers are structural:

1. Fewer steps — better tools that do more per call (10.2 Tool use and function calling), code execution, good decomposition.
2. Verification with retry. If a verifier catches a failed step and you allow $r$ retries, the effective per-step success becomes $p' = 1-(1-p)^{r+1}$ (assuming a reliable verifier). With $p = 0.9$ and $r = 2$: $p' = 1 - 0.1^3 = 0.999$, and $S_{20} \approx 0.98$. This is why coding agents work (10.8 Coding agents and computer use).
3. Checkpoints and human gates that bound how far an error propagates (10.11 Building production agents).
4. Shorter horizons — split long tasks into independently verifiable chunks.

Steps are not truly independent (a confused agent stays confused), so $p^n$ is a model, not a law — but it is the right mental model, and the interactive demo lets you feel it.

The benchmark landscape

Read every score with four questions. Saturation: once the frontier is near the ceiling, gains are noise. Contamination: have the tasks leaked into training data? Infrastructure noise: Anthropic's 2026 analysis of agentic coding evals showed that flaky environments and timeouts can account for a meaningful fraction of "failures", so compare systems under identical infrastructure and repeated trials. Eval awareness: a model that behaves differently when it suspects a test tells you little about deployment. And remember the statistics of 1.8 Statistics essentials: with 100 tasks, a 3-point difference is usually within the confidence interval.

Building your own harness

1. Collect 20–50 tasks from real failures and real requests. Start small and early — Anthropic's research-system team evaluated from about 20 queries onward (10.6 Multi-agent systems). Grow to hundreds as the system matures.
2. Write success criteria as checks on the end state (and on hard constraints: "did not send email", "cost under budget"). Prefer code-based graders; add model-based rubrics for quality, calibrated against a human-labeled subset (9.8 Evaluating and observing LLM apps).
3. Run $k$ trials per task (3–5) and report pass@k, pass^k and mean success; track cost per successful task, steps, latency and tool-error rates.
4. Store full trajectories and build a viewer; most insights come from reading failures and classifying them (the MAST categories from 10.6 Multi-agent systems work for single agents too).
5. Separate capability and regression suites; run regressions in CI on every prompt, tool or model change.
6. Audit the graders periodically: a grader that passes everything or fails valid solutions silently corrupts every decision downstream.

Where this shows up later

• The harness becomes a production regression suite with tracing in 10.11 Building production agents; red-team tasks for injection join it in 10.10 Agent safety and security.
• Your capstone in 10.12 Project: build a research agent with tools, an MCP server and an eval harness ships with a 20-task eval and cost tracking.
• Benchmark literacy returns in system-design interviews (11.4 ML and LLM system design and interviews).