Agent evals that catch regressions

Every agent we ship gets worse at something eventually. A model updates. A prompt gets a helpful edit. A tool changes its error format. None of these shows up in a unit test, and all of them show up in a customer conversation a week later.

So every agent we ship also ships with an eval harness. It runs on every change to the agent and on a schedule when nothing changes, because the model underneath can change without us. This playbook is what goes in the harness, how it scores, and what we do when it turns red.

What an eval is for

An eval is not a benchmark. We are not trying to know how good the agent is in general. We are trying to know whether this version is worse than the last version on the things this partner cares about. That is a narrower question and a much easier one to answer reliably.

Narrow means the golden set is small, specific, and drawn from real traffic. It does not mean the set is easy. The best cases are the ones the agent got wrong once. A set that passes at a hundred percent on the first run is a set that is not testing anything.

The three layers

• Unit evals: single prompt, single expected behavior. Does the router classify this ticket as a refund? Fast, cheap, hundreds of them, run on every commit.
• Trajectory evals: a full task with tools. Did the agent call the right tools in a sensible order and stop when it should? Dozens of them, run on every pull request.
• Outcome evals: end to end against a held-out sample of real cases, scored against the human result. Run nightly and before any release to a wider audience.

Each layer catches a different failure. Unit evals catch prompt edits that break classification. Trajectory evals catch tool changes and runaway loops. Outcome evals catch the slow drift that neither of the others can see because it lives in the model, not the code.

Building the golden set

The golden set starts on day one with the cases in the spec and grows from production. Every time a person edits an agent's draft, the original input and the human answer become a candidate. Every escalation becomes a candidate. Every complaint becomes a candidate with a high weight.

We review candidates weekly, keep the ones that represent a behavior worth protecting, and tag them by intent and by failure mode. Two to four hundred cases is enough for most agents. Bigger sets are slower and not more useful, because the cases stop being distinct and the run stops being something anyone reads.

Scoring

Three kinds of check: deterministic ones where the answer is binary, structural ones about tool use, and model-graded ones with a pinned rubric and a pinned grader model. The grader never floats. If we upgrade the grader we re-baseline every case first and commit the new baseline with the upgrade.

type Case = {  id: string;  intent: string;  input: string;  expected: { answer?: string; tools?: string[]; handoff?: boolean };  weight: number;};export async function score(c: Case, run: AgentRun): Promise<Score> {  const checks = [    exactHandoff(c, run), // hand-off decisions are binary    toolSequence(c, run), // order and count, not exact arguments    await judged(c, run, rubric), // model-graded, rubric and grader pinned    citesSource(run), // no source, no credit  ];  const passed = checks.filter((x) => x.pass).length;  return {    id: c.id,    pass: passed === checks.length,    weighted: (c.weight * passed) / checks.length,    notes: checks.flatMap((x) => x.notes),  };}

Running it

The harness runs on every pull request that touches the agent, its prompts, its tools, or its retrieval index. It also runs nightly against the same set with the same seed. A nightly failure with no diff means the model or a dependency moved. That is the case teams forget to plan for, and it is the one that hurts most because nobody is looking.

eval:  agent: support-triage  set: evals/golden-v14.jsonl  grader: pinned/judge-2026-06  seed: 1337  thresholds:    pass_rate: 0.94          # block release below this    per_intent_min: 0.85     # any single intent below this blocks too    handoff_precision: 0.98  # never miss an always-human case  schedule: nightly  on_fail: page_owner

What a regression looks like

A regression is rarely a global drop. It is one intent going from ninety-five to seventy while everything else is flat. Per-intent thresholds catch it; a single aggregate number hides it. The other common shape is a change in hand-off behavior: the agent gets more confident and stops routing cases it used to route. Hand-off precision gets its own threshold and its own alert for that reason.

When the harness goes red, the first move is the diff. If there is no diff, we pin the previous model version and open an issue. If there is a diff, the change does not merge until the failing cases are either fixed or explicitly re-baselined with a reason in the commit message that a reviewer has to read.

Rules

• Every prompt change is a pull request. Prompts are code.
• Pin the grader. A floating judge makes every result meaningless.
• Thresholds per intent, not just overall.
• Golden cases come from production, not from imagination.
• Re-baselining requires a written reason. The model got better is not one.
• The harness ships to the partner with the agent. You can run it without us.

The support-agent playbook describes the first week of an agent's life. This is what keeps it alive after that. Every agent in our library ships with a harness in this shape, and the config file above is the real one with the partner's numbers removed.