Evals: Measuring What You Ship

Why this is the highest-leverage lesson

Every other lesson describes a change you might make. This one describes how you know whether it helped.

Without evals, an AI system is tuned by anecdote. Someone changes a prompt, three outputs look better, it ships. Two weeks later a metric moves and nobody can bisect it, because there is nothing to bisect against.

The uncomfortable part: evals feel like overhead right up until the first time they catch something, after which nobody argues.

What an eval actually is

Three components:

1. Cases — inputs representative of real use.
2. Expectations — what correct means for each.
3. A grader — a function from output to score.

The engineering discipline is in making all three trustworthy.

Building the set

Size. 50–200 cases for most systems. Below ~30, noise dominates. Above a few hundred, the cost of running it starts to discourage running it, which is worse than a smaller set.

Composition matters far more than size:

Two rows carry disproportionate weight.

Refusal cases are the most commonly missing and the most valuable. Systems regress in both directions — becoming over-cautious after a safety-flavoured prompt edit, or becoming confidently fabricating after a helpfulness edit. Neither shows up in an accuracy score computed only over answerable questions.

Regression cases are how the set earns its keep. Every production bug becomes a permanent case. This is the single habit that compounds most: after a year, your eval set encodes everything your system has ever got wrong, and none of it can silently return.

Sourcing. Real traffic beats invented cases. Sample production logs, stratify across intents, and label. Where privacy prevents this, have domain experts write cases from real transcripts.

Graders

Match the grader to the output shape. Getting this wrong is worse than not measuring, because it produces a confident wrong number.

Prefer deterministic graders wherever the output allows. Every deterministic grader you can use is one fewer source of noise. Teams reach for model graders far too early — a surprising amount of open-ended-looking output can be reduced to checkable assertions.

Model-as-judge, done properly

For genuinely open-ended output it is usually the only affordable option. It works, and the biases are documented rather than folklore:

• Length bias — longer answers score higher, all else equal.
• Position bias — in pairwise comparison, presentation order shifts the verdict.
• Self-preference — judges favour output stylistically like their own.
• Confidence bias — assertive wrongness beats hedged correctness.

Five practices that make it usable:

1. Rubrics, not vibes. “Score 1–5 on whether every factual claim is supported by the provided context” is gradeable. “Rate the quality” is not. Define what each point on the scale means.

2. Reason, then score. Have the judge state its justification before emitting a number. The reasoning is the audit trail, and it improves the score.

3. Calibrate against humans. Label 30–50 cases yourself. Compare with the judge. If agreement is poor, fix the rubric before scaling. Skipping this step is how teams end up confidently optimising the wrong objective.

4. Counter position bias in pairwise comparisons by running both orders and averaging, or use single-answer grading against a rubric instead.

5. Recalibrate when the judge model changes. A judge upgrade silently changes your measuring instrument. Re-run the human agreement check.

Component-level evaluation

Composite systems need per-component metrics, because an end-to-end score cannot direct the work.

For RAG:

retrieval  → recall@k, precision@k, MRR
generation → faithfulness, relevance, citation accuracy
end-to-end → answer correctness, refusal accuracy

An end-to-end failure is ambiguous: wrong passage fetched, or right passage used badly? Those need entirely different fixes. Instrument both or you will tune the wrong half — which, as lesson 5 argues, is the commonest way RAG projects lose months.

For agents, evaluate trajectories rather than final answers: completion rate, turns to completion, tool selection accuracy, recovery rate after an error, and boundary violations. That last should be zero and should be alerted on rather than scored.

Noise, and how to handle it

Sampling means the same input yields different outputs. Ignoring this produces two opposite errors: shipping changes that did nothing, and reverting changes that helped.

Run each case several times — three to five — and use the mean. Expensive, and the only honest option when differences are small.

Know your noise floor. Run the identical configuration twice and record the difference. That number is the minimum movement worth interpreting. Most teams have never measured it and consequently over-read every result.

Use temperature 0 for graders, even when the system under test samples. You want the measuring instrument stable, whatever the subject does.

Report distributions, not just means. A change that improves the average while introducing occasional catastrophic failures is a regression, and a mean hides it.

Running evals in CI

The tension: evals cost money and time, and CI must stay fast.

A workable split:

The gating policy matters more than the tooling. Blocking on any score decrease means noise fails the build, and within a month someone disables it. The policy that survives:

• Hard block on regression cases. Zero tolerance. A previously-fixed bug returning is never acceptable and is never noise.
• Threshold gate on the aggregate. Below the floor, block. Small movements within the noise band, report only.

That last row of the table deserves emphasis. A model change is not a version bump. Defaults around reasoning, sampling and tokenisation differ between generations. Re-baseline score, latency and cost before rolling out — this is exactly the trap lesson 2 describes, and evals are how you catch it.

The limits, stated honestly

• An eval set measures what is in it. It says nothing about cases you did not imagine.
• It goes stale. Usage shifts; the set must be refreshed from production traffic.
• Overfitting is real. Tune long enough against a fixed set and you optimise for it rather than for users. Keep a held-out set you consult rarely.
• A high score is not a working system. It is evidence, not proof.

None of these argue against evals. They argue for treating the set as a living artefact that grows whenever reality surprises you.

The workflow

1. Build 50 cases with expectations, including refusals and regressions
2. Choose the cheapest valid grader per case type
3. Calibrate any model grader against human labels
4. Measure the noise floor: same config, twice
5. Baseline. Record score, latency, cost
6. Change ONE thing. Re-run. Compare against the noise floor
7. Every production bug becomes a permanent case
8. Re-baseline completely on any model change

Try this: Run your current configuration against your eval set twice, changing nothing. The difference between those two runs is your noise floor — and if it is larger than the improvement you were about to ship, you have just learned that you did not measure anything.