Judging AI Output: Your First Evals

The problem with judging by eye

Here is how prompt tuning usually goes. You change something, run it a few times, the answers look better, you ship it. A fortnight later something is subtly worse and nobody can say when it broke.

Two things make eyeballing unreliable, and neither is about carelessness:

Output is sampled. Run the same prompt five times and you get five different answers. Three good ones after a change might be the change, or might be the three you would have got anyway.

You are looking at what you changed. You tightened the tone instruction, so you read the tone. You do not notice that it stopped including the reference number, because you were not looking there.

Evaluation — “evals” — is the fix. It is the highest-leverage practice in applied AI and the one most often skipped, because it feels like overhead until the first time it saves you.

An eval is three things

1. A set of inputs representative of real use.
2. A definition of correct for each one.
3. A grader that scores output against that definition.

Then you run it whenever anything changes: prompt, model, retrieval, temperature. You get a number. Numbers you can compare; impressions you cannot.

Build the set first

Twenty cases beats a thousand, if the twenty are the right ones. Assemble them like this:

Ten ordinary cases. The everyday requests. These catch catastrophic breakage.

Ten hard cases. This is where the value is:

• Genuine ambiguity where you have a policy — mixed sentiment, partial matches.
• Things it should refuse — out of scope, missing information, unanswerable.
• Adversarial input — someone trying to talk it out of its instructions.
• Anything it has already got wrong. Every bug becomes a permanent test case. This one habit compounds more than any other.

A spreadsheet is a perfectly good format. Input in one column, expected output in the next, notes in a third. Do not build infrastructure first.

Choosing a grader

Match the grader to the shape of the output. Using the wrong one is worse than not measuring.

That last row deserves emphasis. In a RAG system, grade the two halves separately. If you only score the final answer, you cannot tell whether a failure came from fetching the wrong passage or from writing a bad answer over the right one — and those have completely different fixes.

Model-as-judge, honestly

For open-ended output, using a strong model to grade is often the only practical option. It works, with real caveats that are documented rather than folklore:

• Length bias. Longer answers score higher, other things being equal.
• Self-preference. Judges favour output resembling their own style.
• Position bias. In A-versus-B comparisons, the order of presentation shifts the verdict.
• Confidence bias. Assertive wrongness outscores hedged correctness.

Three things make it usable anyway:

1. Give the judge a rubric, not a vibe. “Score 1–5 on whether every claim is supported by the provided context” is gradeable. “Rate the quality” is not.
2. Ask for a judgement, not a number in isolation. Have it state the reason, then the score.
3. Calibrate against humans. Grade thirty examples yourself, compare with the judge, and check they broadly agree before trusting it on three hundred.

The workflow

1. Write 20 cases with expected outputs
2. Run the current prompt → record the baseline score
3. Change ONE thing
4. Re-run the full set
5. Compare. Better → keep. Worse or flat → revert.
6. Every production bug becomes case 21, 22, 23...

Step 3 is the one people skip. Change three things at once and a flat score tells you nothing — two may have helped and one hurt.

Step 5 needs judgement about noise. If your set is small and the model is sampling, a one-case difference is probably variance. Look for movement of several cases, or run each case a few times and average.

What to measure beyond correctness

Correctness is not the only thing that regresses:

• Refusal rate — is it now declining things it should answer, or answering things it should decline?
• Format compliance — is the JSON still parseable, every time?
• Length — has it quietly become twice as verbose, and therefore twice as expensive?
• Latency and cost per case — improvements that triple the bill are a trade, not a win.

The honest limits

An eval set tells you about the cases in it. It cannot tell you about the ones you did not think of. It goes stale as usage shifts. And a high score on twenty cases is not proof of a working system.

None of that is an argument against it. It is an argument for treating the set as a living thing that grows every time reality surprises you.

Try this: Open a spreadsheet and write ten inputs for a task you already use AI for, with the answer you would want beside each. Run your current prompt against all ten and count the passes. That number is your baseline — and you now have something no amount of eyeballing gives you: a way to tell whether tomorrow’s change helped.