Common judge failure modes

The bias catalog

Judges are models, so they fail like models: systematically. The classic biases, and the standard mitigation for each:

These biases aren't random noise; they're inherited. Judge models were tuned on human preference data, and human raters prefer longer, more confident, more agreeable text, so the judge does too: that's verbosity and leniency in one stroke. Position bias falls out of how models attend across a long context. The reason the distinction matters: noise averages out as you add cases, but a systematic bias moves every verdict the same direction; it doesn't blur your ranking of prompts and models, it silently rewrites it.

Detecting each bias in your data

The table's mitigations are cheap insurance, but insurance is not measurement. Each bias also has a detection test you can run on your own verdicts, using artifacts you already have:

• Position: re-run your pairwise comparisons with the order swapped and count verdicts that flip. The flip rate is your position bias: a judge that agrees with itself in both orders on 95% of pairs is usable; one at 70% is a coin with extra steps.
• Verbosity: bucket judged replies by length (under 50 words, 50 to 150, over 150) and compare the judge's pass rate per bucket against the human pass rate per bucket from your calibration set. A gap that grows with length means the judge is grading word count.
• Leniency: you already measured it; it's the fail-side row of Lesson 2.3's confusion matrix. High agreement on passes with low agreement on fails is exactly what leniency looks like in data.
• Self-enhancement: grade the same fifty outputs with judges from two model families and split the disagreements by which model generated each output. A gap that appears only on the judge's own family's outputs is the tell.
• No reference: sample twenty verdicts on reasoning-heavy cases and re-derive the answers yourself. Count how often the judge approved a wrong answer because it re-derived instead of compared.
• Score clustering: histogram the raw scores. If most of your 1 to 10 verdicts land on 6, 7, or 8, the scale is decoration, and the histogram just told you to go binary.

Every test above reuses something you already built (the calibration set, the pairwise runs, the stored verdicts), so the full battery is an afternoon, not a project. Run it when you first stand a judge up, and again after any mitigation change, because a mitigation you haven't measured is a mitigation you're hoping about.

The drift re-run protocol

The quieter failure is drift. Judges run on a model endpoint; endpoints get upgraded. The same prompt can silently start grading differently months later. Pin model versions where possible, and re-run your frozen calibration set on a schedule; if agreement drops, the judge changed, not your agent.

Make the re-run mechanical, so it actually happens:

• Freeze the control. The calibration set, meaning fifty outputs, human labels, and the baseline numbers (raw agreement, fail-side rate), is stored alongside the judge prompt version and judge model version that produced the baseline.
• Schedule the re-run. Monthly, plus after any provider upgrade or deprecation notice and any edit to the judge prompt. Temperature 0, the same fifty inputs.
• Diff against baseline. Recompute the same numbers. Pre-commit the alarm threshold before you need it: say, any drop of five points or more.
• On alarm, read the flips. Inputs and labels didn't change, so every flipped verdict isolates the judge. Re-pin the model version if you can; re-tune the prompt and re-calibrate if you can't.
• Log versions with verdicts. Every stored verdict carries judge model + prompt version, so historical scores stay interpretable after the judge moves on.

The mechanism is the same one behind any control experiment: the frozen set holds the agent, the inputs, and the labels constant, so the only thing that can move the number is the judge itself. Five minutes of re-run answers a question that otherwise burns days of debugging: did my agent regress, or did my ruler change length?

Where teams go wrong

Blaming the agent: the dashboard dips, the team spends three days bisecting agent changes, and the real cause was a silent judge-endpoint upgrade; the frozen-set re-run would have settled it before lunch. Mitigation theater: pasting "length is not a virtue" into the prompt and declaring verbosity handled, without running the length-bucket comparison before and after. The averaged dashboard: blending every criterion into one overall score, so a drifting faithfulness judge is diluted by five stable ones and no alarm ever fires; track pass rates per criterion, per judge.

Above all, never let the judge become an unread oracle. Sample ten of its verdicts every week and read them against the transcripts. The point of calibration (Lesson 2.3) was to earn trust; spot-checks are how you keep it.