Calibrating against human labels

The calibration loop

An uncalibrated judge is a random number generator with confidence. Before its scores mean anything, you must show it agrees with a human you trust, and the only way to show that is to label data yourself.

The calibration loop:

• 1. Label a sample. Take 50 to 100 real outputs: for the refund agent, fifty actual replies, each labeled pass/fail on faithfulness by a domain expert, with a one-line reason. One expert, not a committee: a single trusted labeler gives you consistent labels, while averaging disagreeing annotators gives you mush. Painful, and the single highest-leverage hour in this course.
• 2. Run the judge on the same sample and compare.
• 3. Read every disagreement. Judge said pass, human said fail? Usually a missing definition: fix the prompt. Human label wrong? Fix the label; your criterion just got sharper.
• 4. Repeat until agreement is boringly high, then freeze prompt + labels as your calibration set.

Why is labeling the highest-leverage hour? Because the labels aren't just test data for the judge; they're where your criterion becomes real. Writing pass or fail on fifty actual replies, with a reason each, forces definitional decisions that no amount of prompt drafting surfaces: is an invented-but-plausible timeframe a fail? Is a correct reply that ignores half the question a faithfulness fail, or only a completeness one? You'll hit a dozen of these before reply thirty, and every answer sharpens the judge prompt before the judge ever runs.

One trusted expert is the default, not a law. When your domain genuinely requires several (clinical review, multiple locales), have every annotator label the same small overlap set and measure their agreement with each other before trusting anyone's labels. If two experts can't agree between themselves, no judge can be calibrated against either, and the fix is a sharper rubric (Module 04), not a third vote. And keep labeling in-house: a vendor can follow instructions, but the domain judgment that makes labels worth calibrating to is exactly what they don't have.

A worked confusion matrix

Measure agreement honestly. Raw accuracy misleads when labels are imbalanced: if 90% of outputs pass, a judge that always says "pass" is 90% accurate and 100% useless. Check agreement on pass and fail cases separately, or use a chance-corrected statistic like Cohen's kappa.

Here's what that looks like in practice: fifty refund replies labeled by your support lead, then graded by the v1 faithfulness judge from Lesson 2.2:

50 replies, human-labeled for faithfulness: 12 fail, 38 pass
  
                        judge: fail    judge: pass
    human: fail (12)         7              5      <- 5 missed failures
    human: pass (38)         2             36      <- 2 false alarms
  
    raw agreement:   (7 + 36) / 50 = 86%
    TPR (fail side):       7 / 12  = 58%   <- catches barely half
    TNR (pass side):      36 / 38  = 95%

Read it the way the imbalance warning says to. The 86% headline looks respectable, and it's carried almost entirely by the easy pass cases. On the fail side the judge catches 7 of 12: it misses nearly half the failures your expert flagged, and catching failures is the judge's entire job. So the five missed fails are worth more than the thirty-six agreements combined; they're where the prompt is broken. The two false alarms deserve a read too, because a "false alarm" is sometimes a wrong human label, and fixing one sharpens the criterion for free. What the numbers say: don't ship this judge, and don't tweak it blindly either; go read five specific transcripts.

One disagreement, end to end

Take missed failure #3 and walk it the whole way:

output: "You're eligible for a refund on order #4021. Refunds
      take 3-5 business days, and you'll get a confirmation email."
  
    human:  FAIL ("policy states no processing timeframe;
      3-5 days is invented")
  
    judge:  PASS ("the response correctly confirms eligibility and
      gives an accurate estimate of standard refund timing")
  
    diagnosis: the judge graded plausibility against world knowledge,
      not support against the context.
    fix (v2):  add "unsupported claims = fail, even if plausibly true"
      + this reply as the fail example.
    re-run:    fails caught 7/12 -> 11/12; pass side unchanged.

The judge's own reasoning is the diagnosis: "standard refund timing" gives away that it consulted world knowledge, because the v1 prompt never said which source of truth counts. The fix is exactly the v2 move from Lesson 2.2: tighten the definition and paste the disagreement in as the fail example. One disagreement, read carefully, moved fail-side agreement from 58% to 92%. That's the exchange rate on reading your data, and it's why step 3 of the loop says every disagreement, not a sample.

The twelfth fail (a reply that subtly misread a policy exception) stays a disagreement after v2. That's fine. Note it, keep it in the set, and let a later iteration or a sharper criterion catch it. Calibration converges; it doesn't need to hit 100% to beat the alternative, which is an ungraded judge you trust on faith.

Where teams go wrong

Celebrating raw accuracy: the team sees 86%, declares the judge calibrated, and ships a grader that misses half of all failures: the exact trap the confusion matrix exists to catch. Calibrating on convenient outputs: labeling fifty outputs someone generated in an afternoon of ad-hoc prompting instead of fifty real production replies, so the judge is calibrated on a distribution it will never grade again. One-and-done calibration: running the loop once, framing the agreement number, and never re-running it, even as the judge model, the prompt, and your own standards all keep moving (Lesson 2.4).

Expect criteria drift: the act of reading outputs changes your own standard; you notice a failure mode halfway through labeling that you'd been letting slide. That's not sloppiness, it's how evaluation actually works (the "Who Validates the Validators" study documents it well). When it happens, update the definition and re-label. Calibration is a loop you revisit, not a gate you pass once.