Keeping rubrics honest over time

Three forces of decay

A rubric is a snapshot of what quality meant when you wrote it. Three forces quietly invalidate it:

• Your product moves. New tools, new policies, new user segments: criteria written for last quarter's agent miss this quarter's failure modes.
• Your standards move. Grading outputs changes what you grade for, the criteria drift of Lesson 2.3: you need criteria to judge outputs, but judging outputs is what teaches you the criteria. Yesterday's pass is today's borderline, and the rubric on paper still enforces yesterday.
• Your agent overfits. Optimize against a fixed checklist long enough and the agent learns the checklist, not the quality: replies that tick "mentions the order number" while getting worse at everything unmeasured. When a measure becomes a target, it stops measuring (Goodhart's law).

Holdouts, and what leaking looks like

The overfitting force now comes with automation: prompt optimizers that mutate your prompt and keep whichever variant raises the eval score. They work, but only late: an optimizer hill-climbs the metrics you already have, polishing known failure modes and discovering nothing new. Pointed at a weak rubric, it optimizes the checklist, not the quality. Before any aggressive tuning, automated or manual, split your cases: a development set you optimize against and a holdout you only run at release time, refreshed from fresh production failures. When the development score climbs and the holdout doesn't move, the optimizer learned your checklist, not your task.

The mechanics, concretely. Split roughly 80/20: a 200-case suite becomes 160 development cases you run daily and 40 holdout cases you run only at release (results visible, traces unread). Refresh on a cadence; quarterly works: retire a slice of the holdout into the development set and refill it from fresh production failures, so it neither goes stale nor gets slowly memorized by your process. And know that leaking rarely looks like cheating; it looks like diligence. A release fails on holdout case #17, a conscientious engineer reads that trace and patches the prompt against it, and #17 passes forever after: it is now a development case wearing a holdout badge. The tell is divergence: development pass rate climbing 78% → 91% while the holdout sits at 74% → 75%. That gap is the measured size of your overfitting.

A quarterly session, worked

The countermeasure is a maintenance cadence, and it's cheaper than it sounds, an hour or two a month:

• Prune saturated checks. A criterion at 100% for months is a solved problem; retire it to a slim regression suite and spend the grading budget on live failure modes.
• Re-run error analysis on fresh production traces. New failure clusters become new criteria: the same loop that built the rubric keeps it current.
• Re-calibrate the judges. Re-label a fresh sample against the current rubric and check agreement (Lesson 2.3); this catches both judge drift and your own drift.
• Version everything. Rubric, judge prompts, and golden sets change scores when they change, so a score is only meaningful with its rubric version attached. Track them like code, because they are.

Here's one session for the refund rubric (ninety minutes), and every changelog line is one of the four moves above:

rubric: refund-replies, v3.1 → v4.0   (Q3 maintenance)
  
  PRUNED     "includes the ticket/order number": 100% pass
             since March; moved to the slim regression suite.
  ADDED      "quotes the policy version in effect at the
             purchase date": new cluster in fresh traces:
             11 of 60 applied the current policy to orders
             placed before the policy change.
  RELABELED  50 fresh traces against v4.0. Judge-human
             agreement: 94% overall (96% on passes, 88% on
             fails). All 3 disagreements were the new
             criterion → added two fail examples to its
             judge prompt.
  NOTE       scores before this date are v3.x scores; do
             not chart them on the same line.

Read the agreement number the way a doctor reads a blood panel. 94% overall clears the bar, but the split (96% on passes, 88% on fails) says the judge is slightly lenient on exactly the new criterion, and the fix was two fail examples in its judge prompt, not a new judge (Lesson 2.4's leniency bias, caught in the wild). And the version bump is not bookkeeping: a chart that mixes v3 and v4 scores will show a "regression" that is actually the new, stricter criterion doing its job. Score and rubric version travel together, or the trend line lies.

Where teams go wrong: the silent criterion edit. Someone sharpens a definition mid-quarter (reasonable edit, no version bump) and the weekly compass drops five points, triggering a fire drill for a regression that never happened. The agent didn't change; the ruler did. Every criterion edit, however small, is a version bump and a changelog line.

And know when to stop adding. Watch where new production failures land: when month after month they fall into existing clusters instead of opening new ones, coverage has caught up with reality, and the marginal case is a duplicate. From there hold the size roughly constant: a genuinely new failure mode still buys its way in (Lesson 1.3), but pay for it by pruning a saturated check. A suite that only grows eventually fails the pace test of Lesson 1.4.

Closing the loop in production

Offline suites only see the inputs you chose; production sees everything. Close the loop with online evaluation: sample a slice of live traces daily, run your calibrated judges on them, and alert when a pass rate drops: monitoring with the same graders you trust in CI. User signals feed the loop too: a thumbs-down is a lagging metric (Lesson 1.4), but every thumbs-down trace is a candidate golden case, and comparing judge verdicts against user feedback is a free calibration check. And when you change a prompt, ship it to a fraction of traffic first and compare the arms on the same online metrics; an A/B test is just an eval where production picks the inputs.

That's the core discipline: golden cases from real failures, the cheapest calibrated grader, trajectories not just answers, and rubrics that stay honest. The remaining modules take it into the field: the retrieval layer most agents stand on (Module 05), production traffic (Module 06), adversaries (Module 07), the hard agent shapes (Module 08), and the benchmark landscape (Module 09).