From "good" to checkable criteria

Why two reviewers disagree

Ask two reviewers whether a response is "good" and you'll get two answers, not because either is careless, but because "good" bundles six private judgments. A rubric unbundles them: it decomposes quality into criteria so concrete that two graders (or a grader and a judge model) reach the same verdict.

The mechanism matters, because it tells you what a rubric has to fix. Every reviewer runs a checklist they've never written down: was the tone right, were the facts right, did it resolve the issue. Two reviewers disagree not about what they saw but about which checks they ran and how much each counted. Telling them to "be careful" changes nothing, because carefulness applies the private checklist more diligently. The only fix is making the checklist public: same questions, same definitions. That is all a rubric is.

From failure notes to criteria

Where do criteria come from? Not from imagination but from failures. Run the error analysis of Lesson 1.2 and convert each recurring failure into the criterion that would have caught it ("promised a callback that never happens" → "no promises about actions the agent didn't take"). Rubrics built this way stay short and every line earns its place; rubrics built by brainstorming grow to twenty hypothetical checks nobody applies consistently. Here's the derivation end-to-end for the refund agent, starting where error analysis leaves you, with short notes on real traces:

Error-analysis notes, refund agent. Three recurring failures:
  
  #112  Reply promised "you'll get a callback today."
        We have no callback tool. Pure invention.
  #131  Three paragraphs of apology; the refund approval is
        buried in the final sentence. Customer wrote back
        asking whether they were getting a refund.
  #140  Reply says "refunds take 3-5 business days." The
        retrieved policy states no timeframe anywhere.

Each note names the criterion that would have caught it. Trace #112 becomes no promises about actions the agent didn't take, checked by listing the commitments in the reply against the tool calls in the trace. Trace #131 becomes two criteria: states the resolution in the first two sentences (fixes the burying) and no apology-only paragraphs (fixes the filler). Trace #140 becomes every factual claim is supported by the retrieved policy: the faithfulness judge of Lesson 2.2, now earning its place from a real trace instead of a hunch. Add the housekeeping check support tooling needs (the ticket number) and the rubric is complete.

Before you write a criterion down, apply the test for a well-formed one: it's a yes/no question about an observable property, answerable from the output (or trace) alone, that two people would answer the same way. "Is it professional?" fails that test. These pass:

"Good support reply" decomposed:
  □ States the resolution in the first two sentences
  □ Every factual claim is supported by the retrieved policy
  □ Includes the ticket/order number
  □ No promises about actions the agent didn't take
  □ No apology-only paragraphs (apology must come with a next step)

Watch the two-reviewer test run on a bad criterion. Hand trace #131 (the apology avalanche) to two reviewers with the question "is this reply professional?" One says yes: courteous, well-formed sentences, no typos. The other says no: it buries the resolution, and to them that's the definition of unprofessional. Both verdicts are defensible, because "professional" names a feeling, not a property of the text. The criterion asks each reviewer to consult their taste, and tastes differ.

Now run it on "no promises about actions the agent didn't take." Both reviewers perform the same three moves: list every commitment in the reply ("you'll get a callback today"), list the tool calls in the trace (nothing schedules a callback), check one list against the other. There is no taste to consult; the answer is in the artifact. And when a disagreement does surface ("does 'we'll look into it' count as a promise?"), it exposes an ambiguity you can fix by editing the wording (decide once, write it into the criterion) rather than a difference in personality you can't. That's the mechanism behind the yes/no rule: observable properties turn disagreements into edits.

Where teams go wrong here: writing the rubric in a conference room before reading a single trace. The meeting produces twenty plausible criteria guarding against failures the agent never commits, while the failure it commits weekly, the invented callback, isn't on the list because nobody in the room had seen it. The other classic is smuggling the adjective back in: "appropriately empathetic" is "is it professional" wearing a lanyard, and it fails the two-reviewer test just as hard.

A rubric per case

Some agents get a different task every case ("write the outreach email", "summarize this contract"), and no global rubric fits them all. For those, write a rubric per case: a handful of criteria specific to that one input ("mentions the renewal date", "under 150 words", "no pricing commitments"): the unit test of evals, and the same idea as Lesson 3.4's per-case assertions applied to output quality. Drafting them by hand doesn't scale, so let a model propose each case's criteria from the input and your quality guidelines, then review and edit every draft yourself. Generated criteria inherit generated blind spots, and the human pass is what makes them trustworthy.

Here's what that review pass looks like for one refund-agent case: the model's draft, and the human edits that made it gradeable:

case: "Refund order #4021 and switch my future orders
         to the express shipping plan."
  
  model draft                     human edit
  ------------------------------- --------------------------------
  - mentions the refund           EDIT: "confirms the refund for
                                  order #4021 specifically"; the
                                  draft passes a reply that refunds
                                  the wrong order
  - explains the express plan     EDIT: "states the express plan
                                  price from the retrieved plans
                                  page"; "explains" isn't checkable
  - polite and professional       DELETE: fails the two-reviewer
                                  test
  - (nothing)                     ADD: "no commitment to a plan
                                  change the agent didn't execute"
                                  (our known failure mode; the
                                  model never proposes it)

The pattern in the edits is always the same. The model writes criteria that describe the task; the human rewrites them to encode the failure modes. And the criterion the model never proposes is exactly the one from your error analysis, because it hasn't read your traces.

From rubric to graders

Each criterion then slots into the grader ladder from Lesson 2.1: some are code checks (order number present), most become one narrow judge each (claims supported by policy), a few stay human. A rubric is the missing link between "what we mean by quality" and "what our evals actually compute."

Two questions come up every time. How many criteria? As many as you have observed failure modes; five to eight covers most agents. Fewer, and reviewers fall back on taste for everything unnamed; twenty, and they stop applying it consistently. Doesn't yes/no lose nuance? No, the nuance moves. Each criterion stays binary so it stays checkable; shades of quality come back when you aggregate across criteria (Lesson 4.3) and across cases (Lesson 4.2). You lose nothing except the ambiguity.