Pass/fail vs. graded rubrics

Aggregation follows the decision

You've designed the scale for each criterion; now decide how the rubric aggregates, and that depends on what the eval is for.

Why can't one aggregation serve all three? Because the decisions pull in different directions. A gate must be conservative: one bit, no partial credit, because partial credit is how a bad case sneaks through. A compass must be sensitive: the strict pass rate is a step function that barely moves while the agent improves underneath it, whereas per-criterion rates move the week the improvement lands. And triage needs an ordering, which neither a bit nor a trend line provides. The rubric stays identical across all three; only the arithmetic on top changes.

Critical criteria: the veto

Two structural tools cover most needs. Critical criteria: some failures are absolute: fabricated policy, leaked personal data, an unauthorized side effect. Any critical failure fails the case outright, no matter how well everything else scored. This is the fix for the averaging trap, where a 9-of-10 score hides the one failure that would make headlines.

Concretely: a refund reply that nails structure, order number, and tone (nine boxes of ten) while fabricating a policy clause averages to 0.9, comfortably above any threshold you'd plausibly set. The one failure is the only one the customer will screenshot. Mark "claims supported" critical and that reply scores fail, full stop; the flattering average never gets computed. Which criteria deserve the flag? Ask: would this failure alone make you pull the release? Expect two or three yeses. A rubric where half the criteria are critical isn't a rubric with a veto; it's a strict gate pretending to have gradations.

A weighted checklist, worked

Weighted checklists: for non-critical criteria, weight by how much users care, sum the passes, and you get a graded score whose meaning you can always explain: unlike a judge's holistic 7, a 0.83 checklist score decomposes back into exactly which boxes were missed. Graded at the suite level, binary at the criterion level: that combination serves gates, compasses, and triage at once. Here it is end-to-end for the refund rubric of Lesson 4.1:

Rubric v3: refund replies (criticals veto; weights sum to 1.0)
  
  CRITICAL  every factual claim supported by retrieved policy
  CRITICAL  no unauthorized side effect in the trace
  
  0.30  states the resolution in the first two sentences
  0.28  no promises about actions the agent didn't take
  0.25  includes the ticket/order number
  0.17  apology, if present, comes with a next step
  
  Case #131, agent v2.4:
    criticals:  pass, pass          → no veto
    0.30 pass + 0.28 pass + 0.25 pass + 0.17 FAIL
    score = 0.30 + 0.28 + 0.25 = 0.83
    reading: one miss, and the score names it: the apology check.

That's the whole trick of the decomposed 0.83: it isn't 83% of some ineffable quality, it's "everything passed except the 0.17 criterion," recoverable from the number and the weight table alone. Where do the weights come from? User impact, roughly estimated: this team put "no untaken-action promises" near the top because broken promises reopen tickets and burn trust, and the apology check last because it annoys rather than misleads. Rough is fine: the weights exist to order failures, not to be precise, and if you can't defend a weight, set them equal and move on. Lesson 4.2's warning about false precision applies to weights exactly as it does to scales.

One rubric, three consumers

Now feed that one scored suite to all three consumers and watch the same data answer three different questions:

GATE (CI, on merge):
    case #131 → FAIL (strict mode: 0.17 criterion missed)
    suite: 41/50 strict passes, 2 critical failures → BLOCK
  
  COMPASS (weekly trend, per criterion):
    supported claims:      92% → 96%
    resolution up front:   84% → 85%
    no untaken promises:   88% → 74%   ← regressed this week
    apology + next step:   71% → 70%
  
  TRIAGE (fix-next ranking, weight × failure count):
    no untaken promises   0.28 × 13 fails = 3.64  ← fix first
    apology + next step   0.17 × 15 fails = 2.55
    order number          0.25 ×  3 fails = 0.75

The gate blocked the merge on two critical failures; the average never entered the conversation. The compass shows this week's prompt change traded promise-discipline for nothing, a regression the strict pass rate alone would have reported as a vague dip. And triage ranks the promise failures above the more numerous apology failures, because weight times frequency beats frequency: fifteen mild annoyances matter less than thirteen broken promises. Same rubric, same runs, three decisions.

Whatever you choose, pre-commit: write down the passing bar before you run the eval. A threshold chosen after seeing the results isn't a quality bar; it's a rationalization. This is where teams go wrong most reliably: the suite comes back 0.83 average with two critical failures, the release is due Friday, and someone proposes that just this once the gate should be the average, because the criticals are "edge cases." Every aggregation is defensible in isolation; that's exactly why you must pick one before the numbers exist. Post-hoc, you will always find the arithmetic that says ship.