Designing score scales

Why 1 to 10 manufactures noise

The instinct is a 1 to 10 scale, because it feels precise. In practice it manufactures noise. Nobody can define what separates a 6 from a 7, so graders cluster in the 6 to 8 band, judge models even more so, and the decimal you report ("quality rose from 7.1 to 7.4") is mostly grader mood. Worse, a mid-scale score defers the decision: is 6 shippable? You still have to pick a threshold, which is a binary question you've merely postponed. The scale hasn't removed the definitional work; it has moved it from the criterion, where definitions can live, to the threshold, where they can't.

Default to binary. Pass/fail forces the definitional work that makes evals trustworthy: to draw the line, you must say what crossing it means. Binary verdicts calibrate better against humans (agreement is measurable), aggregate cleanly (pass rate), and compose: a 10-criterion rubric with binary checks yields both an overall score and a per-criterion diagnosis.

Anchored scales that settle arguments

When you truly need gradations, use a small anchored scale, every level defined by observable properties, not adjectives:

3: Correct resolution, all claims supported, nothing missing
  2: Correct resolution, but an unsupported claim or missing step
  1: Wrong or no resolution, or any fabricated policy

Watch the anchors work in a live disagreement between two graders:

Reply under review: refund for #4021 correctly confirmed,
  but adds "you'll see the credit within 3 business days" when
  the retrieved policy states no timeframe.
  
  Grader A: "3: the resolution is right and complete."
  Grader B: "2: something feels off about it."
  
  Apply the anchors:
    A 3 requires "all claims supported." The timeframe claim
    is unsupported → this reply cannot be a 3.
    A 1 requires a wrong resolution or fabricated policy.
    The resolution is right → not a 1.
  Both graders: 2.

Without anchors, A and B are negotiating moods and the louder grader wins. With them, the argument becomes "is the timeframe claim supported?", a checkable fact that changed A's verdict without anyone pulling rank. Note what the anchors do: they turn the scale into three binary questions in a trench coat. That's the point: if you can't write the anchor, you don't have a level; you have a feeling. And if you need finer resolution than a few anchored levels, don't stretch the scale: add cases. Statistical power comes from more examples, not more decimals per example.

The arithmetic of small suites

That cuts the other way too: a small suite can't detect small improvements. On 50 cases the margin of error on a pass rate is roughly ±10 to ±14 points, so a 4-point gain between two versions is likely noise. Before believing a delta, check it exceeds the wobble of re-running the same version twice.

To size a suite for the difference you care about, use the square-root rule: the margin of error on a pass rate is roughly ±100/√n points, so 100 cases gives about ±10 and 400 gives about ±5. Detecting a 5-point improvement from two aggregate scores alone takes more cases than most teams have. The escape is pairing: both versions ran the same cases, so compare per case and count the flips: nine fixed and one broken is a real improvement even on a suite where the aggregate delta would drown in noise. And when the comparison gates a release, add a confidence interval: resample your per-case results a few thousand times (a bootstrap, ten lines of code) and report the range, not the bare number. If the interval on the paired difference includes zero, you don't have an improvement. You have a re-roll.

Here's one comparison read both ways, same data, different arithmetic:

The mechanism: pairing refuses to average away the structure. If the change did nothing, flips split roughly evenly; a 9-to-6 split alone is weak evidence (a coin does that often), which is why the aggregate stayed mute. But flips have names, so error analysis resolves what the statistics couldn't: six regressions, one cause. A 9-to-1 split, by contrast, is lopsided enough to stand as evidence by itself. The bootstrap, in full:

# paired bootstrap: does v2 beat v1?
  diff = [v2_pass[i] - v1_pass[i] for i in cases]   # +1, 0, or -1
  obs  = mean(diff)                                 # +0.06
  sims = []
  for _ in range(10_000):
      resample = choose(diff, k=len(diff), replace=True)
      sims.append(mean(resample))
  lo, hi = percentile(sims, 2.5), percentile(sims, 97.5)
  print(obs, lo, hi)   # +0.06, (-0.02, +0.15) → includes zero
  # → a re-roll, not a win

Where teams go wrong: the ritual dashboard delta: one run per version, unpaired, "quality up 3 points" in the weekly review. Every part of that sentence is noise: n too small for the delta, no pairing so flips can't be counted, no repeat run so the wobble is unknown. The fix is an afternoon, not a statistics degree.

When there's no right answer

For the genuinely subjective (tone, persuasiveness, style), where even anchors turn to mush, there's a third mode: pairwise comparison. Don't ask "how good is this reply?"; ask "is A better than B?" Humans and judge models are both more consistent at picking a winner than at placing a score, which is why the big human-preference leaderboards are arena-style, built on comparisons, not ratings. Use it to compare prompt or model variants; keep binary criteria for anything with a right answer.

When the output is a professional deliverable (a report, an analysis, a slide deck), pairwise has a natural upgrade: compare against human work. For a handful of cases, have a domain expert produce the deliverable themselves; then show experts the agent's version and the human's, blinded and unlabeled, and ask which they'd accept. The win rate against a human baseline is the most honest quality bar a work-product agent can have; it's how frontier models are graded on real occupational tasks, with blind expert comparisons against deliverables from experienced professionals.