Preference evals and Elo

Lesson 4.2 ended with pairwise comparison: for subjective quality, "is A better than B?" beats "score this 1 to 10". Arena-style evaluation is that idea at industrial scale. A user asks a question, two anonymous models answer side by side, the user votes for the better answer, and millions of blind votes are aggregated into Elo-style ratings: the system from chess, where beating a strong opponent moves you up more than beating a weak one. The result is a live ranking of models by human preference.

How a rating actually moves

The mechanics are worth one paragraph, because they explain both the power and the noise. Every model carries a rating, and the gap between two ratings implies an expected win probability: a 400-point gap means roughly ten-to-one. After each vote, the winner takes points in proportion to the surprise: beat an equal and you gain about half the update constant; beat a far weaker model and you gain almost nothing; pull an upset and you gain a lot. The exchange is zero-sum: whatever the winner gains, the loser loses. Watch three models play it out:

Ratings start at 1000, update constant K = 32.
    
    Vote 1, A beats B (equals, expected win 0.50):
      A: 1000 + 32 × (1 − 0.50) = 1016      B: 984
    Vote 2, C beats A (A now favored; C expected 0.48):
      C: 1000 + 32 × (1 − 0.48) ≈ 1017      A: ≈ 999
    Vote 3, B beats C (C favored; B expected 0.45):
      B: 984 + 32 × (1 − 0.45) ≈ 1002       C: ≈ 999
    
    After three votes: B ≈ 1002, A ≈ 999, C ≈ 999.

Read the ending honestly: three votes produced a near three-way tie that the next vote would reorder. A rating is an estimate with error bars, and at low vote counts it measures mood, not quality: Lesson 4.2's noise warning in a new costume. Production arenas actually fit a Bradley-Terry model rather than running sequential Elo updates: the same core assumption (win probability from a strength gap), but fit over the entire vote history at once by maximum likelihood, so the result doesn't depend on vote order and comes with confidence intervals. Elo is the online, one-vote-at-a-time approximation of it. The practical reading rule follows: when two models' intervals overlap on a leaderboard, they are tied, whatever the sort order implies, and it takes thousands of votes per pairing before the intervals get tight.

The design earns its place where nothing else works: open-ended generation with no ground truth. There is no verification test for "which model writes the better email", and humans can't place absolute scores on it either. But they can pick winners. Blinding removes brand loyalty, pairing controls for prompt difficulty, and scale averages out any single voter's mood. For broad chat quality, arena ratings are the best public signal there is.

Three ways to over-read it

They are also easy to over-read. Three failure modes to hold in mind:

Each row is detectable, not just theoretical. Style bias: arena operators publish style- and length-controlled ratings precisely because rankings move when they control for them: if a model drops under the controls, charm was doing the work; in-house, a variant that wins votes without moving your verifiable pass rates (Lesson 9.1) is winning the same way. Distribution: cluster your production queries (Lesson 6.2) and compare against the arena's published category mix: refund traffic versus coding-and-essay traffic answers the transfer question in an afternoon. Wrong question: for the models you can access, correlate their arena order with their order on your own suite; when the correlation is weak (and for tool-heavy agents it often is), the rating carries little information about your product.

The in-house arena

So steal the mechanics, not the leaderboard. When you compare two prompt versions or two candidate models inside your product, run the arena pattern yourself: same golden cases, both variants, blind labels, both orders (Lesson 2.4's position fix), and let per-case wins aggregate into a preference. On your distribution, with your graders, pairwise voting is a sharp instrument. On someone else's, it's a shortlist input. The next lesson says what to do with those.

Concretely, the artifact is a vote sheet:

comparing prompt-v12 vs prompt-v13: 50 golden cases, blind pairwise judge
    
    case refund-4021-damaged:
      round 1: left = v12, right = v13   → vote: right  (v13)
      round 2: left = v13, right = v12   → vote: left   (v13)  # orders swapped
      verdict: v13 (survived the swap)
    
    case return-window-question:
      round 1: left = v12, right = v13   → vote: right  (v13)
      round 2: left = v13, right = v12   → vote: right  (v12)  # flipped with order
      verdict: discarded (position bias, not preference)
    
    tally: v13 wins 31 · v12 wins 9 · discarded 10  → ship v13

The mechanics matter in exactly that order. Both orders per pair, because position bias lives in your judge as surely as in arena voters: a verdict that flips with the swap is the slot talking, not the quality, so it's discarded rather than counted. Blind labels, because knowing which answer is "the new prompt" is brand loyalty at team scale. And with two variants you don't need Elo or Bradley-Terry at all: a win count with the discard rule is the aggregation, and the rating machinery only earns its keep when many variants play a whole league. For voters, use a calibrated pairwise judge (Lesson 2.3) for volume and route its discards and near-ties to a human, the same escalation shape as Lesson 6.3's review queue.

Where teams go wrong: they run the comparison once, in one order, on fifteen cases, and ship the "winner". At that sample size the margin sits inside the wobble of re-running the same variant twice (Lesson 4.2), and without the swap half the verdicts are slot preference. The subtler version of the same mistake is skipping the in-house arena entirely and letting the public one make the call, which is precisely the over-reading the next lesson dismantles.