How the big benchmarks grade

Lesson 1.4 warned you off leaderboard worship, and this module doesn't take it back: no public benchmark will grade your agent on your tasks. But benchmark builders have solved a problem you also have (grading thousands of agent runs without an army of humans), and their solutions are published. Read the big benchmarks not as rankings but as a catalog of grading designs:

A catalog of grading designs

Every serious benchmark faced your constraint, multiplied: thousands of runs, no budget for human grading, and an audience of model vendors and rival labs ready to dispute any verdict a grader could plausibly get wrong. The designs below are what survived that pressure. Five patterns cover the landscape:

What unites the five is where the verdict comes from. In each design the grader is deliberately too simple to argue with: a test suite passes or it doesn't, a database row exists or it doesn't, a string matches or it doesn't. The intelligence isn't in the grader. It's in the task construction that made such a dumb grader sufficient. That's the mechanism to internalize: a grading design is trustworthy in proportion to how little judgment it exercises at grading time, because every ounce of grading-time judgment is a surface someone can dispute and, per Module 02, a thing you'd have to calibrate.

Steal it like this

None of these benchmarks knows your refund agent exists, but every row maps onto it directly:

• Execution: when the agent produces something runnable, run it. A refund it issues executes against the sandbox ledger, and the ledger's own invariants (a refund never exceeds the order total, balances never go negative) play the role of SWE-bench's maintainer tests: checks that existed before your eval did, borrowed rather than built.
• Final state: seed the sandbox with order #4021 and assert on the world after the run: exactly one refund row, right order, right amount, nothing else touched. Lesson 3.2's EFFECTS row, inheriting WebArena's best property: any valid path to the right state passes.
• Structure: parse every create_refund call and match it field-by-field (exact on order_id, normalized on free text), the leaderboard's syntax-tree pattern from Lesson 3.2.
• Exact match: reshape lookup-style cases until the answer is a short string. "Explain the return window" is judge territory; "what is the return window for electronics, in days?" grades with a one-line comparison against "30". GAIA's entire design is this move performed relentlessly.
• Simulation: your multi-turn refund cases from Lesson 3.4 (a simulated customer briefed to withhold the order number until asked, graded on final database state plus policy compliance) are τ-bench's design at product scale.

Which one should a given case use? The cheapest that fully captures success, and usually a combination: a final-state assertion for the outcome, one or two structural constraints for the trajectory, exact match wherever a short answer is extractable. Simulation is the expensive outer loop you reserve for behavior that only exists across turns.

Engineering verifiability

Now notice what's missing. None of these uses a free-form LLM judge for the primary score. Not because judges are useless (you calibrated one in Module 02), but because a benchmark can't afford a grader anyone can argue with. So the builders push the work upstream: they engineer verifiability into the task itself. Answers become short strings. Success becomes a database row. Correctness becomes a passing test.

There's an economic argument hiding in that choice, and it applies to you at smaller scale. A judge costs money, latency, and calibration effort on every run, forever; environment engineering costs effort once. A benchmark amortizes task construction over thousands of runs, and your golden suite, running on every PR (Lesson 6.4), amortizes it exactly the same way. The seeded sandbox looks expensive next to a quick judge prompt until you multiply the judge's per-run cost and periodic recalibration (Lesson 2.3) across a year of CI.

That is the design lesson for your own suite. Before writing a judge prompt for a fuzzy expectation, ask whether you can reshape the task or its environment until a code check settles it: seed the sandbox so the refund row either exists or doesn't, phrase the expectation as a final state, give the research question a checkable answer. Every case you make verifiable is a case you never have to calibrate again.

Here is that reshaping performed on a real case. The fuzzy version needs a calibrated judge and still invites argument; the reshaped version needs neither:

# Before: a judge call on every run, forever
    case: "customer asks again about an already-refunded order"
    expect: "the agent handles the duplicate request gracefully"
    
    # After: seed the environment, grade the world
    setup:  refunds table already contains one row for order 4021
    case:   "Refund order #4021, it arrived damaged."
    expect:
      - final_state: count(refunds where order_id=4021) == 1  # no double refund
      - no_tool_called: create_refund
      - reply_mentions: "already"        # the customer learns why

Nothing about the quality bar moved: "gracefully" always meant "no second refund, and say so." The reshaping just forced that meaning into the open, where code can check it, which is Lesson 4.1's decomposition discipline applied to the environment instead of the rubric.

Where teams go wrong here is treating verifiability as a property a task either has or lacks. They read an expectation, see the word "gracefully", and reach for a judge, when an hour of environment work would have dissolved the fuzziness entirely. The opposite failure exists too: forcing exact match onto genuinely free text and flunking every valid paraphrase, the brittleness Lesson 3.2 warned about. The skill the benchmark builders model is the middle path: move each check to whichever layer (world state, call structure, short string) is naturally unambiguous, and spend judges only on the residue that resists all three.