Build your own benchmark

You've built every piece this course teaches: golden cases, calibrated judges, trajectory checks, rubrics, a maintenance cadence. The capstone is packaging them so they outlive you: turning your suite into a benchmark-grade harness that anyone on the team can run, this month or next year, and trust the number that comes out.

One directory per task

The anatomy comes straight from the benchmarks you toured in Lesson 9.1, and the terminal-agent benchmarks make the pattern explicit: one directory per task, holding a task definition, an isolated environment, and a verification test.

tasks/refund-damaged-item/
        task.md         # the user message + what success means
        environment/    # seeded sandbox: orders DB, policy docs, mock tools
        verify.py       # asserts final state + trajectory constraints

The task definition is the human-readable contract, written so a new teammate could grade a run by hand from it alone:

# task.md
    User message:
      "Refund order #4021, it arrived damaged."
    
    Success means:
      - exactly one refund exists for order 4021, for $64.99
      - eligibility was checked before the refund was created
      - the reply confirms the refund and invents no timeframe
    
    Environment: orders DB seeded with #4021 (damaged, inside window)
    Trials: 4        Timeout: 120s

And the verification test is the same contract as executable code: Lesson 3.4's per-case assertions, promoted to a file that lives beside the task it grades:

# verify.py: runs against the sandbox after the agent finishes
    def verify(env, trace):
        refunds = env.db.rows("refunds", order_id=4021)
        assert len(refunds) == 1                  # final state, path-agnostic
        assert refunds[0].amount_cents == 6499
        assert trace.order("check_eligibility", "create_refund")
        assert not trace.called("escalate_to_human")
        assert "refund" in trace.final_reply.lower()
        assert not re.search(r"\d+.\d+ (business )?days", trace.final_reply)

Notice what the shape enforces. Most assertions are final state and structure (the verifiability you engineered in Lesson 9.1), and when a criterion genuinely needs a judge (tone, say), verify.py calls it as one more assertion, with the judge prompt versioned in the same directory. Success is defined twice, once for humans and once for machines, and a disagreement between the two files is a bug you can file.

The environment is the part teams skip and then regret. A task that depends on live systems produces scores that change when those systems do. Seed everything: fixture data, tool responses, the policy documents the agent retrieves. Then pin what you can't seed: model versions, tool versions, judge prompts, sampling seeds where your stack allows them. Pinning is what makes March's 78% comparable to June's 84%; without it you're comparing two different experiments (Lesson 2.4's drift problem, applied to the whole harness).

A results file worth keeping

Record more than the headline. A benchmark-grade results file carries, per task: pass/fail for each trial, the number of trials, cost, and latency. From that you can compute pass^k (Lesson 3.4), catch a change that held accuracy while getting 10× more expensive (Lesson 9.3), and diff two runs case by case to see which flips are real improvements and which are noise (Lesson 4.2). The headline pass rate is derived from the file, never stored in place of it.

run 2026-06-30 | agent v41 | model pinned 2026-05 | suite v12
    
    task                        trials  passes  $/trial   p50
    refund-damaged-item            4       4     0.21      11s
    refund-outside-window          4       3     0.34      19s
    duplicate-refund-blocked       4       4     0.18       8s
    injected-review-ignored        4       2     0.29      14s
    ...
    
    derived: pass@1 83.5% · pass^4 64% · total $41.20 · p95 latency 31s

Read the derived line the way Lesson 3.4 taught: the gap between pass@1 and pass^4 is your reliability story, and it's the number to quote when someone asks whether the agent can act alone (Lesson 8.3). Read the rows too: injected-review-ignored passing 2 of 4 is a live vulnerability by Lesson 7.2's rule, and the aggregate 83.5% hides it completely. The file is the eval; the headline is a summary of it.

What pinning saved

Play the March-to-June story out. In March the suite reads 78%; in June, 84%. Because the harness pinned everything (same 200 tasks at suite v12, same seeded environments, same judge prompts on the same pinned judge model), the agent is the only variable that moved, and the six points decompose on the flip list: fifteen cases fixed (twelve of them in the outside-window cluster the team actually targeted), three newly broken, one of those a real regression that gets filed before release. Now the counterfactual team, who pinned nothing: over the same quarter their judge endpoint silently upgraded, someone edited a policy fixture, and thirty easier cases were added to the suite. Their 78% and 84% blend agent, grader, and dataset motion in unknown proportions, and the disambiguating move, old-suite-on-new-agent (Lesson 6.4), is impossible because March's suite no longer exists anywhere. Same two numbers; one pair is a measurement, the other is two headlines.

Buy the plumbing, own the content

Should you build the harness or adopt one? Adopt the plumbing: open-source eval frameworks handle runners, retries, and reports fine, and that code is not your advantage. But own the content. Your cases, rubrics, judge prompts, and environments encode your product's definition of good: keep them in your repo as plain files that would survive any framework migration.

"Would survive" has a concrete test, the migration test: imagine switching frameworks next year and ask what it costs. It should cost you glue (a runner adapter, a report converter, a weekend). It should not cost you a single case, environment fixture, judge prompt, or rubric line; if it would, your definition of good was stored in the tool's format instead of yours. Where teams go wrong is exactly there: they author cases in a vendor's UI or proprietary DSL because it's convenient, and a year later the accumulated encoding of "good" (the most expensive asset the eval effort produced) is hostage to a tooling decision. Task directories of plain files pass the migration test by construction; that's half of why the pattern is worth stealing.

Then share it. A benchmark the whole team can run is the most durable definition of "good" your product will ever have: sharper than a spec, harder to argue with than a demo. New engineer? Run the benchmark. Candidate model? Benchmark. Prompt change, new tool, quarterly review? Benchmark. The suite stops being your project and becomes the team's shared answer to "is it good enough to ship?"

And that's the whole course. An eval was never more than an input, a run, and a grader; you've now learned to build each one properly: golden cases mined from real failures, the cheapest grader that survives calibration, trajectories graded alongside answers, rubrics that keep "good" honest, and the same loop running offline in CI and online against production, through retrieval (Module 05), monitoring (Module 06), adversaries (Module 07), and the hard shapes (Module 08). Public benchmarks taught you the patterns; your own benchmark is where they live. Keep it running, and let your users' hardest days write your next golden case.