LESSON 9.3 · 20 MIN READ · Module 9: Benchmarks & the landscape

What leaderboards can't tell you

Reading scores with suspicion

A leaderboard score looks like a measurement and behaves like a headline. Before you let one pick your model, know the four ways the number deceives:

CONTAMINATIONBenchmark tasks leak into training data (public repos, forum answers, the benchmark's own paper). A model can "solve" tasks it memorized, so scores on aging public benchmarks inflate without any gain in ability.
SATURATIONWhen every frontier model scores above 90%, the benchmark has stopped discriminating. The remaining gaps are noise, label errors, and edge-case quirks, not a ranking worth a shipping decision.
SINGLE NUMBERAccuracy alone hides what you'll pay. Two agents tie at 60%; one costs 10× more per task. And a best-run score hides reliability: pass^1 and pass^k are different claims (Lesson 3.4), and headline numbers usually report the flattering one.
HARNESS"Model X scores 62%" really means "model X, inside this scaffold, with these prompts, tools, and retry logic". Change the scaffold and the score moves wildly. The scaffold is part of the system under test, and yours is different.

Walking a real decision

Make the four rows a checklist and walk a decision with them. The refund agent runs on a mid-tier incumbent model; two candidates just launched (a frontier flagship and a budget small model), and someone has posted leaderboard screenshots in the team channel. Before any of the three touches your suite, an afternoon of reading applies the checklist.

Contamination first. For each benchmark you're tempted to cite, ask when its tasks went public relative to each model's training cutoff, and prefer variants with refreshed or held-out task sets. If the budget model shines on the oldest public suites and fades on every refreshed one, suspect memorization and discount those wins. Then saturation. Strike any benchmark where all three candidates clear 90%: whatever separates them there is label noise, not capability. What survives, for a support agent, is usually the tool-use and policy-compliance benchmarks in the τ-bench mold, because they still spread the field where chat leaderboards no longer do.

Then the single number. Pull cost per task and pass^k wherever the benchmark reports them; the flagship's headline lead usually narrows once reliability and price sit beside it. Finally the harness. Read what scaffold produced each score: if the leaderboard's agent got retries, tools, and prompt scaffolding your product won't have, the number transfers weakly. The outcome of the afternoon: flagship and incumbent make the shortlist for a full run on your suite; the budget model misses the tool-use bar for the agent itself but gets a note as a guardrail-classifier candidate (Lesson 7.3). Nothing has been decided yet. That's the point. The deciding artifact is your own run, several trials per case (Lesson 6.4), which produces a table no leaderboard can:

your suite: 200 cases × 4 trials each
    
    model        pass@1   pass^4   $/case    p50 latency
    flagship      84%      71%     $0.31       14.2s
    incumbent     81%      69%     $0.09        6.1s
    budget        66%      41%     $0.011       2.3s

The headline says flagship by three points. The table says more: the reliability gap nearly closes at pass^4, the latency doubles, and the 22-cent difference per case is roughly $66,000 a year at ten thousand conversations a day. Whether three points of accuracy is worth that is a product decision (the cost of the extra failures against the infra spend), but it's arithmetic now, possible only because cost and reliability sit next to accuracy in the same file. And before deciding, read the flip list (Lesson 6.4): if the flagship's extra wins cluster in one query segment, a prompt fix on the incumbent might buy the same three points for nine cents a case.

Two reader questions the walk usually raises. Why four trials? Because a single-run comparison of stochastic agents is a coin-flip tournament (Lesson 3.4's point), and pass^4 is where the flagship's headline lead visibly shrinks; a leaderboard reporting best-run numbers would never have shown you that. How often do you redo this? Not on every launch. Re-open the decision when a candidate offers a plausible mechanism for improvement on your failure clusters (better tool use when tool errors dominate your flip lists, longer context when context loss does), not because a general-capability number moved somewhere else.

Why a bare accuracy number misleads

The cost point deserves emphasis, because the research is blunt about it: when benchmarks reward accuracy alone, builders produce needlessly complex, expensive agents, and simple baselines often match them at a fraction of the price. A score without a cost attached is half a result. Demand both from public leaderboards, and report both from your own suite: dollars-per-case and latency belong next to pass rate, a habit the next lesson builds into your results format.

Shortlist, then decide

Where teams go wrong is letting the news cycle run this process. A new model tops a leaderboard, a screenshot lands in the channel, and the team either switches on hype or freezes because testing feels expensive, and both are symptoms of the same missing asset: a cheap, trusted way to answer "is this model better for us?" With the harness the next lesson builds, evaluating a candidate model is an afternoon and a results file, the switching conversation happens over flip lists instead of screenshots, and model choice stops being an argument about headlines.

The practical rule is the one this course opened with, worth repeating now that you can defend it: benchmarks are for choosing a model; your evals are for shipping your product (Lesson 1.4). Use leaderboards to shortlist two or three plausible models. Then run your own suite (your cases, your scaffold, your costs, repeated trials) and let that decide. The shortlist takes an afternoon of reading; the decision deserves your benchmark.

KEY IDEA

Use leaderboards to shortlist models; use your own suite, on your own scaffold, to decide.

REFERENCES