What is an eval, really?

Strip away the tooling and an eval is three things: an input you care about, a run of your real agent on that input, and a grader that turns the result into a score. That's it. If you can write those three down, you've built an eval.

One eval, end to end

To make the shape concrete, follow one case all the way through: the same refund agent this course uses throughout. Everything starts with a case spec: the input plus the expectation, written down before anyone runs it. A minimal one fits in a few lines (Lesson 1.3 adds the metadata a real suite needs):

id: refund-damaged-001
    input: "Refund order #4021, it arrived damaged."
    expect:
      - tool_called: create_refund(order_id=4021)
      - reply_mentions: "refund"

The run step feeds that input to your agent (the production build, same prompt, same tools, same model settings) and records everything it does into a transcript. Here is what came back on one run:

USER   Refund order #4021, it arrived damaged.
    AGENT  → tool: lookup_order(order_id=4021)
    TOOL   ← {status: "delivered", total: 86.00, refund_eligible: true}
    AGENT  → tool: create_refund(order_id=4021, reason="damaged")
    TOOL   ← {refund_id: "rf_9912", status: "processing"}
    AGENT  "Sorry your order arrived damaged. I've issued a refund
            for order #4021. You'll see $86.00 back on your card."

The grader then walks that transcript and checks each expectation. Here the grader is plain code (parse the tool calls, match the arguments, scan the reply), and it commits to a verdict:

{
      "case": "refund-damaged-001",
      "checks": [
        {"check": "tool_called: create_refund(order_id=4021)", "verdict": "pass"},
        {"check": "reply_mentions: refund",                    "verdict": "pass"}
      ],
      "verdict": "pass"
    }

And the score does something next. This is the part teams forget to design. One verdict joins the verdicts of every other case into a suite pass rate. That pass rate gets compared against the last agent version, so "we changed the prompt" becomes "42 of 47 passing, down from 44." A drop blocks the change from shipping, and each failing verdict arrives with its transcript attached, so the person who reads it can see exactly which step went wrong. A score that nobody compares, gates on, or reads is decoration; decide where your scores flow before you build anything else.

Graders, repeatability, checkability

Graders come in three flavors, and you'll meet all of them in this course: code checks (exact, cheap, use them first), LLM-as-judge (for qualities code can't express; see Module 02), and humans (the ground truth you calibrate the other two against).

Two properties make an eval useful rather than decorative. It must be repeatable: the same input and the same agent version should produce a comparable score tomorrow, so you can tell whether a change helped. And it must be checkable: the grader has to commit to a verdict you could defend to a colleague. "The output seemed fine" is neither.

The mechanism behind both properties is the same: an eval is only useful as a comparison. You never care about one score in isolation; you care whether today's agent beats yesterday's. Repeatability makes the two scores commensurable; checkability makes each one mean something. Break either and the comparison silently becomes fiction: you'll change a prompt, watch a number move, and never know whether the agent changed or the measurement did.

One caveat on "repeatable": agents are non-deterministic, so a single run is a coin flip, not a measurement. Run each case a few times and score the rate: "passes 5 of 5" and "passes 3 of 5" are different agents (Lesson 3.4 turns this into a metric).

Eval, test, benchmark

Three words get blurred together in every eval conversation, and it pays to keep them apart. A test, in this course a golden case (Lesson 1.3), is one input with one written expectation: the unit test of the agent world. An eval is the whole measurement apparatus: the set of cases, the runs, the graders, and the scores they produce; people also say "an eval" for a single case, so be precise when it matters. A benchmark is someone else's eval: a public, standardized task set with a fixed grader, built so different models can be compared on the same footing (Module 09). Benchmarks tell you about a base model's general ability; only your own evals tell you about your agent on your tasks. When someone says "our evals are great" ask which of the three they mean; the answer changes what the claim is worth.

Where teams go wrong

Three failure patterns show up in almost every first eval setup. The demo harness: the eval runs a stripped-down copy of the agent (different system prompt, mocked tools, different temperature), so the scores describe a system nobody ships. The RUN row above says "exactly as it runs in production" because every difference between the harness and production is an untested variable, and untested variables are where the surprises live. The vibes grader: a human skims outputs with no written expectation. That verdict changes with mood, can't be delegated, and can't be compared across weeks; it fails the checkability test even when the human is an expert. The scoreboard with no consequence: scores get computed and posted, but nothing gates on them and nobody reads the failing transcripts, so the numbers drift into wallpaper within a month. Each pattern breaks a different leg of the input, run, and grader triad; the fix in each case is to restore the leg, not to add more cases.

The obvious follow-up questions have short answers. Do you need a framework? No: a for-loop over a folder of YAML files and a script that prints verdicts is a complete eval system, and it's what you should start with; buy tooling when you need history, dashboards, and teammates running the same suite. How many cases? Five gets you moving, and Lesson 1.3 tells you where to find them. When do you run it? On every change to the prompt, the model, or the tools, which is to say, constantly.

Everything else in this course (judges, trajectories, rubrics) is just better ways to pick inputs and build graders. The shape never changes.