Build your first golden cases

A golden case is a single input with a written-down expectation, drawn from real usage, that your agent must keep passing forever. Your first suite doesn't need to be big: aim for 20 to 50 tasks drawn from real failures, and even five gets you moving. What matters is maintenance: ten cases you read and update beat a thousand you generated and never look at.

Where cases come from

Where to get them: your own error analysis (Lesson 1.2), support tickets, and the requests your team types most. Pick a mix: a few bread-and-butter cases the agent must never fumble, a few known past failures you've fixed and refuse to re-break, and one or two hard edge cases that define your quality bar.

No users yet? Bootstrap with synthetic inputs, but structure the generation, or it collapses into fifty rephrasings of one request. Write down the dimensions that vary across your users (feature, scenario, persona), hand-write twenty combinations yourself, then let a model turn each combination into a natural-language request: inputs, never expected outputs. The two-step separation is what keeps the phrasing diverse. Run the requests through your agent, read a sample of the traces like any other error analysis, and swap in real cases as soon as you have them.

Writing the case spec

For each case, write the expectation as the strictest check code can express. "The agent should handle the refund" is not checkable; these are:

case: "Refund order #4021, it arrived damaged."
    expect:
      - tool_called: create_refund(order_id=4021)
      - reply_mentions: "refund"
      - reply_does_not_mention: "replacement"   # a past failure mode
      - no_tool_called: escalate_to_human

In a real repo, each case carries a little metadata around that core, and every field earns its keep. Six months from now a failing case with no provenance is a mystery: nobody remembers whether the expectation is still right, so nobody dares touch it. source answers "why does this case exist," owner names who can rule when the agent and the expectation disagree, added lets you audit how the suite grew, and runs plus pass_if encode the non-determinism policy per case rather than globally:

# cases/refunds/damaged-4021.yaml
    id: refund-damaged-001
    source: trace_7f3a12        # production failure, 2026-05-14
    added: 2026-05-15
    owner: mira
    tags: [refunds, tool-use]
    input: "Refund order #4021, it arrived damaged."
    runs: 5                     # agents are stochastic; score the rate
    pass_if: 5/5                # bread-and-butter case: no flakiness allowed
    expect:
      tool_called:
        - create_refund: {order_id: 4021}
      no_tool_called: [escalate_to_human]
      reply_mentions: ["refund"]
      reply_does_not_mention: ["replacement"]
    notes: >
      Agent used to offer a replacement instead of the refund the
      user asked for. Fixed in prompt v14; this case keeps it fixed.

Running the suite

Run the suite like unit tests: on every prompt change, model upgrade, and tool edit, ideally in CI (the automated checks that run on every code change), so a regression blocks the merge instead of reaching users. When a golden case fails, either the agent broke (fix it) or the expectation was wrong (fix the case, and write down why). Both outcomes are information.

Concretely, a run looks like this (a few minutes of wall-clock time, most of it the agent's own latency):

$ evals run --suite golden --agent v27
    refund-damaged-001      5/5  pass
    refund-no-order-002     5/5  pass
    refund-ineligible-003   3/5  FAIL  2 runs refunded without eligibility check
    address-change-004      5/5  pass
    angry-escalation-005    4/5  FAIL  pass_if is 5/5; 1 run promised a callback
    ...
    suite: 42/47 passing   (v26: 44/47)
    new failures vs v26: refund-ineligible-003, angry-escalation-005
    transcripts: runs/2026-07-09/

Two lines in that output matter more than the total. The diff against the previous version turns a number into a decision: two cases that passed yesterday fail today, so this change doesn't merge until someone looks. And the transcript path is where they look: the score says whether something broke, only the trace says why. A runner that prints a percentage and nothing else has thrown away the half of the output you act on.

Flaky cases and growth

Flaky cases (pass on one run, fail on the next) will show up early, and the wrong instinct is to treat them like flaky unit tests and add retries. A case at 3/5 is not noise; it's a measurement of an agent that fails this task 40% of the time, which is exactly what you built the suite to detect. Handle flakiness explicitly instead: raise runs on that case until the rate is stable enough to trust, set pass_if to match the stakes (5/5 for bread-and-butter tasks, 4/5 might be acceptable for a hard edge case while you work on it), and if a case must be parked, quarantine it with an owner and an expiry date rather than deleting it. Never "retry until green". A suite that reruns failures until they pass has been trained to lie to you, and the mechanism is one-way: once you stop trusting red, the suite stops protecting anything.

Grow the suite the same way it started: every genuinely new production failure becomes a candidate golden case. That single habit (failure in, case in) is the whole data flywheel in miniature. And add the case before the fix, test-driven style: reproduce the failure as a failing golden case, then change the agent until it passes; the fix is proven and the regression test exists by construction.

Once a case passes, perturb it: paraphrase the request, add typos, change the formatting. Same expectation, five phrasings. An agent that passes "Refund order #4021, it arrived damaged" but fails "hi, my order (4021) came broken, can i get my money back?" hasn't learned the task; it memorized a sentence. Score consistency across the variants, and treat a case that only passes in its original phrasing as a failure; production users never phrase anything the way your suite does. Research on prompt robustness finds even character- and word-level perturbations move model scores substantially.

Where teams go wrong

Three patterns kill more golden suites than any tooling gap. The thousand-case dump: a model generates a huge case set on day one, nobody reads it, and within weeks it's failing for reasons nobody can explain. The maintenance principle from the top of this lesson isn't a preference, it's the mechanism that keeps a suite meaning something. Prose expectations: cases whose expectation is "handles the refund politely" can't fail crisply, so every red mark becomes a meeting; if you can't express the check in the spec format above, the case isn't ready. Deleting failing cases: under deadline pressure, the case that blocks the merge quietly disappears. But the rule is that a failure means either the agent broke or the expectation was wrong, and both outcomes get recorded, not erased. A deleted case is a documented failure mode you've chosen to forget.

Practical questions, answered: your first five cases take an afternoon, not a sprint; one error-analysis session (Lesson 1.2) hands you the candidates. And the cases live in your repo, next to the code, reviewed in pull requests like everything else, because an expectation about agent behavior is a spec, and specs that live outside version control drift.