End-to-end vs. step-level checks

Two granularities, two jobs

Two granularities, two jobs. End-to-end evals run the whole agent on a task and check the outcome: user got the right answer, environment ended in the right state. Step-level evals isolate one decision: given this exact context, does the agent pick the right tool? Given these retrieved chunks, is the summary faithful?

Use both, in a pyramid you already know from software testing: many step-level checks (unit tests) on every change, a smaller end-to-end suite (integration tests) on merges and releases. The two layers cross-validate: if step metrics look great but end-to-end sinks, the composition is broken (steps starve each other of context, or errors compound); if end-to-end is fine but a step metric is red, your step eval is testing something users don't need.

Capability probes

There's a useful probe between the layers: capability checks without an environment. When end-to-end fails and step metrics look fine, you still don't know whether the model lacks the skill or your scaffold is getting in its way. Strip the harness and test the capability directly: given a fixed context, can the model produce a valid plan, pick the right tool from your catalog, follow the output format? These run as cheap single-call cases, no sandbox required, and they split every failure into "model limit" (revisit your model shortlist; see Lesson 9.3) or "scaffold bug" (fix the prompt, context assembly, or tool descriptions).

Two probes for the refund agent, ready to adapt:

# Probe 1: tool selection, one model call, no sandbox
    context: full tool catalog + the conversation so far
    input:   "Refund order #4021, it arrived damaged."
    expect:  first proposed call is get_order or check_eligibility
             (anything but create_refund as the opening move)

# Probe 2: plan validity, one model call, no sandbox
    context: order record and policy chunk pasted inline
    input:   "Write the step-by-step plan to resolve this refund."
    expect:  - eligibility appears before refund in the plan
             - every tool the plan names exists in the catalog

Both probes are single model calls against fixed context (nothing executes), so a few hundred of them run in CI for pennies. Read them jointly with the end-to-end result. Probes pass while end-to-end fails: the model has the skill and your scaffold is starving it; look at context assembly, tool descriptions, the prompt. Probes fail: no amount of scaffold work will save you; sharpen the tool catalog or revisit the model choice.

Reliability and pass^k

One more end-to-end habit: run each case more than once. Agents are stochastic: the same agent on the same task succeeds Tuesday and fails Wednesday. The strict version of this is pass^k: the probability the agent succeeds on all k independent trials, averaged across tasks. In τ-bench, the best agent solved over 60% of retail tasks on a single try, but fell below 25% when required to succeed on all 8 trials. If reliability matters, report the all-k number, not the best run.

Work the arithmetic once on a suite you could actually own (20 cases, 5 trials each) and the gap stops being abstract:

20 cases × 5 trials = 100 runs
  
      cases   trials passed   per-case pass rate
       12          5/5             1.00
        5          4/5             0.80
        2          2/5             0.40
        1          0/5             0.00
  
    pass@1 = average single-trial success
           = (12·5 + 5·4 + 2·2 + 1·0) / 100 = 84/100 = 84%
  
    pass^5 = cases that passed ALL five trials
           = 12 / 20 = 60%

Same suite, same hundred runs, and a 24-point gap between the headline number and the reliability number. The mechanism is plain exponentiation: a case the agent passes with probability p passes k independent trials with probability p^k, so an 80%-reliable case contributes 0.8^5 ≈ 33% to pass^5, not 80%. Flaky cases are nearly invisible in pass@1 and dominate pass^k. Report pass@1 while you're exploring what the agent can do at all; report pass^k once it acts on real accounts: a customer who gets the wrong outcome one time in five doesn't experience an 80% agent, they experience a broken one.

End-to-end also has a multi-turn problem: real users don't hand the agent one message; they clarify, change their mind, and withhold details until asked. Fixed single-turn cases can't test that. The tool is a simulated user: a second LLM given a persona and a goal ("you want a refund but don't know your order number; reveal it only if asked") plays the customer for a full dialogue, while your usual graders score the outcome and trajectory. It's how τ-bench runs its tasks, and the practical way to eval clarifying questions and context held across turns. Keep the simulator's brief tight: an unconstrained fake user drifts off-task and grades nothing.

Per-case assertions

The practical bridge between layers: per-case assertions. Instead of one global rubric, each end-to-end case carries its own list of checks: outcome checks ("final state contains exactly one refund") plus a few trajectory constraints ("eligibility checked before refund; no escalation"). You get end-to-end realism with step-level diagnosability, one case at a time.

Here is the complete assertion file for the running case (the constraint spec of Lesson 3.2 promoted to a full end-to-end case with trials and outcome checks):

id: e2e-refund-4021
    input: "Refund order #4021, it arrived damaged."
    trials: 5
  
    outcome:                                # what the user and the world got
      - final_state: refunds == [{order_id: 4021, amount: 89.99}]
      - reply_mentions: "refund"
      - judge: "reply promises nothing beyond the refund it created"
  
    trajectory:                             # how it got there
      - must_call_before: [check_eligibility, create_refund]
      - must_not_call: [escalate_to_human]
  
    pass: all assertions on all trials      # this case feeds pass^k

Where teams go wrong. Best-run reporting: quoting the trial that passed ("it works, I saw it work"), which is pass@5 wearing a suit; decide in advance which statistic gates the release. Inverted pyramid: running the slow end-to-end suite on every commit until it takes an hour, at which point nobody runs it at all; step-level on every change, end-to-end on merges and releases. One global rubric: grading every end-to-end case against the same generic checklist, which is how "mentions the order number" gets asked of a case that has no order; per-case assertions exist precisely so each case carries its own definition of done.