Agents that touch the world

Every eval so far ran in a sandbox, and for good reason: the agent under test issues refunds. But eventually the agent gets write access to production, and the gap between "passes the suite" and "trusted with real money" needs a bridge. The bridge is shadow mode: run the candidate on real traffic, but log its intended writes instead of executing them. The reasoning is real and the tool reads are real. Only the side effects are intercepted. Grade the logged actions with the same trajectory checks you run offline (Lesson 3.2). Production inputs, zero production side effects.

What a shadow run produces

Shadow mode's quiet advantage is that the ground truth is free. The tickets it shadows are still being resolved by your human support team, so every shadowed run comes with the label you'd otherwise pay for: what a human actually did with the same ticket. Line the two up and the report grades itself:

shadow report: refund agent v2.4, 2026-06-22..28, 412 shadowed tickets
    ticket  agent's intended write                 human's actual action       match
    #7741   create_refund(4021, $49.00, card)      refunded $49.00 to card     yes
    #7743   create_refund(4022, $129.00, card)     partial refund $89.00       NO  over-refund
    #7750   (no write, escalate)                   refunded $19.00             NO  under-action
    #7752   create_refund(4031, $12.50, credit)    refunded $12.50 to credit   yes
    ...
    action accuracy = matching decisions / tickets = 366/412 = 88.8%
    over-action     = 31/412 (7.5%)  agent would write more than the human did
    under-action    = 15/412 (3.6%)  agent would write less, or escalate instead

Read the two mismatch directions separately, the habit Lesson 7.4 drilled: over-action is money out the door and under-action is customers stuck in a queue, and a single accuracy number hides which problem you have. Ticket #7743 is the expensive kind: the human issued a partial refund because only the carafe was damaged, and the agent would have refunded the whole order.

The promotion decision, with numbers

Now walk the promotion decision. Before the shadow run started, the team wrote down the bar (Lesson 4.3): promote to gated when action accuracy holds at or above 95% for two consecutive weeks, with zero over-refunds above $100. The run came back at 88.8%, and the noise math of Lesson 4.2 says that on 412 cases the wobble is roughly ±5 points, so this isn't "almost 95"; it's clearly below the bar. Two of the 31 over-actions exceeded $100. The call: no promotion. The 46 mismatches go through error analysis (Lesson 1.2), where most trace to one cause: the agent never retrieved the partial-refund policy, a retrieval fix (Lesson 5.4), not a prompt fix. Fix, shadow another two weeks, re-ask. The bar written before the run is what makes the "no" unarguable; a bar chosen afterward would have drifted down to meet 88.8.

How long to shadow? Two constraints: enough volume that the error bar is smaller than the distance to your bar, and enough calendar time that the rare, expensive ticket shapes have actually appeared: a week that never included a $500 refund request proves nothing about $500 refunds.

The gated stage pays for itself twice, because every approval and every rejection is a free labeled case. A reviewer who rejects a proposed refund has just labeled a failure. Route it through the review queue's exits (Lesson 6.3) and it becomes a golden case before the agent ever acts alone.

Inject the faults you fear

An agent trusted with writes must also be graded on the days the world misbehaves. Fault injection seeds the sandbox so tools fail on purpose (timeouts, 500s, empty result sets, permission errors) and asserts graceful recovery: the agent retries sensibly, tells the user the truth about what happened, and never proceeds as if the failed call had succeeded. Lesson 3.3 taught you to recognize dead ends in traces. Fault injection manufactures them on demand, so recovery becomes a suite you run instead of a behavior you hope for.

Build the suite as a matrix (your tools down one side, fault types across the top) so coverage is a fact you can see rather than a feeling:

fault-injection matrix: each ✓ is one seeded sandbox case
                        timeout   500   empty result   permission denied
    lookup_order          ✓        ✓        ✓                ✓
    check_eligibility     ✓        ✓        ✓                none
    create_refund         ✓        ✓       n/a               ✓
    send_confirmation     ✓        ✓       n/a               ✓

One cell, worked. A timeout on create_refund is the nastiest fault in the grid because it's ambiguous: the call may have landed before the timeout fired, so a blind retry double-refunds and giving up strands the customer:

case: fault-create-refund-timeout
    seed: create_refund times out on the first call; a retry succeeds
    expect:
      - tool_called: create_refund(order_id=4021)     # retried after the timeout
      - final state: exactly one refund row for #4021 # not zero, not two
      - reply confirms the refund only after a call returned success
      - reply_does_not_mention: "processed" on any run where all attempts failed

That "exactly one row" assertion is doing more than checking the model: it grades the whole system's idempotency, and many teams discover here that the real fix is an idempotency key on the refund tool, not a cleverer prompt. That's a feature of fault injection, not a bug: an agent that touches the world is agent plus tools plus infrastructure, and the eval should fail whichever layer drops the ball.

Where teams go wrong with the whole progression: they treat shadow mode as a ceremony (three days on light traffic, an eyeballed log, a promotion on vibes), or they shadow without recording what the human did, which leaves nothing to compute accuracy against, or they write the bar after seeing the number it needs to beat. Each shortcut converts the bridge back into the leap of faith it was built to replace.

Computer-use agents, the ones that click through real interfaces, are the extreme case of touching the world. The grading pattern is final state in a seeded VM, the design the serious benchmarks use (Lesson 9.1): set up the machine, let the agent work, assert on what changed. Flakiness discipline matters as much as the grading. Real environments are noisy, so run several trials per task and separate agent failures from environment failures in the report: an agent blamed for a hung installer is a metric your team will learn to ignore.