Routers, sub-agents, handoffs

Everything so far graded one agent. Then the refund agent grows up: a router reads each request and dispatches it to a billing sub-agent or a returns sub-agent, a planner decomposes the hard tickets, and a critic reviews drafts before they go out. It looks like new eval territory. It's mostly machinery you already own, pointed at more components.

The router is a classifier

Start with the router, because it has the cleanest grade in the whole system: it's a classifier. Build a labeled set of request→correct-route pairs ("where is my order" goes to order lookup, "the blender arrived broken" goes to returns) and score routing accuracy exactly the way you scored tool selection (Lesson 3.2), including the abstain case where no sub-agent fits and the right move is to say so. A wrong route is the multi-agent version of a wrong tool, and it poisons everything downstream.

The labels come from the same place golden cases do (Lesson 1.3): production traces. Pull a hundred routed requests, read each one, and write down where it should have gone, including the ones that should have gone nowhere. The label space is small and fixed, so grading is string equality against the label, with no judge anywhere:

# routing_set.yaml: request → correct route (excerpt)
    "Where is my order #8812?"                           -> order_lookup
    "The blender arrived broken, I want my money back"   -> returns
    "You charged me twice for the same kettle"           -> billing
    "hi can i change the delivery address on 4021?"      -> order_lookup
    "My refund still hasn't shown up on my card"         -> billing
    "Do you sell replacement carafes for the SR-7?"      -> product_qa
    "I was double charged AND the box came crushed"      -> billing    # money first; returns second
    "What's your favorite kettle, personally?"           -> abstain    # no sub-agent fits, say so

Then report the way Lesson 5.2 reports recall: per class, never one aggregate. The router's classes are its routes:

routing accuracy: 240 labeled requests, agent v2.3
    route          n     accuracy   most confused with
    order_lookup   88    97%        billing (2)
    returns        61    93%        order_lookup (3)
    billing        54    76%        returns (11)
    product_qa     25    96%        none
    abstain        12    58%        product_qa (4)
    overall        240   89%

The aggregate says 89% and sounds fine. The breakdown says billing is broken (one billing request in four lands on the returns agent, which will politely process a return for a customer who was double-charged) and that the router almost never abstains, because like the eager tool-callers of Lesson 3.2 it forces a route even when none fits. Those are two different fixes (sharper route descriptions for billing versus returns; an explicit abstain option with examples), and only the per-route table tells you to make either.

When the answer is bad, which agent broke?

That word, downstream, is the real difficulty. When the final answer is bad, which agent broke? Attribution takes instrumentation: one session ID shared across every component, and a source tag on every message saying which agent produced it. With that in place, first-divergence debugging (Lesson 3.3) names the agent that failed, not just the step. The contamination rule from that lesson applies across agents too: a bad answer downstream of a bad plan is contamination, so fix the planner before you touch the executor.

Do you need new tooling for that? No: it's the trace metadata of Lesson 6.1 with two fields added. The mechanism to respect is that agents consume each other's outputs as context: a planner that writes one wrong constraint doesn't cause one wrong step, it biases every step the executor takes afterward. That's why blame concentrates upstream, and why per-agent scores computed on contaminated runs will quietly slander your innocent executor.

Grade the handoff artifacts

Between the agents flow handoff artifacts, and each one is an intermediate output you can grade in isolation: fix the input, run one component, apply its own criteria, the per-case assertion pattern of Lesson 3.4:

The brief deserves a worked example, because brief starvation is the signature multi-agent failure and the least visible one. Here's what the router hands the returns sub-agent for "the blender from my order arrived cracked, I bought it just over a month ago and I want my money back":

# brief: router -> returns sub-agent
    task:   "process a return"
    item:   "blender"
    reason: "damaged"

Three things are missing: the order ID (#4021), the purchase date (day 34, outside the standard 30-day window), and the user's actual goal (a refund, not a replacement). The downstream failure writes itself. The returns agent re-asks for the order number the customer already gave (the symptom users report as "it forgot what I told it"), then applies the standard window instead of the damaged-item exception and refuses a refund the policy allows. And the trace of that failing run shows the returns agent behaving perfectly on the context it received. Only a brief eval (a fixed request, the generated brief, a checklist of fields it must carry) indicts the real culprit, and it runs without invoking the returns agent at all.

The critic gets the same treatment as any judge, because that's what it is. Build a drafts-labeled set: collect forty or fifty real drafts the critic has reviewed, have your domain expert label each pass or fail with a one-line reason (Lesson 2.3's loop, unchanged), then run the critic on the same drafts and score agreement, misses and false blocks separately, since a critic that rejects everything "catches" every bad draft. A critic well below expert agreement isn't reviewing; it's adding latency. Re-run the set whenever the critic's prompt changes, like any judge.

The seams are the system

Where teams go wrong here: they eval every component, watch five green dashboards, and skip the composed system. Or, the mirror image, they grade only the final answer and re-prompt whichever agent spoke last, patching the executor for the planner's failure. Both mistakes come from treating a multi-agent system as either a bag of parts or a black box. It's neither; it's parts plus seams.

Keep the end-to-end suite regardless. Sub-agents that pass every isolated check still starve each other of context when composed (the brief drops the order number, the critic approves against last month's policy), and only a full-system run catches it. That's Lesson 3.4's cross-validation point again: step metrics green while end-to-end sinks means the seams are broken, and in a multi-agent system, the seams are most of the system.