Why the path matters

From answers to trajectories

A chatbot produces text; an agent produces a trajectory: a sequence of decisions, tool calls, observations, and retries that ends in an answer or an action. Grade only the final answer and you're blind to most of what happened.

The blindness is structural, not a matter of effort. The reply is a few hundred tokens at the end of the run, while every property you actually care about (did it look up the right order, did it read the current policy, did it change anything it shouldn't have) was settled several steps earlier. Two runs can be word-for-word identical at the output layer and differ at every layer underneath. A grader whose input is only the reply cannot separate them, because the behavior you want to check is not in the text it reads. No cleverer judge prompt fixes that; only widening the grader's input to the whole trajectory does.

Right answer, wrong reasons

The failure that motivates this module is right answer, wrong reasons. The agent quotes the correct refund policy, but never called the policy lookup tool; it guessed from training data, and next month, when the policy changes, it will guess wrong with the same confidence. Outcome-only evals score both runs identically. A trajectory eval catches the second one today.

Here is that failure as two concrete runs of the refund agent, side by side. Read the final sentence of each first; they are identical.

# Run A: guessing                     | # Run B: grounded
    01 model_call  respond                | 01 tool_call  get_order(order_id=4021)
       -> "Damaged items are refundable   | 02 tool_result {delivered 12 days ago}
           within 30 days, so your        | 03 retrieval  "damaged item refund policy"
           refund is being processed."    | 04 tool_call  check_eligibility(4021)
                                          | 05 tool_call  create_refund(4021, 89.99)
    # zero tool calls; no refund exists   | 06 model_call respond -> same sentence

An answer-only grader scores these runs the same: same claim, same polite confidence. A trajectory grader has something to hold on to. Run B contains a get_order event, a policy retrieval, an eligibility check, and exactly one create_refund with the right arguments. Run A contains none of them, and its claim that a refund "is being processed" is falsified by the world itself, because no refund row exists after the run. Run A passed on borrowed time: the 30-day figure happened to match what was in the training data. The eval that separates the two runs isn't smarter; it just reads more.

The reverse failure matters just as much: right-looking answer, harmful path. "Your refund is processed!", said after calling create_refund three times, or with someone else's order ID. Side effects live in the trajectory, not the reply.

Reading a trace

The raw material is the trace: a structured record of every step (model calls, tool invocations with arguments and results, retrieved chunks, timing, errors). Instrument your agent to emit traces from day one, in development and production. Traces are simultaneously your debugging tool, your eval input, and (per Lesson 1.2) the source of your next golden cases. An agent without traces isn't evaluatable; it's just anecdotes.

Concretely, here is the full trace of Run B, in the shape your instrumentation should emit (one line per event, with a type, a payload, and timing):

# trace 8f3c · input: "Refund order #4021, it arrived damaged."
    01 model_call    plan                                          420ms
         -> "Get the order first; check policy second."
    02 tool_call     get_order(order_id=4021)                       85ms
    03 tool_result   {status:"delivered", days_since_delivery:12, total:89.99}
    04 retrieval     query="damaged item refund policy"            130ms
    05 retrieval_result  top: returns-policy#damaged (score 0.91), +2 chunks
    06 model_call    decide                                        610ms
         -> "12 days < 30-day window, damage reported -> eligible."
    07 tool_call     check_eligibility(order_id=4021, reason="damaged")  140ms
    08 tool_result   {eligible:true, method:"original_payment"}
    09 tool_call     create_refund(order_id=4021, amount=89.99)    230ms
    10 tool_result   {refund_id:"rf_2210", status:"created"}
    11 model_call    respond                                       540ms
         -> "Your $89.99 refund for order #4021 is on its way ..."

Read a trace in three passes. First, grounding: every claim in the step-11 reply should trace back to an earlier observation (the amount to step 3, the eligibility to step 8, the policy language to step 5). A claim with no upstream step is a guess, even when it's true. Second, actions: find every step that changed the world (here, only step 9) and check its arguments and its count (exactly one refund, right order, right amount). Third, cost: the timing column sums to about 2.2 seconds across 11 steps, and both numbers become baselines that Lesson 3.3 turns into metrics. Ten minutes of this per day, on real traces, is the highest-signal habit in this module.

Two questions come up immediately. What should you log? Everything the model saw and did: full prompts, parsed tool arguments (not just strings), raw results, retrieved chunk IDs and scores, timings, token counts, errors. Storage is cheap, and the field you dropped is always the one the next bug needs. Doesn't tracing slow the agent down? Not if you emit events asynchronously; it adds bookkeeping, not model latency. And when payloads contain customer data, redact the sensitive fields at write time but keep the structure, so your assertions still have arguments to check.

Where teams go wrong

Three patterns account for most wasted effort here. Log-and-forget: the team wires up tracing, dashboards fill with spans, and nobody ever reads one trace end-to-end; collection without the error analysis of Lesson 1.2 is storage, not evaluation. Prose logs: steps are logged as human-readable strings ("called the refund tool for the damaged order"), which no checker can assert on; log structured events with typed fields and render prose from them for humans, never the reverse. Tracing failures only: if you keep traces only when something breaks, you have no picture of what a healthy trajectory looks like and no baseline to diff a suspicious run against. Trace every run; sample what you read.