Tracing: instrument everything

Lesson 3.1 told you to emit traces from day one. This lesson is about what that instrumentation should capture once real users arrive, because everything else in this module reads traces. Online judges score them, review queues route them, golden cases are cut from them. Get tracing wrong and the rest of the module has nothing to run on.

The anatomy of a trace

A production-grade trace is a tree of spans:

Here is one real run of the refund agent, condensed:

{
    "trace_id": "tr_9f2c",
    "metadata": {
      "session_id": "sess_88d1", "user_segment": "enterprise",
      "agent_version": "2.4.1", "prompt_version": "refund-v12",
      "model_version": "sonnet-2026-05"
    },
    "span": {
      "name": "agent.run",
      "input": "Refund order #4021, it arrived damaged.",
      "latency_ms": 6480, "cost_usd": 0.0142, "error": null,
      "children": [
        { "name": "llm.plan", "input": "<system prompt refund-v12 + user msg>",
          "output": "call lookup_order(4021)",
          "latency_ms": 910, "cost_usd": 0.0031, "error": null },
        { "name": "tool.lookup_order", "input": { "order_id": 4021 },
          "output": { "status": "delivered", "amount": 84.00, "date": "2026-06-30" },
          "latency_ms": 140, "cost_usd": 0, "error": null },
        { "name": "retrieval.policy", "input": "damaged item refund policy",
          "output": ["policy_v3#chunk_12", "policy_v3#chunk_14"],
          "latency_ms": 220, "cost_usd": 0.0001, "error": null },
        { "name": "llm.decide", "input": "<order record + 2 policy chunks>",
          "output": "call create_refund(4021, 84.00)",
          "latency_ms": 1180, "cost_usd": 0.0044, "error": null },
        { "name": "tool.create_refund", "input": { "order_id": 4021, "amount": 84.00 },
          "output": { "refund_id": "rf_5512" },
          "latency_ms": 310, "cost_usd": 0, "error": null },
        { "name": "llm.reply", "input": "<refund result + conversation>",
          "output": "Your refund of $84.00 for order #4021 is on its way…",
          "latency_ms": 1030, "cost_usd": 0.0038, "error": null }
      ]
    }
  }

Every question this module will ask is answerable from that one object. Which policy chunks did the agent see before deciding? retrieval.policy's output, so a faithfulness judge (Lesson 2.3) checks the reply against exactly those chunks, not the whole knowledge base. Did the refund amount match the order? Compare tool.lookup_order's output to tool.create_refund's input: a pure code check. Where did 6.5 seconds go? Sum the children. What did the run cost? It's on the root. And when create_refund times out next Tuesday, the error field on that span names the step, and the parent trace shows what the agent did about it.

One standard format

Emit spans in a standard telemetry format, not a homegrown schema. The broader software world settled this problem for distributed tracing with an open, vendor-neutral standard, and the same idea applies to agents: instrument once, and any storage or analysis backend can read the result. You can switch backends next year without touching agent code, and off-the-shelf libraries for your model and tool clients emit conformant spans automatically. A homegrown schema means re-instrumenting every time your tooling changes, and it will change.

The mechanism worth understanding: a standard format is a contract between writers and readers. Your agent code is the writer; judges, dashboards, review tools, and replay harnesses are the readers, and readers multiply over time. With a shared schema, each new reader costs nothing; with a bespoke one, each needs an adapter, and every schema tweak breaks all of them at once. The trace above is deliberately boring JSON with predictable field names. Boring is the feature.

Three rules, and why each holds

• Instrument on day one. Retrofitting after an incident is too late; the traces you need most are the ones you didn't record.
• Sample when volume hurts. Keep every trace that errored or got flagged; sample the healthy majority. Never sample away failures to save storage.
• Redact PII at capture time. Card numbers, emails, addresses: scrub before the span is written. Once stored, personal data spreads into eval datasets, judge prompts, and review queues; the only reliable filter is at the source.

Each rule has a mechanism behind it. Day-one instrumentation works because traces are only valuable in hindsight: the run you'll desperately want to replay is, by definition, one you didn't know mattered when it happened. Sampling asymmetrically works because failures are rare and healthy traffic is redundant: your ten-thousandth successful refund teaches you nothing the hundredth didn't, but every errored trace is a candidate golden case (Lesson 1.3), and once dropped it cannot be recovered.

Redaction deserves its own patterns, because "scrub PII" hides a design choice. For structured fields (tool inputs and outputs, where you know the schema), use an allowlist: enumerate the safe fields (order_id, amount, status, refund_id) and drop everything else by default, so a new field in a tool response stays private until someone argues otherwise. For free text (user messages, model outputs), you can't enumerate fields, so fall back to a denylist of pattern detectors: card numbers (digit runs passing a Luhn check), emails, phone numbers, postal addresses, government IDs, and your own API keys and session tokens, which teams forget until one lands in a judge prompt. Replace each hit with a typed, numbered placeholder (<EMAIL_1>, <CARD_1>), stable within a trace, so a reviewer can still follow "send it to <EMAIL_1>" across turns without seeing the address. Allowlists fail closed, denylists fail open; that's why structured data gets the former and only free text settles for the latter.

The replay test

A test for your instrumentation: pick any user complaint and try to replay exactly what the agent saw and did, step by step, from the trace alone. If you can't, neither can your judges or your reviewers.

Run it on the trace above. The complaint: "your agent refunded me the wrong amount." Walk the tree: lookup_order returned 84.00, create_refund was called with 84.00, the reply said $84.00. The agent was internally consistent; the dispute is about what the order record holds, which routes the ticket to the orders team, a conclusion reached in ninety seconds without grepping application logs. Had the trace recorded only the final reply, the same complaint would mean reconstructing the run from guesswork.

Where teams go wrong: they instrument the model calls and stop, because that's what the client library made easy. The resulting trace shows everything the agent said and nothing it saw: no retrieval outputs, no tool results, prompts logged as template names ("refund-v12") rather than rendered text. Every downstream consumer starves at once: the faithfulness judge has no context to check against, and reviewers open three internal tools per trace to reconstruct the inputs. The fix is mechanical (wrap every tool and retrieval client the way you wrapped the model client), but the discipline is noticing that a trace missing its inputs is not a smaller trace. It's a transcript of half a conversation.