Choosing what to measure

You cannot eval everything, and trying to is how teams end up with a wall of metrics nobody trusts. Choose by two questions: what does the agent do most often, and what is most expensive when it goes wrong? The intersection, your critical paths, is where evals earn their keep.

A worked session: the refund agent

Run the two questions as an actual exercise, not a slogan. Pull a week of traffic and list what the agent spends its time on, with rough shares. Then, for each path, name the worst realistic failure and its blast radius (what it costs when it happens):

PATH                      TRAFFIC   WORST FAILURE, BLAST RADIUS
    process a refund            41%     wrong or duplicate refund: money out
                                        the door, unrecoverable
    answer a policy question    27%     invented policy: user acts on it,
                                        trust and maybe legal exposure
    check order status          18%     wrong order's status: mild confusion,
                                        user re-asks
    update shipping address      9%     silent no-op: package goes to the
                                        old address
    everything else              5%     (long tail)

Now take the intersection. Refunds and policy questions are both frequent and expensive; those are the critical paths, and they get eval coverage first. Order status is frequent but its failure is cheap, so it gets a couple of golden cases and no more. The address change is rarer but its failure is silent and costly, so it earns a targeted eval on the write action even at 9% of traffic. Frequency alone would have you polishing status checks; cost alone would have you gold-plating rare disasters; you need both axes.

Make the metrics task-specific. Generic scores ("helpfulness: 7.2") don't tell you what to fix. Decompose your agent's job and measure each part in its own terms:

Notice that the table is derived from the critical paths, not brainstormed. The refund path is tool calls all the way down, so it exercises the TOOL USE row (right arguments, exactly one refund) and the SAFETY row (no refund without eligibility). The policy-question path is read-then-answer, so it exercises RETRIEVAL (did the right policy page come back) and ANSWER (is the reply faithful to it). Every metric traces back to a named path and a named failure, and that traceability is the test to keep applying: if you can't say which user pain a metric prevents, cut the metric.

For RAG specifically, the first and third rows above combine into a standard shape: a triangle of question, context, and answer, with each pair its own eval. Question→context is retrieval quality, context→answer is faithfulness (no claims beyond the context), question→answer is answer relevance (it addresses what was asked). Together they localize the failure: bad retrieval means fix the index; faithful-but-irrelevant means fix the prompt, not the retriever.

Leading and lagging

Separate leading metrics from lagging ones. Lagging metrics (user thumbs-down, task completion in production) tell you how you're doing but arrive late and explain nothing. Leading metrics (retrieval recall, tool-call accuracy, judge pass-rate on golden cases) move the moment you change something and point at the broken component. Build your suite from leading metrics; watch the lagging ones to confirm the suite still tracks reality.

Count your own pace among the leading metrics, too: experiments run per week, minutes to run the suite. A slow eval suite is a leading metric going the wrong way.

A vanity metric, dissected

Here is the counterexample to keep in mind. Suppose the team adds "average politeness, 1 to 10, judge-rated" to the dashboard. It climbs from 8.2 to 8.6 over a month, the chart is green, and it feels like progress. But run it through this lesson's machinery and it fails everywhere: no cluster in the error analysis was about rudeness, so it doesn't sit on a critical path; when it dips, it names no component to fix; and a reply that refunds the wrong order can still score a 9 on politeness, so the expensive failure is invisible to it. That's the definition of a vanity metric: a number that moves without meaning anything you'd act on. The litmus test is one question: what decision changes if this number drops ten points? If nobody has an answer, delete the metric before someone starts optimizing it.

Where teams go wrong

The recurring failure patterns here are about selection, not construction. The metric wall: thirty numbers on a dashboard, none gating a release. When everything is measured, nothing is trusted, and the team stops looking; a handful of metrics that block merges beat a wall that decorates. Grading what's easy: the suite fills up with format and latency checks because code can grade those for free, while faithfulness (the top cluster in the error analysis) goes unmeasured because it needs a judge; the suite drifts toward what's cheap instead of what's important, which is exactly why Module 02 teaches you to build judges rather than skip the hard qualities. Benchmark chasing: the team tracks its model's public leaderboard scores as if they were product metrics.

On that last one: resist public-benchmark worship. Leaderboard scores tell you a base model's general capability; they say almost nothing about your agent on your tasks. Benchmarks are for choosing a model. Your evals are for shipping your product.

How many metrics should you end up with? One per critical-path component you would actually act on. For the refund agent worked above, that's roughly six to eight: tool-call accuracy and final-state checks on refunds, retrieval recall and faithfulness on policy answers, a couple of safety checks, and suite runtime. Not thirty.