Vibes vs. measurement

Every team starts the same way: type a few prompts, eyeball the answers, ship when it "feels good." That's a vibe check, and it fails for a predictable reason: you test the inputs you can imagine, and your users bring the ones you can't. A demo samples the center of the distribution; production lives in the tails.

Vibe checks also can't answer the only question that matters during iteration: did this change make things better or worse? Swap a prompt, upgrade a model, add a tool: some cases improve, others silently regress. Without a fixed set of cases and a consistent grader, you're comparing today's impression against last week's memory.

The way out is not a giant test suite. It's error analysis: collect real traces of your agent working (a trace is the recorded transcript of one run, every step and tool call included), read them one by one, and write a short note on each failure (what went wrong, at which step). After a few dozen traces, cluster the notes. The clusters are your failure taxonomy, and they tell you exactly which evals are worth building first. Teams that skip this step build evals for failures they imagined; teams that do it build evals for failures they have.

A worked session

Here is what one session looks like for the refund agent. Pull a random sample of recent traces (thirty to fifty, not cherry-picked from complaints) and read each from the top, noting the first thing that went wrong in one line. Don't invent categories before you read; pre-made categories make you see only what you expected to see, and the whole point is to be surprised. Notes first, clusters after. Ten notes from a real-shaped session:

#01  refund issued without checking eligibility first
    #02  asked for the order number the user already gave
    #03  fine
    #04  quoted a "3-5 day" refund window: not in policy, invented
    #05  passed the user's email as order_id, then stalled on the error
    #06  fine
    #07  refused an eligible refund; misread "delivered" as "shipped"
    #08  promised a follow-up call; no callback tool exists
    #09  cited a 14-day return deadline (policy says 30)
    #10  asked for the order number the user already gave

Now cluster. Notes that share a cause get a name, and the names get counts:

Invented policy or capability   3   (#04, #08, #09)
    Ignores info already given      2   (#02, #10)
    Wrong tool arguments            1   (#05)
    Misread a tool result           1   (#07)
    Skipped eligibility check       1   (#01)
    No failure                      2   (#03, #06)

That little table is the payoff. Before the session you might have guessed the agent's problem was tone; the counts say it's making things up about policy, three times in ten traces. So your first eval is a faithfulness check (Module 02 builds the judge for it), your second targets multi-turn context (users shouldn't repeat themselves), and each counted trace becomes a candidate golden case (Lesson 1.3). The mechanism is simple: counting converts opinions about what's broken into a ranked to-do list, and reading is the only way to get the counts.

• Look at your data. Reading 30 real traces teaches you more than any dashboard.
• Count before you fix. "Wrong tool arguments: 14 of 50 traces" turns an anecdote into a priority.
• Keep vibes for hypotheses. Intuition tells you where to look; measurement tells you whether you were right.

Who does it, and how long

Budget the session honestly: thirty traces at two to four minutes each is roughly ninety minutes to two hours of reading, plus half an hour to cluster and count. Do it weekly while the agent is young and monthly once the taxonomy stabilizes. When new traces keep landing in existing clusters instead of opening new ones, you've hit saturation and can stretch the interval. The reader should be the person building the agent, sitting with (or being) someone who knows the domain: for the refund agent, someone who actually knows the refund policy. One consistent reader beats a rotating committee, because clusters drawn by five people with five private definitions never line up. And don't outsource it: the reading is where the team's model of its own failures comes from, and a vendor can't grow that for you. No production traffic yet? Generate synthetic inputs the structured way (Lesson 1.3 shows how), run them through the agent, and read those traces exactly the same way.

Expect this to cost something. Teams that do it well spend a real fraction of engineering time on evals, commonly cited at 10 to 20%, continuously, not as a one-off project. That's not overhead on the product: for an agent, the grading system is part of the product.

Where teams go wrong

Three patterns waste most error-analysis effort. Dashboard-first: the team buys an observability tool, watches aggregate charts, and never opens a trace. But an average latency or a global thumbs-down rate contains no taxonomy; the clusters only exist in the transcripts, and no chart will read them for you. Fix-on-first-failure: trace #1 shows a bug, someone patches the prompt, the session ends. You've fixed a one-of-fifty anecdote while the fourteen-of-fifty cluster ships another week; counting first is the discipline that prevents it. Cherry-picked samples: reading only the traces users complained about skews the taxonomy toward vocal users and misses the silent failures, like the wrong refund that nobody noticed yet. Random sampling is what makes the counts mean anything.