Online evaluation

Lesson 4.4 closed with the idea in one paragraph: sample live traces and score them with the judges you already trust. This lesson is the machinery.

Match the machinery to your volume

First, match the machinery to your traffic. At tens of runs a day, skip everything below and read every trace yourself; you have no volume to sample and no better use of the time. At hundreds, cluster recurring issues. At thousands, sampled judges and drift detection earn their keep, and online A/B tests (Lesson 6.4) only gain statistical power past several thousand runs a day. Building sampling infrastructure for traffic you don't have is procrastination with extra steps.

Start with sampling, because you can't judge everything: at production volume the judge bill would rival the agent's. Sample in two layers: a random slice (say 2 to 5%) as an unbiased baseline, then oversample where the signal is: traces with tool errors, thumbs-down sessions, unusually long trajectories, and segments your offline suite underrepresents. The random slice gives you trends you can trust; the oversampled layers find failures faster.

For the refund agent at 10,000 runs/day:

# Sampling policy: refund agent, ~10,000 runs/day
  # Budget: ~900 judged traces/day (~9% of traffic)
  
  random_baseline:     2% of all runs                     # ~200/day, the trend line
  tool_errors:         100% of runs with a failed tool    # ~80/day, always judge failures
  thumbs_down:         100% of thumbs-down sessions       # ~40/day
  long_trajectories:   50% of runs with > 12 steps        # ~60/day, loops and flailing
  new_query_clusters:  100% of runs in clusters < 14 days # ~120/day, drift candidates
  enterprise_segment:  +5% extra on enterprise traffic    # ~400/day, thin in the offline suite
  
  rule: a trace matching several layers is judged once, tagged with each
  rule: only random_baseline feeds dashboards and alerts

The last rule is the one teams miss: the oversampled layers are deliberately biased toward trouble, so a pass rate computed over them is meaningless as a trend: report trends from the random slice, mine the biased layers for cases. The baseline's size isn't arbitrary either: by Lesson 4.2's square-root rule, ~200 judged traces put roughly ±4 points of daily wobble on the pass rate at the ~90% pass rates you'll typically see (the rule's worst case is ±7).

Judges and alerts

The graders are the same calibrated judges you run offline (Lesson 2.3), running asynchronously, minutes behind the traffic. Same criteria in both places is the point: when online faithfulness drops, you can reproduce the failure in the offline suite with the same grader and debug there. Alert on deltas, not absolutes, with a pre-committed threshold (Lesson 4.3): "page if the 24-hour faithfulness pass rate falls five points below the 7-day mean."

alert: faithfulness_drop
  metric: pass_rate(judge=faithfulness-v3, sample=random_baseline)
  compare: last 24h  vs  trailing 7-day mean
  page_if:   drop >= 5 points
  ticket_if: drop >= 3 points on 3 consecutive days
  never: alert on the absolute pass rate

Why five points? The 24-hour window holds ~200 judged traces (a ±4-point noise band) while the 7-day mean averages ~1,400 and barely moves. A 5-point bar sits just outside the daily wobble (real shifts page you, at roughly one false page a month), and the 3-point ticket catches slow leaks. Why never absolutes: the absolute rate moves with traffic mix: a marketing push bringing easier questions "improves" the agent without a code change. The delta against your own recent past is the only number that isolates the agent from its audience.

Drift: watch the inputs

Watch the inputs, not just the pass rate. Drift is the query distribution shifting under you: a product launch or a new customer segment brings question types your suite never saw, and judges can't flag questions nobody sends them. Cluster incoming queries and compare against your suite's coverage: a growing cluster far from every golden case is next month's incident report. Each such cluster is a batch of candidate golden cases (Lesson 1.3).

Worked example. Your company launches gift cards; nobody tells the agent team. Week one, clustering surfaces a new cluster ("refund my gift card balance", "can I return a gift card") at 1% of traffic; week three it's 4%, and its nearest golden case is a plain product refund, not close. Inside the cluster, the faithfulness judge passes only 71% versus 93% overall, and some passes are wrong too, because the judge has no gift-card rubric to fail them against. What you do: route twenty cluster traces to the review queue (Lesson 6.3); the expert labels them and finds the root cause: the same missing gift-card document Lesson 5.1 diagnosed, so the agent improvises (a Module 05 fix, not a prompt fix); eight labeled traces become golden cases and the rubric gains a gift-card line before the fix ships. Total elapsed: days. Waiting for thumbs-downs instead: weeks.

Guardrails and correction

Online evals have a synchronous cousin: the guardrail.

That budget means a guardrail cannot be your best judge. Use small classifiers, rules, or a fast cheap model asked one narrow question, and hold it to the same standard as any grader. Evaluating the guardrail itself is a discipline of its own, and Lesson 7.3 grades it as a classifier, false positives and all.

There's a third role between scoring and blocking: correction. When a fast check fails a response, retry before the user sees it: regenerate with the failure reason appended, or route the request to a stronger model. Promote an evaluator into this synchronous path only when it clears two bars: latency the user won't feel, and a false-positive rate you've actually measured (Lesson 7.3), because every false positive now costs a visible delay or a blocked good answer.

One concrete retry flow, on the refund agent: before the reply ships, a small model answers one narrow question: "does the drafted reply state the same amount that create_refund was called with?" On a mismatch, regenerate exactly once with the reason appended: "Your draft said $94.00 but the refund issued was $84.00. Correct the amount, change nothing else." If the retry fails the same check, escalate rather than loop. The check adds ~150ms to every response and the retry ~800ms to the ~2% that trip it (invisible against a six-second run), and it converts a visible, trust-burning error into a dashboard line.

Where teams go wrong here: they run a different judge online than offline: a cheaper model with a shortened prompt, to control the bill. Now the two numbers aren't comparable: online faithfulness dips, the offline suite swears everything is fine, and every alert ends in "the judges disagree," which trains the team to ignore alerts. If cost forces a cheaper online judge, calibrate it against the offline one on the same traces; better yet, keep the good judge and shrink the sample. Fewer trustworthy verdicts beat many you have to argue with.