Human review workflows

Judges scale grading, but humans stay at the top of the ladder (Lesson 2.1), and in production you can't read everything. A review queue is the answer: sampled traces routed to a domain expert who spends fifteen minutes a day on them. It's Lesson 2.4's weekly spot-check, made systematic and fed by live traffic.

What belongs in the queue

What belongs in the queue:

• Judge-flagged fails: confirm or overturn the verdict; every confirmed fail is a labeled example for free.
• User signals: thumbs-down, abandoned sessions, escalations to a human agent. Lagging metrics (Lesson 1.4), but each one points at a trace worth reading.
• High-stakes actions: every refund above a threshold, every account change, regardless of what any grader said.
• A random baseline: a small unconditional slice. It's the only sample that can show you what your judges are missing.

Those four sources become routing rules, and writing them down as configuration, not tribal knowledge, is what keeps the queue honest when traffic doubles:

queue: refund-agent-review     # target: ~25 traces/day, one reviewer, 15 min
  route:
    - if: judge_verdict == fail                        -> priority: high,   cap: 10/day
    - if: user_feedback == thumbs_down                 -> priority: high,   cap: none
    - if: tool_called == create_refund && amount > 200 -> priority: high,   cap: none
    - if: escalated_to_human                           -> priority: medium, cap: 5/day
    - if: random(1%)                                   -> priority: low,    cap: 5/day
  dedupe: one trace per session per day
  order: high first, then oldest
  overflow: shed low-priority first; never shed high-stakes actions

The caps are the load-bearing part. A queue sized to the reviewer's fifteen minutes gets emptied daily; an uncapped queue becomes a four-hundred-item inbox that the reviewer samples arbitrarily, then avoids, then abandons. Note what's uncapped: thumbs-downs and large refunds, because those are the rows where missing one is expensive. Everything else degrades gracefully under load: the judge-fail cap just means the judges' worst ten of the day, which is the right ten to read.

Fifteen minutes, end to end

Whether reviews actually happen is decided by friction. The tool that works shows one screen with everything the reviewer needs (the trace, the retrieved policy, the customer's order history), asks one question ("is this reply faithful, y/n?"), and moves on a keystroke. This almost always means building your own: generic annotation tools can't render your domain's data, so reviewers open three tabs per trace and soon review nothing. A custom view in a lightweight app framework takes about a day and pays for itself in throughput.

Here's the session in practice. 9:05, the support lead opens the queue; the counter says 23. The screen is one trace: left pane, the conversation with tool calls rendered inline (create_refund(4021, $84.00) → rf_5512); right pane, the two policy chunks the agent retrieved and the customer's order record; top bar, the judge's verdict and its one-line reason ("reply promises expedited processing, not in policy"). At the bottom, one question: Is this reply faithful to the policy shown? Reading takes thirty to sixty seconds because everything is on the screen: no tab-hopping, no log-grepping. Then a keystroke: y or n; n opens a one-sentence reason box and an optional tag (wrong-amount, invented-policy, tone); g flags the trace as a golden-case candidate; enter loads the next. Twenty-odd traces fit in the fifteen minutes, and each verdict is routed the moment the key goes down: a y on a judge-flagged fail is an overturn; it lands in the judge's calibration set as a false alarm. An n is a confirmed fail, a labeled example, and if flagged with g, the trace plus the reason become a drafted golden case with the expectation pre-filled. Verdicts on the random slice feed one number nothing else can produce: how often the judges pass traces a human would fail.

Every review has two exits

A review that ends at a label is half-finished. Every reviewed trace has two exits: it becomes a golden case (Lesson 1.3), where a new failure mode joins the suite before the fix ships, or a judge-calibration example (Lesson 2.3), especially when the human overturned the judge; disagreements are the most valuable rows in a calibration set. Track that disagreement rate per criterion, too. When it rises, either the judge drifted or your standards did (Lesson 4.4); recalibrate either way.

The disagreement dashboard is deliberately dull: one small chart per criterion, human-vs-judge disagreement rate by week, split into two lines: "judge failed it, human passed it" (false alarms) and "judge passed it, human failed it" (misses, computable only from the random slice, which is why that slice exists). Two stories it tells. Story one: faithfulness disagreement climbs from 4% to 15% over three weeks, entirely on the false-alarm line, and the climb starts the exact week a reworded judge prompt shipped: the judge drifted; roll the prompt back or recalibrate it against the pile of labeled overturns the queue has conveniently been accumulating. Story two: the tone criterion's disagreement climbs but nothing about the judge changed; the reviewer started failing double-apology replies they used to pass, after a team decision that groveling reads worse than a plain fix. The standard moved. The judge isn't wrong; it's enforcing last quarter's taste. Update the rubric line, update the judge prompt, and re-anchor on freshly labeled examples. Same symptom, opposite causes, and the split line is what tells them apart.

One named owner

Finally, give the eval system one named owner, usually the domain expert already doing the labeling, because whoever defines pass and fail owns quality. Engineers own the harness, the CI wiring, and the dashboards; the owner owns the rubric, the golden set, and the final word on disagreements. Evals without an owner decay the way untested code does: nobody is wrong when the suite goes stale, so it does.

Where teams go wrong: they build the queue and skip the exits. Verdicts accumulate in a spreadsheet nobody reads back into the system (no golden cases cut, no calibration examples filed, no disagreement rate computed), and after a month the reviewer, quite rationally, concludes the labels change nothing and stops. The symmetrical failure is over-asking: twelve questions per trace instead of one, five minutes per review instead of forty seconds, and the same abandonment by a different road. The fixes are the same discipline seen twice: one question per trace, and a visible pipeline from every keystroke to a case, a calibration row, or a dashboard the team actually watches. Reviewers keep reviewing when they can see their labels land.