Writing judge prompts

Anatomy of a judge prompt

A judge prompt is a spec, and vague specs produce vague verdicts. The anatomy of one that works:

• One criterion. Don't ask for correctness, tone, and completeness in one pass; run three narrow judges. Narrow judges are easier to calibrate and their failures are easier to diagnose.
• A concrete definition. Spell out what pass and fail mean for your product. "Faithful = every factual claim is supported by the provided context" beats "faithful = accurate".
• Labeled examples. Two or three real pass cases and fail cases, with one line each on why. These do more than any instruction.
• Reasoning before verdict. Ask for a short critique first, then the label. Judges that must explain first agree with humans more often.
• Structured output. A fixed format, {"reasoning": "...", "verdict": "pass"|"fail"}, so results parse reliably into your eval harness, the code that runs cases and records scores.

Each element closes a specific failure. Reasoning-before-verdict works because it forces the judge to commit to evidence before committing to a label: a judge that must quote the unsupported claim can't wave the reply through, and when it's wrong, the written critique shows you exactly how it misread the criterion, which is what makes calibration (Lesson 2.3) debuggable rather than merely measurable. Structured output closes a quieter failure: a judge that answers in prose gets parsed with regex, the regex misses one phrasing in twenty, and those verdicts silently vanish; the score you report becomes partly a parsing artifact.

Version one, and what calibration finds

Nobody writes the good prompt first. Here is the version everyone actually writes first:

Is the RESPONSE faithful to the CONTEXT?
  
    <context>{retrieved_context}</context>
    <response>{agent_response}</response>
  
    Answer "pass" or "fail".

It reads fine, and it will happily return verdicts all day. Then you calibrate it against fifty human-labeled refund replies (Lesson 2.3), and the disagreements cluster into one pattern. Your labeler failed a reply that said "refunds take 3-5 business days" because the retrieved policy never states a timeframe; the judge passed it, reasoning that the claim was "accurate and standard for refund processing." The judge wasn't broken; it was answering a different question. "Faithful" was never defined, so it fell back on world-knowledge plausibility. And plausible inventions are exactly the hallucinations that matter, because human reviewers skim past them too.

The v2 fix is surgical, not a rewrite: define faithfulness as support by the context, say explicitly that unsupported claims fail even if plausibly true, and turn the calibration disagreement itself into the fail example.

A complete judge prompt, ready to adapt:

You are grading one criterion: FAITHFULNESS.
    Definition: every factual claim in the RESPONSE must be supported
    by the CONTEXT. Unsupported claims = fail, even if plausibly true.
  
    <context>{retrieved_context}</context>
    <response>{agent_response}</response>
  
    Example fail: response says "refunds take 3-5 days" but the
    context never states a timeframe.
    Example pass: response only restates policies present in context.
  
    First write a 2-3 sentence critique quoting any unsupported claim.
    Then output JSON: {"reasoning": "...", "verdict": "pass"|"fail"}

Trace each line back to its origin: the definition line kills the plausibility loophole, the fail example is the calibration disagreement almost verbatim, and the critique-first instruction produces the quoted evidence that made the disagreement diagnosable in the first place. That's the iteration loop in miniature. You don't improve a judge prompt by wordsmithing; you improve it by finding a real disagreement and encoding its lesson.

The pattern generalizes: completeness

A second criterion, built on the same skeleton. Completeness fails differently than faithfulness (the customer asks two things and gets one answer), so the judge needs the user's message, not the retrieved context:

You are grading one criterion: COMPLETENESS.
    Definition: the RESPONSE must address every distinct request in
    the USER MESSAGE. Ignoring or silently deferring a request = fail.
    Extra content does not compensate for a missed request.
  
    <user_message>{user_message}</user_message>
    <response>{agent_response}</response>
  
    Example fail: user asks for a refund AND a status update on a
    second order; response handles the refund, never mentions the
    second order.
    Example pass: response processes the refund and says the second
    order ships Tuesday.
    Example pass: response processes the refund and says it cannot
    check the second order but offers a way to find out; declining
    a request explicitly still counts as addressing it.
  
    First write a 2-3 sentence critique listing each request and
    whether it was addressed.
    Then output JSON: {"reasoning": "...", "verdict": "pass"|"fail"}

Everything transferable stayed fixed: one criterion, a concrete definition, examples with the reason attached, critique before verdict, the same JSON shape. Everything criterion-specific changed: the inputs (user message instead of context), the definition, and the examples, including a second pass example teaching the judge that "we can't do that, here's who can" counts as addressing a request, a boundary its own calibration run exposed. Once the skeleton is in your head, a new criterion costs you a definition and three examples, not a fresh design.

Where teams go wrong

The kitchen-sink prompt: one judge asked to check faithfulness, tone, and completeness in a single pass. It returns one verdict, so a fail doesn't say which criterion tripped, and calibration is impossible, because your human labels are per-criterion and the judge's aren't. Invented examples: pass/fail examples written from imagination rather than pulled from calibration disagreements, teaching the judge to catch failures your agent never produces while missing the ones it does. Silent edits: someone "clarifies" one word in the definition, verdicts shift across the whole suite, and the dashboard reports an agent regression that never happened.

Run the judge at temperature 0: no sampling randomness, so verdicts are repeatable. Default to a judge at least as strong as the model being graded (weaker judges are noisier), though a well-calibrated smaller judge on a narrow binary criterion can earn its keep; that's what the calibration set is for. Then treat the prompt as code: version it, and re-run your calibration set (next lesson) whenever you touch it, because a one-word edit can shift verdicts across your whole suite.

Two reader questions come up every time. How many examples? Two or three per side; past that you crowd the context, and judges start matching surface features of the examples instead of applying the definition. And which model? Start strong, then let the agreement numbers from calibration, not the price list, tell you whether a cheaper judge holds.