When a model should grade a model

The ladder of graders

LLM-as-judge means using a strong model, with a careful prompt, to grade your agent's output. It's the tool that makes evals scale past what you can hand-review, and it's overused. The rule: climb the ladder of graders and stop at the cheapest rung that works.

The ladder is ordered by more than price. A code check is deterministic: when it flips from pass to fail, the agent changed, full stop. A judge is a second stochastic model, and its errors are the expensive kind: confident, systematic, and invisible until you compare them against human labels. Every judge you add is a component you now have to calibrate (Lesson 2.3), monitor for drift (Lesson 2.4), and pay for on every run. Skip the ladder and grade everything with judges, and your eval suite becomes exactly as unreliable as the agent it's supposed to measure; you'll spend debugging time asking whether the agent regressed or the grader did.

The economics compound the argument. A judge verdict costs a model call, so a 200-case suite with three judged criteria is 600 model calls per run. The same checks in code finish in milliseconds for free, and the suite cheap enough to run on every prompt edit is the one that actually catches regressions.

Six qualities, six decisions

Walk the refund agent through the ladder, one quality at a time. The question is the same every time: what is the cheapest rung that can actually see this quality?

The pattern behind the table: anything about structure (IDs, formats, tool calls, dates, thresholds) belongs to code, and anything about the meaning of prose belongs to a judge. The policy-compliance row is the one to internalize, because it's the shape most real qualities have: a hybrid, where the discipline is carving out every code-checkable piece before you write a judge prompt for the remainder.

Two graders you'll meet in every eval library sit between these rungs, and only one deserves a place. Similarity metrics (word-overlap scores and embedding cosine against a reference answer) look like code checks but grade the wrong thing: they reward sounding like the reference, not behaving correctly, and a reply can be fluent, similar, and still refund the wrong order. Skip them everywhere except inside retrieval itself, where embedding similarity is doing its native job (Module 05). Small learned classifiers (a fine-tuned model or entailment checker scoring one narrow property) are a real rung between code and judge: once you have a few thousand labels they're faster and cheaper than an LLM judge, and you calibrate them exactly the same way (Lesson 2.3).

Verifiable proxies

A surprising amount fits on the code rung if you look for a verifiable proxy. Instead of asking a judge "is this SQL right?", execute it and compare result sets. Instead of "did the agent fix the bug?", run the repo's tests; that's how the strongest coding benchmarks grade, and it's the pattern to steal: check the world, not the prose.

Two more worked proxies, both from the refund agent. First, the refund amount: "did the agent refund the right amount?" sounds like a judgment about the reply, but your seeded sandbox knows the order total and your policy knows the rules, so recompute the correct amount inside the test and assert the refund row matches it. Second, policy grounding: instead of a judge deciding whether the reply "reflects the current returns policy," have the agent cite policy section IDs and check with plain string matching that every cited section exists in the retrieved chunks. Neither proxy grades prose; both grade facts the environment can verify. That's the mechanism: a proxy converts "is this text right?" into "is the world right?", and the world, unlike prose, has a ground truth your test can read directly.

Reach for a judge when the quality is real but fuzzy, and even then split the fuzzy question into narrow ones first. "Rate this response 1 to 10" is a bad judge task. "Does the response claim anything not supported by the provided context, yes or no?" is a good one. One judge, one criterion, one binary verdict.

case: "Refund order #4021, it arrived damaged."
    graders:
      code:                                # run first, free, deterministic
        - json_schema: refund_reply.schema.json
        - tool_called: create_refund(order_id=4021)
        - refund_amount_equals: orders_db[4021].total   # verifiable proxy
        - order_within_window: orders_db[4021].date     # policy, code half
      judge:                               # one criterion each, binary
        - faithfulness_to_policy
        - completeness
        - tone_professional

Note the order in that config: code checks run first, and a case that fails them never reaches the judges. That's not just thrift: an invalid reply doesn't need its tone graded, and keeping judges off garbage keeps their verdicts interpretable.

Where teams go wrong

Three patterns account for most wasted grading budgets. Judge-first defaulting: the team wires an LLM judge for everything, including JSON validity and ID matching, because one integration covers all cases, and then wonders why the suite is slow, expensive, and a few percent noisy on checks that should be deterministic. The mega-judge: one prompt asks for correctness, tone, and completeness and returns a single 1 to 10 score; when the score drops nobody knows which quality moved, and the judge can't be calibrated because no human label corresponds to one number that means three things. Reference-answer comfort: grading every case by similarity to a stored golden reply, which quietly converts "did the agent do the right thing?" into "did the agent phrase it like last month?", and starts failing correct answers the day you improve the phrasing.