Faithfulness, step by step

The faithfulness judge in Lesson 2.2 returns one verdict for the whole reply. That works until the reply is 90% right: three sentences straight from the policy and one invented timeframe. A whole-answer verdict either flunks a mostly-good answer or waves the fabrication through, and either way it doesn't say which sentence to fix. The production-grade version grades claims, not answers:

• 1. Decompose. Extract the answer's atomic claims: standalone factual statements, one fact each, pronouns resolved so every claim is checkable on its own.
• 2. Verify. Check each claim against the retrieved context: a narrow judge per claim ("is this claim supported by the context, yes or no?") or an NLI model scoring entailment.
• 3. Score. Faithfulness = supported claims ÷ total claims.

reply: "You're eligible for a refund on order #4021. Refunds
    take 3-5 business days, and you'll get a confirmation email."
  
    claims:
      1. Order #4021 is eligible for a refund.   -> supported (policy §2.1)
      2. Refunds take 3-5 business days.         -> UNSUPPORTED (no timeframe in context)
      3. A confirmation email will be sent.      -> supported (policy §2.4)
  
    faithfulness = 2/3

The decomposition prompt

Step 1 is its own model call, and its prompt deserves the same care as any judge prompt (Lesson 2.2). A working one:

Decompose the RESPONSE into atomic factual claims.
  
  Rules:
  - One fact per claim; split any sentence that asserts several.
  - Resolve pronouns and references from the conversation:
    "it" -> "order #4021", "your card" -> "the customer's Visa".
  - Keep conditions attached: "if the item arrived damaged, a full
    refund applies" is ONE claim, not "a full refund applies".
  - Drop non-claims: greetings, apologies, offers to help, and
    statements of opinion or empathy.
  
  <response>{agent_response}</response>
  
  Return JSON: {"claims": ["...", "..."]}

Opinions, conditionals, multi-fact sentences

Each rule exists because a naive decomposer fails on a specific sentence shape. Opinions and pleasantries: "We're so sorry your blender arrived damaged" asserts nothing checkable; extract it as a claim and it lands as unsupported (the context says nothing about sorrow), quietly deflating every score with noise that has nothing to do with hallucination. Drop them. Conditionals: strip the condition from "if the item arrived damaged, you're eligible for a full refund" and you're left grading "you're eligible for a full refund", a stronger, unconditional statement the context may not support, so a true sentence gets flunked, or a false unconditional promise passes because the policy happens to contain the conditional's words. The condition is part of the fact; keep them together. Multi-fact sentences: "refunds take 3 to 5 business days and go to your original payment method" is two claims. Graded as one, a fabricated timeframe drags a correct payment-method fact down with it; worse for diagnosis, you can no longer see that it's always the timeframe half that's invented.

The decomposer is where teams go wrong by omission: it's a model component that determines the denominator of your headline metric, and almost nobody evals it. An over-splitter that shreds one fact into three claims moves your faithfulness score with no change in the agent; an under-splitter hides fabrications inside compound claims. The fix is Lesson 2.3 in miniature: hand-decompose twenty real replies yourself, run the decomposer on the same twenty, and reconcile the differences before you trust a single downstream number.

Claim-level grading buys three things. Partial hallucination becomes visible: 2/3 is a different signal than a coin-flip pass/fail on a borderline answer. Failures become diagnosable: the unsupported claim is quoted in the output, so error analysis writes itself; you can see it's always timeframes the agent invents. And the judge's job shrinks to one claim, one context, one binary verdict, exactly the narrow shape that Lesson 2.1 says calibrates best.

NLI or judge for the verify step

Step 2 offered a choice (a narrow judge per claim, or an NLI model scoring entailment), and it's a genuine cost/quality trade, the small-classifier rung from Lesson 2.1 made concrete:

In practice you layer them: cheap and strict in front, expensive and careful behind, the same architecture Lesson 7.3 uses for guardrails. The NLI model scores everything; claims it's confident about, in either direction, are settled; the uncertain middle band routes to the judge. You get judge-quality verdicts on the hard cases at classifier prices on the easy ones, and most claims are easy.

Walking a borderline claim

Borderline cases are where the definition, not the model, decides, so walk one. The reply says: "You'll receive your refund within a week." The context says: "Refunds are processed within 3 to 5 business days." Supported? A strict reading says no: "week" appears nowhere, and five business days can span seven calendar days plus a weekend; the claim smuggles in a promise the policy doesn't quite make. A generous reading says yes: any reasonable customer hears them as the same commitment. An NLI model typically lands on "neutral," which your pipeline has to map to a verdict anyway; a judge will go whichever way its prompt leans. The resolution is to stop asking the model and start asking your product. Decide the rule (say, paraphrase and unit conversion count as support; added specifics and rounded-up promises do not), write it into the judge's definition, and file this exact case into the calibration set so the line you drew still holds after the next prompt edit. A borderline claim adjudicated once is a criterion made sharper; a borderline claim re-litigated every month is noise.

Notice what this pipeline never needed: a golden answer. The retrieved context is the reference, so the metric is reference-free: the same property as the synthetic retrieval suite in Lesson 5.2, and the founding idea of the RAGAS framework, which built faithfulness (via this statement decomposition), answer relevance, and context relevance to run without ground-truth labels. The consequence is reach: reference-free metrics run on any trace, including live production traffic, not just the cases you curated.

Know the limit, though. Faithfulness checks the answer against the context, not against the world. If retrieval surfaced last year's returns policy, a perfectly faithful answer is faithfully wrong. Reference-free metrics give you scale; a small golden set with known-correct answers (Lesson 1.3) still anchors correctness. Run both.