LESSON 5.1 · 24 MIN READ · Module 5: RAG & retrieval evals

The six relationships

The complete RAG eval space

Lesson 1.4 introduced the RAG triangle (question, context, answer) and named one eval per leg. Here's the part that completes the picture: each leg reads in both directions, and the direction changes what you're measuring. Question→context grades the retriever; context→question asks whether the question was answerable at all. That gives six pairwise relationships, and these six are the entire space: every RAG metric in every framework, whatever it's called, is one of them wearing a different name.

Why should direction change anything? Because a directional relationship names a defendant. Question→context puts the context on trial: the question is taken as given, and the grader asks whether the retrieved chunks serve it. Context→question flips the trial: the chunks are taken as given, and the question is asked whether it could be served at all. Same two artifacts, different component under test: the first grades your retriever, the second grades your corpus. Every metric in this module works this way: hold one artifact fixed as the standard, interrogate the other, and the verdict lands on the component that produced the artifact under interrogation. That mechanism is what makes the six a debugging tool rather than a taxonomy for its own sake.

The six, and the failure each catches

CONTEXT RELEVANCEQuestion→context. Do the retrieved chunks address what was asked? The retriever's grade; Lesson 5.2 is the deep dive. Refund-agent failure it catches: "can I return a damaged blender?" retrieves the warranty FAQ and the shipping-rates page: on-topic for the store, useless for the question.
ANSWERABILITYContext→question. Could anyone answer this question from these chunks? When the customer asks about a policy the docs never cover, the right behavior is "I don't know"; grade that the agent said so instead of improvising. Catches: gift-card refund questions answered with a confident, invented policy, because no document covers gift cards at all.
FAITHFULNESSContext→answer. Does the answer claim only what the context supports? Lesson 5.3 is the deep dive. Catches: the reply that tells the customer "refunds take 3 to 5 business days" when no retrieved chunk states any timeframe.
SUPPORT COVERAGEAnswer→context. Read back from the answer: is every claim traceable to a specific retrieved chunk? Same pair as faithfulness, opposite direction; this is the citation and attribution eval. Catches: a reply that cites "per our returns policy §2" for a sentence that actually came from the model's training data, not from any chunk in the prompt.
ANSWER RELEVANCEQuestion→answer. Does the answer address what was actually asked? The end-to-end grade: everything upstream can be right and this can still fail. Catches: "how do I send this back?" answered with a faithful lecture on refund eligibility, every claim supported and the customer still without a shipping label.
SELF-CONTAINMENTAnswer→question. Does the answer stand alone without the question beside it? "Yes, you can" reads fine in the chat window and means nothing when the reply is forwarded to a colleague or quoted in a ticket. Catches: "Yes, that's within the window" pasted into the support ticket for order #4021, where nobody reading it knows which order or which window.

Two of those rows look like duplicates, so answer the reader's objection now: faithfulness and support coverage share a pair but not a job. Faithfulness asks whether support exists: every claim must be backed by something in the context, wherever it sits. Support coverage demands a pointer (this claim, that chunk), which is why it's the eval behind citations. A reply can pass faithfulness while failing support coverage: every sentence loosely supported by the context as a whole, no sentence traceable to the specific section the reply cites. If your product shows sources, you need the pointer, not just the existence proof.

Start with three; hold three in reserve

Don't treat the six as a to-do list. Start with the three primary ones: context relevance, faithfulness, answer relevance, one per pair, the three Lesson 1.4 named. The reverse directions are diagnostics: you add them when a primary metric fails and you need to know why. Faithfulness dropping and answerability dropping means the model improvised because the context was thin: a retrieval problem dressed up as a hallucination problem, and a prompt fix would have been aimed at the wrong component.

Where teams go wrong is the opposite instinct: switch on every metric their eval framework offers, on day one. Now there are six dashboards (more, in practice, because frameworks ship the same relationship under two different names and both get tracked as if they were independent signals). Nobody can say what "context precision 0.71" asks of them on Monday morning, so the numbers get glanced at and never drilled into, and the team is back to vibes, now with charts. Three primary metrics you act on beat six you scroll past, and the reverse directions earn their keep precisely because you don't run them continuously. They're the questions you ask a failure, not the weather you check.

A diagnosis, walked end to end

Here's the diagnostic flow on a real-feeling failure. Monday's dashboard for the refund agent:

faithfulness (claim-level, Lesson 5.3)   0.91 -> 0.78   over one week
  context relevance                        0.84 -> 0.83   flat
  answer relevance                         0.88 -> 0.87   flat
  
  drill-down: run answerability on the 42 failing traces
    answerable from the retrieved chunks:  31%   (suite baseline: 82%)
    dominant query cluster: "refund a gift-card purchase"  (29 of 42)
  
  corpus check: no document covers gift-card refunds

Read it in order. Faithfulness dropped, so the reflex diagnosis is hallucination: tighten the prompt, add a sterner instruction about sticking to the context. But context relevance is flat: the retriever is returning the same quality of chunks it always did, so nothing upstream of the prompt changed. That's your cue to run the reverse diagnostic on the failing traces, and answerability collapses. For two-thirds of them, nobody, model or human, could have answered from what was retrieved. The query cluster names the cause: gift cards launched two weeks ago, the support docs were never written, and the model has been improvising policy to fill the vacuum.

Conclusion: a content gap wearing a hallucination costume. The durable fix is a document; the interim fix is an agent that says "I don't know" about gift cards, exactly the behavior the answerability row grades. The prompt fix you were about to write would, at best, have converted confident inventions into apologies, and the dashboard would have called that a win. This is the whole argument for the six relationships in one incident: the failure surfaced on one leg, but the cause lived on another, and only the decomposition let you follow it home.

The payoff of exhaustiveness is confidence in your taxonomy. When a RAG trace fails, it fails along one of these six dimensions, so "the answer was bad" always decomposes into a named, measurable relationship. Lesson 5.4 turns that into a debugging routine.

KEY IDEA

Every RAG eval is one of six directional relationships among question, context, and answer.