Measuring retrieval

Retrieval is the most measurable component in your whole agent: no judge, no rubric, just labeled data and arithmetic. A label is a query paired with the chunk(s) that answer it. Then recall@k (of the relevant chunks, how many came back in the top k) and precision@k (of the k returned, how many were relevant). For the refund agent: "can I return a damaged blender after 30 days?" has one relevant chunk, the damaged-items section of the returns policy. Either it came back or it didn't.

The arithmetic, on five queries

Here's the method at toy scale. Five labeled refund-agent queries, k=5, with your labeled chunk IDs on the left and what actually came back on the right:

query                             relevant   top-5 returned         recall  prec.
  Q1  damaged blender, day 34        c12       c12 c07 c31 c02 c44     1/1    1/5
  Q2  where is order #4021           c88       c90 c91 c17 c88 c03     1/1    1/5
  Q3  refund to original card?       c15 c16   c15 c40 c41 c42 c43     1/2    1/5
  Q4  return without a receipt       c51       c60 c61 c62 c63 c64     0/1    0/5
  Q5  refund an international order  c23 c24   c23 c24 c09 c10 c11     2/2    2/5
  
  recall@5    = (1.0 + 1.0 + 0.5 + 0.0 + 1.0) / 5 = 0.70
  precision@5 = (0.2 + 0.2 + 0.2 + 0.0 + 0.4) / 5 = 0.20

Walk the columns. Recall is computed per query, then averaged. Q3 found one of its two relevant chunks, so it contributes 0.5: the answer will be built on half the policy, which is exactly how confident half-answers happen. Q4 is a clean miss and contributes zero. The average, 0.70, sounds respectable until you notice it means one query in five got nothing useful at all. Precision reads worse than it is: with only one or two relevant chunks in existence, k=5 caps Q1's precision at 1/5 no matter how well the retriever does. That's normal: at small k over sparse labels, read precision as a trend line (is the prompt filling with junk?) and treat recall as the primary number. And even this tiny aggregate hides shape: 0.70 is three perfect queries, one half, and one zero, a preview of the segmentation rule below.

Rank matters more than the definitions suggest. Models attend unevenly to a stuffed prompt: a relevant chunk buried in the middle of twenty is technically recalled and practically invisible. Two habits keep the metric honest: set k to the number of chunks you actually put in the prompt, not what the index can return; and when you're tuning a reranker, use a rank-aware measure like MRR (on average, what position does the first relevant chunk land at?). Recall@20 can hold steady while the answer quality collapses, because position 19 is where relevance goes to be ignored.

MRR, worked on the same five

MRR makes the worked example one line longer. For each query, find the rank of the first relevant chunk and take its reciprocal: Q1 ranks it 1st (1.0), Q2 ranks it 4th (0.25), Q3 1st (1.0), Q4 nowhere in the top five (0), Q5 1st (1.0). Mean reciprocal rank: (1.0 + 0.25 + 1.0 + 0 + 1.0) / 5 = 0.65. The reciprocal is the mechanism worth understanding: slipping from rank 1 to rank 2 costs half the score, while slipping from rank 9 to 10 costs about a hundredth; the metric cares intensely about the top of the list and barely at all about the bottom, which mirrors how a model actually reads a crowded prompt. Q2 is the payoff: recall@5 scores it a perfect 1.0, MRR scores it 0.25, and MRR is telling the truth. The order-status chunk is technically present and practically buried under three chunks about other things.

Synthetic questions from your own corpus

The labeling bottleneck has a standard escape: generate the questions from the corpus. For each chunk, have a model write a question that this chunk answers. The (question, chunk) pair is a labeled case by construction: does retrieval return chunk X for the question generated from chunk X? No golden answers, no annotators, thousands of cases in an afternoon, and each eval runs in milliseconds because nothing is generated. The same move powers research-grade frameworks: ARES generates synthetic queries and fine-tunes lightweight judges on them to score RAG systems with almost no hand labeling.

The generation prompt is where the quality lives; here's a working one, with three pairs it produced:

For the document chunk below, write ONE question a customer might
  ask that this chunk (and only this chunk) answers.
  Phrase it the way a real customer would: first person, informal,
  no section titles, no vocabulary copied verbatim from the chunk.
  Return JSON: {"question": "..."}
  
  <chunk id="c12" source="returns-policy.md §2.3">
  Items that arrive damaged may be returned for a full refund
  within 60 days of delivery, twice the standard 30-day window.
  </chunk>
  
  --- generated (question, chunk) pairs ---
  ("My blender showed up cracked, can I still send it
    back after a month?",                                    c12)
  ("Do refunds go back to the card I paid with, or do
    I just get store credit?",                               c15)
  ("I lost my receipt. Any way to return the toaster
    I bought last week?",                                    c51)

Each guard fights a degeneration: "only this chunk" keeps the label unambiguous (a question four chunks could answer makes recall ill-defined), and the phrasing rules attack the weakness named next: a generator left alone lifts the chunk's own phrasing, and embedding search trivially finds a chunk from its own words.

One caveat keeps you honest: synthetic questions are easy, because they're phrased in the chunk's own vocabulary. Expect recall in the high nineties, and treat the suite as a smoke test rather than a quality bar. If synthetic recall is 70%, something basic is broken (chunking, indexing, an embedding mismatch) and you should fix that before anything clever. Real production queries, phrased in customer language, are the actual bar; swap them in as they arrive (Lesson 1.3's "failure in, case in" habit applies to retrieval too).

Where teams go wrong with all of this is measuring a system nobody ships. Recall@50 against everything the index can return, while the prompt gets the top five: the metric passes, the model never sees the chunk. A synthetic suite promoted from smoke test to quality bar, so the dashboard says 96% while customers phrase questions no generator imagined. A reranker tuned on plain recall@20, a metric that literally cannot see the one thing a reranker changes. The common thread: every retrieval metric is a claim about what the model will actually see, so parameterize it by the prompt you actually build (k as shipped, questions as customers phrase them, rank measured whenever rank is what you're tuning).

Finally, never read one aggregate number. Tag queries by class, topic (returns policy, order status, shipping) and capability (lookup, comparison, multi-document synthesis), and compute recall per class. An 89% aggregate can hide order-status queries at 55%, and it's the breakdown, not the average, that tells you what to fix. Lesson 5.4 builds the improvement loop on exactly this table.