The retrieval flywheel

You now have a metric for every leg of the triangle. The flywheel is how they change your product. Start where Lesson 1.2 starts (pull failing traces and read them) but with the triangle as your checklist. For each failure, ask in order: does the needed chunk exist in the index at all? Did retrieval return it, high enough to matter? Did the model use it faithfully? Did the answer address the question? The first "no" localizes the failure, and each location has its own fix:

Segment, then do the arithmetic

Then segment. Cluster production queries by topic and capability, compute each leg's metric per cluster, and prioritize by volume × failure rate: a cluster that's 40% of traffic at 60% recall matters more than a 2% cluster at zero. For the refund agent, returns-policy questions might sit at 95% recall while the "where is my order" cluster fails, because order-history chunks are formatted as tables the embedder mangles. That's a concrete, fixable diagnosis you would never extract from an aggregate score.

One full turn of the flywheel

Here's a whole round, with numbers shaped like the ones you'll actually see. Production queries clustered by topic and capability, per-segment metrics computed overnight:

segment                        volume   recall@5   faith.   ans.rel.
  returns policy / lookup          38%      0.95      0.97      0.94
  order status   / lookup          31%      0.61      0.90      0.71
  eligibility    / comparison      14%      0.88      0.82      0.89
  shipping       / lookup           9%      0.92      0.95      0.93
  gift cards     / lookup           5%      0.34      0.58      0.49
  multi-order    / synthesis        3%      0.71      0.86      0.68

Now the arithmetic. Per 1,000 queries, expected retrieval failures = volume × failure rate. Order status: 310 queries × 39% missed ≈ 121. Gift cards, despite the far worse rate: 50 × 66% ≈ 33. Multi-order synthesis: 30 × 29% ≈ 9. The eye goes to gift cards' ugly 0.34; the math says order status is nearly four times the pain. Reading its failing traces confirms the diagnosis the segment paragraph predicted: order-history chunks are tables, the embedder mangles them, and "where is my order" retrieves everything except the customer's actual orders. The fix is unglamorous: render each order row as a plain sentence ("Order #4021, one blender, shipped June 2, delivered June 5") before embedding. Then re-run the suite, which takes minutes because retrieval evals are arithmetic, not generation (Lesson 5.2):

after fix #212 (order rows re-chunked as sentences):
    order status / lookup    recall@5   0.61 -> 0.93
                             ans.rel.   0.71 -> 0.90
                             faith.     0.90 -> 0.92
    all other segments       within ±0.02 of baseline

Read what moved and what didn't. Recall jumped, because that's what the fix targeted. Answer relevance followed it up almost point for point: the model was never bad at answering order-status questions; it was answering them without the orders. Faithfulness barely moved, because it was never the problem: the model was faithful to whatever it got, which is exactly why the triangle told you not to touch the prompt. And the flat line across every other segment is the quiet half of the result: the re-run doubles as a regression check, proof the re-chunking didn't buy order status at returns-policy's expense. Next turn of the wheel: gift cards, whose 0.34 recall and 0.58 faithfulness are the missing-content signature; the fix there is a document, not a retriever. And every turn banks its inputs: the failing queries that drove this round join the labeled suite, so next month's segments are sharper than this month's.

Where teams go wrong in this loop is at the pick-the-fix step. They chase the worst percentage instead of the biggest product: a quarter spent polishing a 3% segment while a 31% segment bleeds. They ship the chunker change after checking only the aggregate, which can rise while a small segment quietly collapses inside it. And they spend weeks tuning embeddings on a cluster whose real problem is that nobody wrote the document; the first question in the localization checklist exists precisely because no retriever, however tuned, can find content that doesn't exist.

Why this loop compounds

These are leading metrics in Lesson 1.4's sense, and that's the point of the whole module. Per-segment recall and faithfulness rate move minutes after you change the chunker, and they name the broken component; a thumbs-down arrives days later and names nothing. Count your pace among them too: retrieval experiments run per week is the number that predicts whether the system improves.

The loop, end to end: production queries flow in; cluster them; run per-segment evals; pick the fix where volume meets failure rate; re-run the suite; ship; repeat. Every turn adds real cases to the suite and sharpens the segments. Most agents read before they act; after this module, the reading is measured, and "the agent gave a bad answer" is no longer a mystery. It's a leg, a segment, and a fix.