Guardrails and how to eval them

A guardrail is a runtime validator: it inspects input before the agent acts on it, or output before the user sees it, and blocks or flags what fails. Wiring guardrails into your stack is a production concern (Lesson 6.2); this lesson is about the part teams skip. A guardrail is itself a classifier, and an unevaluated classifier standing guard is just a second model taken on faith.

Score it like a classifier

So eval it like a classifier, on two corpora. The adversarial suite (Lesson 7.2) gives you the true positive rate: what fraction of attacks does it catch? A benign corpus (real, legitimate traffic sampled from production) gives the false positive rate: what fraction of honest requests does it block? Report both or neither. A catch rate alone is meaningless, because a guardrail that blocks everything catches everything.

The benign corpus is the one teams have to be told to build, so build it deliberately: sample real, legitimate requests from production logs: the fuller the variety of honest phrasings, the more trustworthy the false-positive number. Don't hand-write it, and don't reuse the benign twins from Lesson 7.4's over-refusal suite as your only source; those are chosen to look like attacks, so they over-state how often the guardrail trips on ordinary traffic. You want the guardrail measured against what customers actually send, not against a set curated to be tricky.

The false positive number is where the money is. Say your guardrail catches 95% of injection attempts and blocks 2% of legitimate refund requests. If one message in ten thousand is an attack, it turns away two hundred real customers for every attacker it stops. That can still be the right trade (one successful fraud may cost more than two hundred apologies), but it's an arithmetic decision, and you can't do the arithmetic without both rates and your real base rate.

The confusion matrix, with numbers

Make that arithmetic concrete. Run the guardrail against an attack corpus of 500 cases and a benign corpus of 5,000 sampled real requests, and you get four counts (the confusion matrix) plus the two rates derived from them. Then project those rates onto real traffic, where the base rate does the damage:

guardrail v1, measured on two corpora
                         flagged        passed
      attack corpus       475 (TP)       25 (FN)     n=500     TPR = 95%
      benign corpus       100 (FP)    4,900 (TN)     n=5,000   FPR = 2%
    
    project onto real traffic  (base rate: 1 attack per 10,000 messages,
    1,000,000 messages/day):
      attacks       100  ->     95 blocked,        5 slip through
      legit     999,900  ->  19,998 wrongly blocked, 979,902 pass
      cost ratio: ~200 real customers turned away per attacker stopped

The mechanism to internalize is that the base rate, not the rates on the corpora, decides who feels the guardrail. On the balanced test corpora the guardrail looks excellent: 95% caught, only 2% false alarms. In production, legit traffic outnumbers attacks ten-thousand-to-one, so the tiny 2% false-positive rate applies to a huge denominator and the excellent 95% catch rate applies to a tiny one. Twenty thousand blocked customers against ninety-five stopped attacks is the same guardrail, read at the real base rate. This is why a catch rate quoted without a false-positive rate isn't a partial result. It's a misleading one.

Layers and thresholds

In practice guardrails come in layers, and each layer needs its own threshold, tuned on the same two corpora:

Tuning the classifier's threshold is where the trade-off becomes tangible. The classifier emits a score from 0 to 1; the threshold is the line above which a message is flagged. At 0.5 you might measure the 95%/2% above. Tighten it to 0.3 (flag more aggressively) and the true positive rate creeps up toward, say, 98%, but the false positive rate jumps to perhaps 6%, which at the base rate above means roughly 60,000 blocked customers a day instead of 20,000. Loosen it to 0.7 and the false positive rate falls to 0.5% (about 5,000 blocked) while the catch rate slips to 85% and fifteen attacks now leak. What moves, and why it matters: because legitimate traffic is 10,000× the attack traffic, each point of false-positive rate you add blocks about 10,000× more people than the point of catch rate it buys you saves: the base-rate ratio itself. So the right move at the classifier layer is a conservative threshold that blocks only confident detections, with borderline scores escalated to the judge rather than blocked outright: spend a slow, expensive model call precisely on the cases where the cheap layer is unsure.

The pattern filter deserves its own caution, because its false positives are the cheapest to create and the most expensive to suffer. A keyword rule that blocks the word "ignore" to catch "ignore prior instructions" will also block "please ignore my earlier message, I found the order number": a perfectly honest customer, turned away by a regex. Since the pattern layer sees every message before anything smarter does, one over-broad rule taxes all of your traffic at once. Keep the list to near-certain signatures, measure each rule against the benign corpus before you add it, and let the classifier and judge handle everything a blunt string match would get wrong.

Cheap and strict up front, expensive and careful behind. And treat the whole stack as code: any change to a pattern list or a threshold re-runs both corpora in CI, because a guardrail regression is an agent regression.

Where teams go wrong: they tune the threshold to maximize catch rate on the attack corpus alone, never running the benign corpus, and then discover in production that support volume spiked because the guardrail is eating legitimate refund requests. The benign corpus isn't optional ballast. It's half the measurement, and the half that determines whether the guardrail is usable.