CI gates and experiments

You now have several kinds of evals. The remaining question is when each one runs. The answer extends the pyramid from Lesson 3.4 into a schedule:

Gates that block vs. gates that warn

Gates only work with pre-committed thresholds (Lesson 4.3): the bar is written down before the run, and any critical-criterion failure vetoes the merge outright. Set judge pass-rate bars with the noise math of Lesson 4.2 in mind: a gate at 80% on a 50-case suite will block innocent PRs and pass guilty ones unless the margin exceeds the wobble of re-running the same version twice.

Written as CI configuration, the schedule and the thresholds look like this; note which lines block and which merely warn:

# ci/evals.yml (sketch): refund agent
  on_pull_request:                     # must finish in < 10 min
    step_checks:                       # tool args, schemas, trajectory structure
      cases: all
      block_if: any_failure            # deterministic: a fail is a fail
    golden_cases:                      # 120 cases, 1 trial each
      block_if: pass_rate < 95%        # baseline 98%; ±4 noise on 120 cases -> margin holds
      block_if: any_critical_case_fails  # the "never fumble" list vetoes outright
    judged_smoke:                      # 30-case judged subset
      warn_if: faithfulness < 90%      # judged deltas on 30 cases are noise; never block
  
  nightly:
    end_to_end: { cases: all, trials: 5, metric: "pass^3", alert: owner }
    full_judged:
      compare_to: last_release
      artifact: changed_cases.md       # reviewed by a human, see below
  
  on_release:
    require: [nightly_green, changed_cases_reviewed]
    online_ab: only_for_flagged_major_changes

The mechanism behind the block/warn split: a gate is a promise that a red result means a real problem, and every false block spends that credibility. Deterministic checks can afford to block because they don't wobble. Judged scores on a 30-case smoke suite wobble by more than any delta you'd care about, so they warn: a human glances, and the full-size judged comparison waits for nightly, where 120 cases and five trials buy enough statistical power to mean something.

Version everything that touches the score

A score is a function of four things: agent code, dataset, judge prompts, and rubric. Version them together (same repo, or pinned references) so every recorded score carries the versions that produced it. Otherwise you get phantom regressions: someone sharpens a judge definition and every dashboard drops five points with no agent change. When a PR changes both the agent and the suite, run old-suite-on-new-agent once, so you know which change moved the number.

On disk, versioning-together is just a directory layout plus one lock file:

evals/
    cases/
      golden/          refund_core.yaml     # each case: id, expectation, source trace_id
                       gift_cards.yaml      # the cluster from Lesson 6.2, now a suite file
      adversarial/     injection.yaml       # Module 07's cases live here too
    rubrics/
      support_reply.md                      # human-readable criteria; the owner signs off
    judges/
      faithfulness/    prompt.md
                       calibration.jsonl    # 60 human-labeled traces from the review queue
      tone/            prompt.md
                       calibration.jsonl
    runs/
      2026-07-08-nightly.json               # score + pins: {agent: "2.4.1", suite: "v37",
                                            #   judge: "faithfulness-v3", rubric: "9e1c…"}
    pins.lock                               # suite v37 := cases@<sha> judges@<sha> rubrics@<sha>

Two details in that tree do real work. Every golden case carries the trace_id it was cut from, so "why does this case exist" is answerable years later. And each judge's calibration set lives next to its prompt, which means a PR that edits prompt.md shows up in the same diff as the calibration data it must be re-validated against; the reviewer can't approve one without noticing the other.

Read the flips, not the delta

For bigger bets (a new model, a rewritten system prompt), the sequence is offline first, online second. Run both candidates on the full suite, several trials each; offline is where iteration is cheap and inputs are controlled. Only a candidate that wins offline earns an online A/B (Lesson 4.4): a fraction of traffic, both arms scored by the same online judges from Lesson 6.2, with lagging user metrics as the tiebreak.

In any comparison, the most useful artifact isn't the delta; it's the changed-cases report: exactly which cases flipped, in each direction. A move from 78% to 82% could be four fixes, or nine fixes and five fresh breaks. Lesson 4.2 warned that small deltas are often noise, and the flip list is how you tell: real improvements fix the specific cases you targeted, while noise flips a random scatter that flips back on re-run. Read the newly failing cases before you celebrate the aggregate.

Here's one, for the prompt rewrite that shipped the gift-card fix:

Comparison: refund-agent 2.4.1 -> 2.5.0    suite v37, judge faithfulness-v3
  Aggregate: 78% -> 82%   (117 cases, paired, 5 trials)
  
  FIXED (9):
    #012  gift-card refund, no policy doc     fail -> pass   targeted
    #045  gift card, partial balance          fail -> pass   targeted
    ...   5 more from the gift-card cluster   fail -> pass   targeted
    #083  typo'd order number                 fail -> pass   NOT targeted; verify
  
  BROKE (5):
    #051  refund > $200, must escalate        pass -> fail   CRITICAL
    #007  refund + address change in one msg  pass -> fail   new: ignores the address
    #066  apology without next step           pass -> fail
    #078, #101                                pass -> fail   flaky in nightly history

The reading, line by line: seven of nine fixes are exactly the gift-card cases the rewrite targeted; that's the signature of a real improvement, not noise. The untargeted fix (#083) gets verified rather than banked; free wins are sometimes real and sometimes a grader hiccup. On the broken side, #078 and #101 have flip-flopped across recent nightlies, so re-run before investigating. #007 is a genuine new break: the rewrite narrowed the agent's attention. And #051 fails a critical criterion: the agent issued a large refund without escalating. Under the pre-committed rules, that single line vetoes the release, four-point aggregate gain and all, which is the entire argument for reading flips instead of deltas, compressed into one row.

Where teams go wrong: they gate merges on judged pass rates from small suites, exactly what the config above refuses to do. The gate blocks an innocent PR on Tuesday, another on Thursday; by the next sprint engineers have a skip-evals label and the gate is dead: worse than no gate, because the dashboard still glows green. The durable arrangement is asymmetric on purpose: deterministic checks block, judged suites warn and page the owner, and the threshold only ratchets up when the case count grows enough to support it. A gate survives by almost never being wrong.

That closes the loop this module built. Traces (Lesson 6.1) feed online judges (6.2) and review queues (6.3); reviews feed golden cases and calibration; and the suite gates the next change. Production isn't where evals stop; it's where they get their data.