Memory and long sessions

A single-turn suite asks "was this reply right?" A long session asks a harder question first: did the whole conversation meet the user's goal, yes or no? Grade at the session level before the turn level. A conversation can contain six individually fine replies and still lose the plot, and only the session verdict sees it. Once a session fails, drop to turn level to find where: Lesson 3.3's first divergence, applied to turns instead of steps.

The mechanism behind session-first is averaging: per-turn pass rates blend six good turns and one fatal one into 86%, a number that looks like a healthy agent. The session question is a single binary a judge can hold (given the user's goal and the full transcript, was the goal met?), which makes it exactly the narrow, calibratable shape Lesson 2.1 asks for. Grade sessions with that one judge, then spend turn-level attention only on the sessions that failed.

Replay the prefix, grade the next move

The cheap instrument for turn-level grading is N−1 replay: take a real conversation, feed the agent the first N−1 turns verbatim, and eval only the next decision. No branching dialogue, no second model playing the customer: one frozen context, one graded step. Compared with a full simulated-user run (Lesson 3.4), replay is cheaper, perfectly reproducible, and aimed at the exact decision that failed in production. The simulated user still earns its place for behavior that needs a live counterpart. Replay covers everything that doesn't.

Reproducible because nothing branches: a live multi-turn run samples a new path every time, but a frozen prefix pins every variable except the one decision under test. The cases come from the sessions that failed session-level grading. Cut each at its first divergence (Lesson 3.3) and the prefix plus an assertion is the case:

case: replay-1188   # cut from production session 58f2, first divergence at turn 5
    frozen context (turns 1 to 4, verbatim):
      user:  "I want to return the blender from order #4021."
      agent: "I can help. Was the item damaged, or is this a change of mind?"
      user:  "It arrived cracked. Also I'm traveling, contact me by email only."
      agent: "Understood: damaged item, and we'll use email."
    graded decision (turn 5), user says: "So what do I do now?"
    expect:
      - tool_called: check_eligibility(order_id=4021)
      - no_tool_called: schedule_callback      # user said email only
      - reply_mentions: "email"

Know when replay misleads, though. The frozen turns were produced by the old agent. Suppose your new candidate, given turn 1, would have asked "which order?" at turn 2, a better opening. It would never have reached this turn-5 state at all, so you're grading it on a conversation it wouldn't have had, and the assertion can fail a genuinely better agent (or pass a worse one that only shines on the old agent's trajectory). Replay answers one question (given this exact history, is the next move right?), which makes it ideal for regression-testing the same conversational policy, and wrong for comparing candidates whose early behavior differs. For those, go back to the simulated user, which lets each candidate steer its own conversation.

Corrections need a live counterpart

Behavior that needs a live counterpart starts with corrections. Real users fix the agent mid-conversation ("no, the other order"), and whether the second try uses the correction is invisible to any single-turn suite. Write cases where the simulated user delivers a specific correction after the first attempt, and grade the attempt that follows:

case: "Refund my order." (two orders exist: #4021, #4022)
      turn 1: agent proposes a refund for #4021
      correction: "No, the other one, the blender."
      expect (turn 2):
        - tool_called: create_refund(order_id=4022)
        - reply_mentions: "blender"

Models differ sharply here. The MINT benchmark measured exactly this (task-solving across turns with feedback) and found that models strong on a single try are not necessarily the strongest once feedback enters the loop. Test it when you choose a model, not just when you debug one.

Memory across sessions

Persistent memory adds a third granularity: across sessions. The test needs two runs. Seed the memory store, run a session where the user states a preference ("always refund to store credit"), then start a fresh session and assert the preference is recalled. Assert the other direction too: whatever the user asked the agent to forget must not surface. Under the hood, memory reads and writes are tool calls, so grade them like any trajectory step (Lesson 3.2): did the agent write the fact worth keeping, and did it retrieve the right one later?

As a case spec, the two-session shape looks like this. Note that session 2 starts with an empty context window, so the only bridge between the runs is the memory store itself:

case: memory-store-credit-preference
    session 1 (day 1):
      user: "From now on, refund me in store credit, never to my card."
      expect:
        - memory_write: {kind: preference, refund_method: store_credit}
    session 2 (day 4, fresh context; only the memory store persists):
      user: "The kettle from order #5310 arrived dented. Refund it."
      expect:
        - memory_read returns the store-credit preference
        - tool_called: create_refund(order_id=5310, method=store_credit)
        - reply_mentions: "store credit"

The fresh context is the entire point of the design. Run both turns in one session and the context window does the remembering: the eval passes with the memory system unplugged, which is to say it proves nothing. The forget direction gets the same two-session shape with the assertion flipped: session 1 says "drop the store-credit thing", session 2 must refund to the original payment method and must not cite the retracted preference. Benchmarks in this space (LongMemEval is a good public example) decompose memory into exactly these skills (extraction, cross-session reasoning, knowledge updates, abstention) and find that assistants drop sharply on sustained interaction, so assume nothing here is free.

Where teams go wrong with memory: they grade the store instead of the behavior. The dashboard shows the preference was written, so memory "works", but written is not retrieved, and retrieved is not used; the only eval that counts is a fresh session where the recalled fact changes a tool call. The same teams tend to skip the forget case entirely, and a memory that can't forget is a liability with a compliance department. Both evals are two model runs each. Run them.