Voice and other modalities

A voice agent looks like a new evaluation problem, and the temptation is to grade it holistically: play a call, ask a judge how it went. Resist that. Voice is text plus two extra layers, and the rule that makes it tractable is score each layer separately, in order, because no downstream eval can fix an upstream miss. If transcription heard the wrong order number, the wrong order number gets refunded, and no amount of dialogue quality downstream changes that.

WER on your audio, one call at a time

Word error rate is worth computing by hand once, because the mechanism explains its blind spot. Take a reference transcript (a human listening to the recording) and the hypothesis (what your speech-to-text produced), align them, and count substitutions, deletions, and insertions over the reference length:

reference (human):  "hi yes um i'd like a refund for order
                         four oh two one the blender arrived cracked"
    hypothesis (ASR):   "hi yes i'd like a refund for order
                         four oh seven one the blender arrived cracked"
    
    errors: 1 deletion ("um"), 1 substitution ("two" -> "seven")
    WER = (S + D + I) / N = (1 + 1 + 0) / 17 = 11.8%
    
    entity span "four oh two one": 1 substitution / 4 tokens = 25%
    plain WER calls this transcript fine; entity WER says the agent
    is about to look up order #4071 for a #4021 customer.

Both errors count once, and that's the blind spot: WER weights "um" and "two" identically, but the deleted filler costs nothing while the substituted digit sends a refund toward the wrong order, possibly the wrong customer. The fix is entity-weighted WER: tag the spans that drive tool calls (order numbers, names, amounts, dates) in your reference transcripts and compute error rate over those spans separately, or weight their tokens heavily in the aggregate. In the example, overall WER is a comfortable 11.8% while entity WER is 25%, and entity WER is the number that predicts downstream damage. Build the reference set from your own recorded calls (redacted at capture per Lesson 6.1), stratified the way Lesson 5.2 stratifies queries: accents, noise levels, connection quality.

Specifying the audio-only behaviors

Those voice-only qualities deserve definitions. Time-to-first-word is a correctness criterion, not a nicety: a user who hears two seconds of silence starts talking again, and now the turns are tangled. Barge-in handling is whether the agent stops when interrupted instead of talking over the customer. And entity round-tripping checks that numbers, dates, and names survive speech-to-text and text-to-speech intact: assert that the order number the user spoke is the order number the agent reads back.

Each of these is testable with a scripted audio injection at a fixed offset: a case spec like any other, just with timestamps in it:

case: barge-in-during-policy-readout
    setup:  agent begins a ~20-second reply reading the returns policy
    inject: at t=3.0s, caller audio: "no no, just tell me my refund status"
    expect:
      - agent audio stops within 500 ms of interruption onset
      - the interrupted sentence is not restarted afterward
      - the barge-in utterance appears in the transcript (not dropped)
      - next agent turn addresses refund status, not the policy

Note the third assertion: the failure mode isn't only talking over the caller, it's losing what the caller said while the agent was talking. The interruption has to be both honored and heard. Run these with recorded clips, not live humans, so the injection offset is identical every trial and time-to-first-word can be reported as a distribution (median and worst-case) rather than an anecdote. For end-to-end coverage under realistic speaker and noise conditions, VoiceBench is the public design worth studying: it stress-tests assistants across speaker, environment, and content variations rather than clean studio audio.

Screens are the same layering

The same layering transfers to vision and screen agents. Grade grounding first (did the model read the screen correctly, checked as element detection and OCR against labeled screenshots) and only then grade decisions given a correct reading. A wrong click is either a perception failure or a policy failure, and the layer split tells you which one you're fixing.

screenshot: refund-console.png, labeled elements
      button "Approve refund"   bbox (812, 440, 948, 472)
      button "Reject"           bbox (960, 440, 1044, 472)
      field  "Amount"           value "$49.00"
      row    "Order #4021"      status "eligible"
    
    grounding assertions (graded before any click is):
      - reads Amount as "$49.00"
      - locates "Approve refund" inside its labeled bbox (IoU >= 0.5)
      - reports no "Refund all" control (none exists on this screen)

The last assertion is the screen agent's hallucination check: models invent plausible UI elements the way they invent plausible citations, and an agent that "clicks" a control that isn't there will act on whatever actually occupies those pixels. Labeling twenty screenshots takes an afternoon and buys you the perception-versus-policy split for every failure after.

Where teams go wrong in this lesson's territory: they grade the call holistically after all (a judge listens to the recording and scores "call quality 7/10"), and when the score drops nobody knows which layer moved; or they accept the vendor's WER, measured on clean read-aloud corpora, as if it described their callers on speakerphone in a kitchen; or they burn a week tuning prompts to fix "the agent refunds the wrong orders" when the transcript said 4071 all along. Every one of these is the same mistake (skipping the layer order), and the first hour of debugging any voice failure should be reading transcripts against audio, not prompts.

That closes the module, and it closes the argument. Routers turned out to be classifiers, memory turned out to be tool calls, production writes turned out to be logged trajectories, and voice turned out to be layered metrics: every "hard" shape decomposed into machinery you already own from the first seven modules. New agent shapes don't need new eval theory. They need the same three pieces (an input, a run, a grader) aimed at the right layer.