Scoring tool calls

The strictness ladder

Tool calls are the most gradeable part of a trajectory because they're structured data; no judge needed. Score them at increasing strictness:

The ladder is ordered by what each rung can catch. Selection is the cheapest and catches the grossest failures: the agent reaching for a tool that doesn't exist, or reaching for any tool when the right move is a clarifying question. Arguments catch grounding failures: the right tool pointed at the wrong order is worse than no call at all. Order catches policy violations no single call reveals: every call individually fine, the sequence forbidden. Effects is the strictest and most honest rung, because it grades what happened rather than what was attempted: a retry that fired create_refund twice looks like persistence in the call log and looks like a double refund in the database.

Two rules keep tool-call evals honest. First, compare structures, not strings: parse the call and check fields with type-appropriate matchers (exact for IDs and enums; normalized or judge-scored for free text). This is how the serious function-calling benchmarks grade (parsing each call into a syntax tree and matching name, parameters, and typed values against a set of accepted answers), and it eliminates whitespace-and-phrasing false failures.

Second, allow equivalent paths. There is usually more than one correct trajectory. Express expectations as constraints ("must call create_refund(order_id=4021) exactly once; must check eligibility first; must not call escalate_to_human") rather than one blessed sequence. Where you can assert on final environment state, prefer that: it's naturally path-agnostic, and it catches the one thing that matters most, unintended side effects.

A constraint spec you can run

Those two rules combine into a per-case artifact: a constraint spec. Here is a complete one for the running refund case, in a shape you can adapt to any harness today:

case: refund-4021-damaged
    input: "Refund order #4021, it arrived damaged."
  
    must_call:
      - tool: create_refund
        args: {order_id: 4021, amount: 89.99}  # typed match on listed fields;
        count: 1                               #   unlisted fields are ignored
    must_call_before:
      - first: check_eligibility               # order asserted ONLY where it
        then:  create_refund                   #   genuinely matters
    must_not_call:
      - tool: escalate_to_human
      - tool: create_refund
        args: {order_id: {not: 4021}}          # someone else's order = critical
  
    final_state:
      refunds: exactly [{order_id: 4021, amount: 89.99, status: "created"}]
      orders:  unchanged

The checker behind this is about forty lines of code, not a framework. It parses the trace into an ordered list of (tool, parsed_args) events. must_call scans the list for events matching the tool name and every listed field (typed comparison, so the integer 4021 doesn't silently equal the string "4021") and asserts the match count. must_call_before takes the index of the first matching event on each side and compares them. must_not_call fails on any matching event, anywhere. final_state ignores the trace entirely: it queries the sandbox after the run and diffs the result against the expected snapshot. Every constraint returns its own pass/fail, so a failing case names the promise that was broken: "called escalate_to_human" reads very differently from "refunded the wrong order".

Equivalent paths, concretely

Now watch the spec earn the equivalent-paths rule. Here are two runs that both satisfy every constraint above:

# Trajectory A                      # Trajectory B
    1 get_order(4021)                   1 retrieval: damaged-item policy
    2 retrieval: damaged-item policy    2 get_order(4021)
    3 check_eligibility(4021)           3 check_eligibility(4021)
    4 create_refund(4021, 89.99)        4 create_refund(4021, 89.99)
  
    # Different lookup order; same constraints satisfied:
    # eligibility precedes refund, exactly one refund, no escalation.

A grader that diffed against one recorded golden sequence would pass A and fail B, even though B is arguably the better plan, reading the policy before touching the account. The constraint spec is indifferent to the difference, and that indifference is the feature: model upgrades routinely reorder information-gathering steps, and your suite should not turn red on every upgrade as a matter of principle. The order constraint survives only where the business rule lives: eligibility before refund. Two remaining questions answer themselves once the spec exists. Free-text arguments like a refund note either go unmatched or get a narrow judge of their own (Lesson 2.2). And count: 1 is doing quiet, important work: an agent that retries a timed-out refund call is one flaky network day away from refunding twice, and only a count assertion or a final-state check will ever notice.

Where teams go wrong

Golden-trajectory diffing: recording one passing run and diffing every future run against it. The suite fails on every model upgrade, everyone learns to ignore the red, and a real regression scrolls past in the noise. String-matched arguments: asserting create_refund(order_id=4021, amount=89.99) as a literal substring, so reordered JSON keys or a trailing zero fail the case; parse first, then compare fields. Calls without effects: asserting that the call happened but never checking the sandbox, which misses the double-fire, the write that failed but was reported as success, and every side effect you didn't think to forbid. If you can only afford one rung of the ladder, buy the effects rung.