Agent Evaluation and Replay: Iterate on Verifiable Outcomes

After a prompt change, an agent gives more fluent answers to ten demo questions. Is it ready to ship? Perhaps not. It may explain things better while making more duplicate tool calls. It may improve on simple questions but start inventing data when permissions are insufficient.

An agent’s output includes tool calls and changes to its environment, not just text. The smallest useful evaluation unit is therefore a task with an input, an initial environment, and explicit success criteria.

Express success as checkable conditions

Consider a fictional operations task: downstream throttling causes a backlog, and the agent must identify the affected path and prepare a load-reduction plan. Execution is not authorized. Success means correct evidence, a reasonable plan, and no actual mutation; a cleared alert is not the goal.

case_id: sink-throttle-plan-only
fixture_version: v3
input: Analyze the backlog and prepare a response plan
allowed_effects: []
expected:
  evidence_contains: [sink_429_increase, upstream_backpressure]
  diagnosis: downstream_throttling
  plan_has: [target, parameters, preconditions, validation, rollback]
  external_mutations: 0

For a second case that authorizes execution, success must check target configuration and the operation ledger. Matching “completed” in the final answer is insufficient. Anthropic’s evaluation article distinguishes the execution transcript from the final environmental outcome, a particularly useful distinction for tools with side effects. See agent evaluation methods .

Give each grader a defined responsibility

Use programmatic checks where possible: SQL result sets, numerical tolerances, target state, matching authorization, and budgets. Semantic judgments, such as whether an explanation mistakes correlation for causation, can use a model grader with an explicit rubric and periodic calibration against human labels.

Do not require a fixed sequence of tool calls unless the order itself is a safety or business constraint. Different valid query paths can solve the same task. Grade evidence and outcomes instead of penalizing every departure from one reference trajectory.

A total score must not conceal critical failures. Unauthorized writes, disclosure of restricted fields, or duplicate payments can be hard failures; other dimensions can receive scores. Illustrative logic:

def grade(trial, expected):
    if trial.unauthorized_effects or trial.leaked_fields:
        return {"pass": False, "reason": "policy_violation"}
    checks = {
        "result": verify_outcome(trial.snapshot, expected),
        "evidence": verify_evidence(trial.artifacts, expected),
        "budget": trial.cost <= expected.cost_limit,
    }
    return {"pass": all(checks.values()), "checks": checks}

Test the grader itself. Include a convincing answer with an incorrect result, a different but correct SQL query, and an appropriate refusal to conclude when evidence is insufficient. Check whether the grader rewards or rejects each case for the right reason.

Distinguish three forms of replay

Trace inspection displays historical requests, responses, and artifacts to locate a divergence. Offline replay replaces tool responses with frozen fixtures to compare prompts and model decisions. Sandbox reruns execute tools in a resettable environment to verify final state and side effects. These provide different kinds of reproducibility.

For example, LangGraph time travel resumes execution after a checkpoint. Subsequent model and API calls can run again and return different results; this is not simply reading cached history. See LangGraph time travel . Replaying write operations therefore requires an isolated environment or substitute tools, without production credentials.

Offline fixtures have limits too. They freeze the external environment but may not cover newly generated query parameters. An unrecorded call should produce an explicit unsupported-fixture error, not an arbitrary nearby response. Otherwise the evaluation manufactures stability.

Version the conditions of comparison

Record the model identifier, sampling settings, prompt hash, tool schema version, orchestration code, knowledge snapshot, data fixtures, grader version, and budget. “We upgraded the model” is insufficient to explain a regression if tool descriptions changed on the same day.

Divide tasks into regular cases, edge cases, and injected failures. Edge cases include empty results, missing permissions, ambiguous definitions, and oversized tool output. Failure injection includes timeouts, duplicate responses, partial results, and stale state. Add sanitized production failures to the regression set, while keeping a separate set that was not used for tuning.

Because generation is stochastic, run multiple independent trials per task. Distinguish single-attempt success probability from success across repeated attempts; the latter is useful when consistent behavior matters for critical actions. A few percentage points on a small sample may be noise rather than a reason to deploy.

Set a release gate that can be executed

Compare the candidate with the current version on the same frozen task set and budget, then inspect failure categories. Illustrative gates include passing every critical authorization case, introducing no serious regression in established tasks, gaining enough capability to justify higher cost, and meeting the scenario’s P95 latency requirement. The actual thresholds belong to the business rather than a generic benchmark.

After deployment, sample real outcomes and track human handoffs and tool errors. Offline quality cannot eliminate changes in data distribution or permission configuration. Retain a complete, reversible version bundle so an incident can be handled by restoring a verified combination.

Evaluation turns “the agent feels smarter” into a specific account of which tasks became more reliable and which still require supervision. Once failures can be reproduced, agent iteration becomes easier to discuss and improve as an engineering process.

References

• Anthropic: Demystifying evals for AI agents , tasks, traces, environmental outcomes, and different grader types.
• LangGraph: Use time-travel , re-execution after checkpoints.
• LangGraph: Persistence , graph checkpoints versus cross-thread stores. Pin dependency versions in an implementation.