From a failed run to a regression case.

1. Make a prediction first

Start with “A recoverable timeout”. The direct policy makes one attempt. The guarded policy retries timeouts up to three attempts, verifies the returned metric and time window, and refuses write operations. Predict their outcomes before opening the trace.

Run the fixtures, inspect the case and compare the first timeout with the second successful reply. The guarded answer cites the successful step. The failed step remains in the trace and counts toward the call budget.

2. Work through the failure families

• Supported answer: both policies can answer from matching evidence.
• Transient and persistent timeout: distinguish a useful retry from unbounded retrying.
• Empty result and missing value: a missing observation is not a measured zero.
• Wrong metric or window: plausible numbers can still answer the wrong question.
• Write proposal: a restart needs a separate approval path. The fixture never restarts anything.

3. Bring a sequential run record

Download the example bundle and format reference. Keep schemaVersion: 1, give every run a caseId and policy, and put its sequential tool steps in steps. Use mode: "recorded" for observations captured by your application. Export elapsed milliseconds relative to the start of the run, not Unix timestamps.

The request names the intended tool, operation, metric and window. Each successful tool step supplies its actual arguments and result. The output contains a disposition, optional numeric value, and step IDs used as evidence. The contract defines the expected action, maximum calls and optional expected value.

This is the Lab’s native format, not an OTLP importer. Adapt your existing trace exporter to these fields. Never relabel fixture timings as observations from a model.

4. Compare on the same ground truth

Keep the same case ID, request, contract and timing mode for both versions. Select the earlier and later policy in the workbench. A regression means a previously passing case now fails; improvement means the reverse. Missing or incompatible cases are listed as unmatched and excluded from these counts.

Inspect every regression before trusting the aggregate pass count. If the expected answer or request changed, treat it as a new test condition. The contract is supplied by you and needs independent review.

5. Run a check locally

Download and extract the local kit. With Node.js 18 or newer, run:

node cli.mjs --policy guarded --out guarded.json
node cli.mjs --input your-runs.json --out checked.json

Exit 0 means all supplied cases pass, 1 means a case failed, and 2 means invalid input or a runner error. --policy both includes the deliberately weak direct policy, so its exit 1 is expected. The kit makes no network, model or live tool calls.