How to turn an AI agent failure into a regression test

An agent updates the right customer, reports success, and leaves two audit entries behind. Your reviewer reads the answer and approves the run. The duplicate appears later.

Use that failure to define a test of the workflow’s actions and stored outcome. This guide works through a synthetic CRM example and includes a case template plus a small Node.js checker. It contains no customer results or benchmark claims.

1. Write down the failure you need to catch

In our example, a user approves changing customer customer_demo_17 to qualified. The CRM commits the update, but the client times out before receiving the response. The agent repeats the write and creates a second audit entry.

Keep the request, starting state, tool responses and observed side effects. Record the prompt version, model, tool schema and runner revision so you can compare changes. Replace customer identifiers and confidential content with synthetic fixtures before sharing a case.

A useful failure statement is: “After a timeout following a committed CRM update, recovery must leave one committed mutation and one audit entry for the approved operation.” It gives your reviewer something to check beyond the final answer.

2. Define the contract before choosing a grader

For this case, we want the agent to update the intended customer with the approved values, keep the operation’s idempotency key stable across write attempts, and finish with one committed write and one audit entry. We cap write attempts at two for this fixture. Your workflow may need a different recovery policy.

A timeout leaves the outcome uncertain. If the API supports durable idempotency, a retry can reuse the same operation identifier. Otherwise, your workflow may need a status check or a human handoff. Check the service’s actual semantics, key scope and retention window before relying on deduplication. The Amazon Builders’ Library explains this retry contract.

Keep the outcome separate from the tool-call count: two attempts can produce one committed operation. A single attempt can also return a timeout after committing. Read the fake service’s ledger or a sandbox audit log to determine the effect; a successful-looking answer cannot establish it.

3. Reproduce the uncertain outcome in a fixture

Use a fake CRM or a sandbox with synthetic records. Configure the first write to commit and then raise a timeout. Keep its committed state when the agent recovers. If you reset the fixture before the retry, you erase the failure you wanted to test.

Capture a normalized result with:

• tool_calls: write attempts, arguments, operation key and whether approval existed before each attempt;
• committed_writes: the fake service’s mutation ledger;
• audit_entries: the fake service’s audit ledger;
• final_record: the state after recovery, and status: the workflow outcome.

Your adapter must collect the ledgers from the service fixture. Asking the agent to describe what it changed gives you a different signal. Preserve any unrelated records too, and extend the checks if your workflow can change them.

4. Download the case and run the checker

Download these three files into one directory:

• Regression case template
• Synthetic passing result
• Node.js result checker

Run this command with Node.js 20 or later:

node check-agent-regression.mjs agent-regression-case.json agent-regression-result-example.json

The supplied example prints PASS. It demonstrates the file format; it does not run or evaluate your agent. The checker runs locally, requires no packages, makes no network requests and uploads no data.

Replace the example with a result exported from your own fixture run:

node check-agent-regression.mjs agent-regression-case.json my-run-result.json

To see a failing check, duplicate an entry in the example’s committed_writes array and rerun. The checker prints FAIL: committed writes and exits with a nonzero status. It also checks the target, approved values, approval before writes, stable operation key, final state, audit entries and attempt budget.

This starter contract covers one approved CRM update. It does not check answer quality, concurrency, crash recovery or every possible tool action. Adapt the template and checker together when your workflow needs additional assertions.

5. Compare the baseline and candidate

Run your current workflow and the proposed change against the same fixture and fault injection. Record each run’s outcome and which assertions failed. Confirm the baseline reproduces the known failure before using this case as evidence of a fix.

For model-driven behavior, repeat the case and keep the individual results. A small run count can show a regression without establishing a reliable production failure rate. Separate deterministic assertions about stored state from human review of explanations or handoffs. LangSmith’s evaluation guide describes final-response, trajectory and individual-step evaluation.

Use a failing assertion as a release check when your team agrees the behavior is required. Keep an owner for the case and revisit it when the tool contract changes. For this fixture, reset the synthetic CRM between independent runs while preserving state throughout each run’s recovery sequence.

6. Add nearby cases after the first one works

Start with the incident you can reproduce. Then add cases for a timeout before commit, an ambiguous customer match, missing approval and a rejected write. Define the expected recovery separately for each case. A missing-approval case should not inherit this template’s expected successful update.

Watch the cost of retries as well as correctness. Our AgentMeter cost guide explains how to measure spend per workflow. Keep cost observations separate from the service ledger you use to grade actions.

Build the first checks around your workflow

If your team has an agent workflow and keeps retesting the same failures by hand, apply for the Agent Evals pilot. We help define cases, agree on outcomes and wire a repeatable check into your existing stack. We agree scope and price before starting.