A production AI agent rarely fails because the model suddenly becomes incapable of speaking English. It fails because a dependency times out after the agent has already formed a plan, a tool returns an empty page with a successful status code, a provider changes a field shape, or the context that looked current is already stale.
These failures are uncomfortable because the agent can still produce a fluent answer. It may even report that the task succeeded. A dashboard that tracks latency and HTTP error rate can therefore look healthy while the agent has duplicated an action, invented a recovery, or continued from a fact that expired ten minutes ago.
This is where chaos engineering becomes useful. The goal is not to randomly break an AI system for drama. The goal is to introduce a bounded, observable failure and verify that the system preserves the properties that matter: authority is not expanded, external side effects are not duplicated, stale observations are not treated as facts, and the workflow reaches a visible terminal state.
The thesis: Do not ask whether an AI agent can complete the happy path. Inject the failures that make its next decision ambiguous, then verify the resulting state with deterministic oracles.
The distinction matters. Offline evaluation asks whether an agent can solve a task under a selected input. Incident response asks what to do after a failure reaches users. Chaos testing asks a more operational question before that happens: when this dependency fails at this exact point, does the agent fail safely and recover honestly?
Why ordinary agent tests miss the dangerous failures
A conventional test often mocks every tool as fast, complete, and truthful. The model receives a clean schema, the retriever returns the right document, and the final assertion compares text with an expected answer. That is useful for basic correctness, but it does not exercise the boundary between reasoning and execution.
ReliabilityBench separates agent reliability into consistency, robustness, and fault tolerance. Its fault-tolerance dimension covers infrastructure failures such as timeouts, rate limits, partial responses, and schema changes; its evaluation uses end-state verification rather than text similarity. This is a better mental model for production: a response can be worded differently and still be correct, while a beautifully worded response can hide a broken state.
The number of steps makes the problem sharper. If each action has an independent five-percent failure chance, a twenty-action workflow is not “95% reliable.” Its probability of completing every step is approximately 0.95^20, or about 36%. Real systems have correlated failures and retries, so the arithmetic is not a service-level promise. It is a reminder that a small local failure rate becomes a large workflow problem when the agent has many opportunities to act.
MLflow’s production guidance similarly frames agents as distributed systems that need runtime governance, deterministic execution for critical operations, embedded evaluation, and shadow deployment for major changes. Chaos experiments turn those principles into evidence instead of assumptions.
Start with a failure model, not a random fault generator
A useful experiment begins with a hypothesis. “The agent should handle tool errors” is too vague to test. “If the inventory tool times out after the agent has prepared a reservation request, the system must not reserve anything and must ask for a fresh inventory check before retrying” is specific enough to produce an oracle.
Map the workflow into boundaries where the next action can change. These are usually model calls, tool calls, retrieval reads, approval gates, queues, state stores, and external write APIs. For each boundary, list the failure shape and the property that must survive it.
| Boundary | Fault to inject | Dangerous agent behavior | Required invariant |
|---|---|---|---|
| Model provider | Timeout, 429, truncated output, provider unavailable | Retry with a different instruction and duplicate a write | No write occurs without a valid, versioned action intent. |
| Read tool | Empty result, stale timestamp, partial page, wrong content type | Treat absence as proof or continue with expired facts | Every decision records observation age and freshness status. |
| Write tool | Timeout after server accepted request | Retry without an idempotency key | One business operation has at most one committed effect. |
| Schema boundary | Missing field, extra field, wrong enum, valid JSON with wrong meaning | Infer missing authority from context | Invalid or ambiguous output becomes a typed refusal. |
| Approval gate | Approval expires, reviewer rejects, duplicate click | Continue from an old approval | Approval is bound to exact action hash, actor, scope, and expiry. |
| Worker and queue | Restart, duplicate delivery, delayed message | Re-plan from a partial state | State transition is monotonic and replay is safe. |
| Context store | Stale memory, conflicting records, compaction loss | Cite old memory as current truth | Conflict is surfaced; no destructive action uses unresolved context. |
This matrix is more valuable than a long list of HTTP errors because it connects a fault to a decision. The same timeout is harmless for a read-only weather lookup and dangerous after a payment provider may have accepted a charge.
Define the safety envelope before injecting anything
Chaos testing must not become an excuse to damage a real customer account. The safest starting point is an isolated environment with synthetic identities, fake payment instruments, reversible tools, and a deterministic state store that can be reset between runs.
A practical safety envelope has four layers:
- Data boundary. Use generated or scrubbed data. Do not copy production secrets into a test cluster merely because the agent needs realistic context.
- Authority boundary. Replace destructive tools with simulators, or restrict them to a namespace that cannot reach real systems. Bind every capability to a tenant, workflow, and expiry.
- Side-effect boundary. Route writes through a reversible adapter that records the intended effect and supports compensation. A “mock” that silently calls the real API is not a mock.
- Stop boundary. Give every experiment a kill switch, wall-clock limit, maximum action count, and abort condition. The experiment controller must be able to stop the workflow without asking the model to cooperate.
The experiment itself should be versioned. Record the agent build, model identifier, prompt and tool versions, fault profile, seed or replay input, environment, and oracle version. Without this evidence, a pass is difficult to reproduce and a failure is difficult to explain.
Fault profiles should model semantics, not only transport errors
A transport-only test suite tends to overvalue status codes. Agents experience the meaning of a result, not just its HTTP envelope. A 200 OK response containing an empty page, yesterday’s inventory, or a schema-valid but semantically impossible amount can be more dangerous than a clean 500.
Use fault profiles that preserve enough realism to exercise the decision boundary:
experiment: reservation-timeout-after-intent
workflow: reserve_inventory
seed_case: synthetic-order-042
faults:
- boundary: inventory.read
mode: stale
age_seconds: 900
- boundary: reservation.write
mode: accept_then_timeout
server_commit: true
client_response: timeout
safety:
environment: isolated
tenant: chaos-lab
max_actions: 8
max_wall_clock_ms: 30000
external_writes: simulated-only
oracle:
- committed_reservation_count <= 1
- retry_requires_fresh_inventory
- final_state in [awaiting_confirmation, completed, safely_failed]Useful fault families include omission, delay, value corruption, and duplication. Omission removes a field or result. Delay makes the agent reason under a deadline. Value corruption changes a currency, timestamp, status, or identity while preserving the outer schema. Duplication delivers the same event twice. A fifth family, semantic contradiction, returns two individually valid observations that cannot both be true.
Do not inject every fault into every run. Start with one fault at one boundary, then compose only the combinations that correspond to a credible incident. A large random matrix generates noise and makes failures hard to triage.
The recovery contract belongs outside the model
The model may propose a recovery, but the runtime must decide whether recovery is allowed. A recovery contract should answer five questions:
| Question | Example contract |
|---|---|
| Can the operation be retried? | Only if the previous attempt has an unknown outcome and the request carries the same idempotency key. |
| What must be re-read? | Inventory and price must be fresh within 60 seconds. |
| What authority survives? | Read access survives; write authority expires after 5 minutes. |
| What evidence is required? | Tool result, request hash, state version, and policy decision. |
| When must the workflow stop? | After two failed recovery attempts or any invariant violation. |
This is not prompt wording. It is executable policy around the agent. The system should reject a tool call that violates the contract even if the model explains why the call seems reasonable.
For a write-oriented tool, the adapter should separate “request sent,” “server accepted,” and “client observed response.” A timeout does not tell the agent whether the operation failed. The correct state is often unknown, which requires reconciliation rather than an immediate blind retry.
planned -> dispatched -> outcome_unknown -> reconcile
\-> committed
\-> not_committed
\-> unresolved_manual_reviewThis is where chaos testing connects to idempotency without replacing the existing idempotency playbook. The experiment asks whether the system actually uses its contract under uncertainty, not whether a key exists in a design document.
Use state-based oracles, not an LLM judge alone
A fluent recovery message is not proof of recovery. Each experiment needs a deterministic oracle that can inspect the system state before and after the run. The oracle may compare database snapshots, event counts, object versions, authorization decisions, queue offsets, or a signed action ledger.
ReliabilityBench’s action metamorphic relations provide a useful pattern: after a fault or an equivalent perturbation, correctness can be determined by end-state equivalence rather than identical wording. For example, an agent may say “I could not complete the reservation” or “The reservation remains pending while inventory is refreshed.” Both can be acceptable if the state is pending, no duplicate reservation exists, and the user receives an honest next step.
A minimal oracle record might look like this:
{
"experiment_id": "exp_01J...",
"initial_state_hash": "sha256:...",
"final_state_hash": "sha256:...",
"observed_actions": 4,
"committed_effects": 0,
"freshness_violations": 0,
"authority_expansions": 0,
"terminal_state": "awaiting_confirmation",
"verdict": "pass"
}Keep the oracle independent from the same model path being tested. If the model decides whether its own answer is correct, the experiment can report confidence while the database reports damage.
Measure resilience as a surface, not a single pass rate
A single success percentage hides important differences. Track at least three dimensions:
| Dimension | What to measure | Example question |
|---|---|---|
| Consistency | Same scenario, repeated seeds, same invariant outcome | Does the same fault sometimes trigger a write and sometimes a refusal? |
| Robustness | Equivalent inputs, reordered constraints, irrelevant context | Does a paraphrase change the safety decision? |
| Fault tolerance | Timeouts, rate limits, partial results, restarts, stale data | Does the workflow converge to a safe terminal state? |
For operational use, add recovery latency, extra model/tool calls, duplicate-effect rate, stale-context acceptance rate, human-escalation rate, and evidence completeness. A recovery that preserves state but consumes ten times the normal budget may still be unacceptable.
Do not optimize for “the agent always refuses.” A system that refuses every request has perfect safety under one narrow metric and no utility. The target is a calibrated response: continue when the operation is safe and evidence is sufficient, pause when the outcome is unknown, and refuse when authority or correctness cannot be established.
Promotion gates turn experiments into an engineering practice
Run experiments first as a pull-request check for policy and adapter changes, then as a scheduled suite against a production-like environment. Keep a small canary set that completes quickly and a larger suite that covers compounded faults overnight.
A promotion gate can be expressed plainly:
promote only if:
no invariant violation
duplicate_effect_rate == 0
authority_expansion_rate == 0
stale_context_acceptance_rate == 0
evidence_completeness >= 99%
recovery_p95 <= workflow_budget
unresolved_unknown_outcomes <= approved_thresholdThresholds should be risk-specific. A support-ticket agent may tolerate a human handoff; a payment or deletion workflow may require zero ambiguous writes. Keep exceptions explicit, time-bound, and owned by a person or team. A permanent exception is usually an untested assumption with a nicer name.
When a test fails, save the complete evidence pack: trace identifiers, fault timeline, state snapshots, tool requests and responses after redaction, policy decisions, model and tool versions, and the smallest replayable input. The purpose is not to shame the model. It is to make the boundary that failed visible enough to repair.
What not to do
Do not begin by injecting live failures into customer traffic. Do not treat a model-generated apology as a rollback. Do not use only 500, timeout, and rate-limit responses while ignoring stale and semantically wrong data. Do not compare only final text. Do not let the agent expand its own authority to recover. Do not call a test environment safe if it shares production credentials, queues, buckets, or webhook endpoints.
Most importantly, do not confuse chaos engineering with a one-time reliability campaign. New tools, providers, schemas, prompts, memory policies, and orchestration code create new failure surfaces. The fault matrix should evolve with the system and remain part of the release evidence.
A practical starting sequence
Start with one workflow that can create an external effect and one read-only workflow that depends on freshness. Capture a known-good trace. Add a simulator around the most important tool boundary. Inject one timeout after intent formation, one stale result, and one malformed response. Write the state oracle before running the experiment. Then add a worker restart and a duplicate event.
The first goal is not a perfect benchmark. It is to discover whether the system can distinguish failed, not started, and outcome unknown. Those states lead to different recovery actions. Once that distinction is reliable, add provider outages, rate-limit bursts, semantic contradictions, and composed faults.
A production-ready agent is not one that never encounters an error. It is one whose authority, state, and user promise remain bounded when the world stops behaving like a demo.
Comments (0)