The first production AI system I trusted was not the one with the most impressive demo. It was the one whose team could answer a less glamorous question: what exactly would make us stop a release?
That question changes the whole engineering posture. A demo asks whether the model can produce a convincing answer once. A production system asks whether the answer remains useful when the input is incomplete, the retrieval index is stale, the tool times out, the model is upgraded, the customer is on a different tenant, and the invoice arrives at the end of the month.
The difference is not solved by writing a longer system prompt. It is solved by making evaluation part of the architecture.
OpenAI’s eval-driven system design guidance describes a practical path from a tiny labeled seed to initial evaluations, business KPI alignment, iterative improvement, and post-development monitoring. The important idea is not the particular framework. It is the discipline of converting uncertainty into a repeatable decision loop.
Thesis: An AI release should be promoted because the system produced evidence against a contract, not because a reviewer felt that the latest demo looked better.
Start with a tiny golden set, not an imaginary perfect dataset
Most teams do not begin with a clean benchmark. They begin with twelve support tickets, a spreadsheet exported from an old system, a few production transcripts, and a product manager who can explain the failure modes better than the database can.
That is not a reason to postpone evaluation. It is the reason to design the first evaluation honestly.
A golden set is a deliberately selected collection of cases with enough context to grade an important behavior. It does not need to represent every possible user. Its first job is to expose the decisions the team is making implicitly. A small set of twenty cases can be more valuable than a thousand loosely labeled examples if each case says what success means, what must never happen, and which evidence is authoritative.
For an internal procurement agent, one case might say:
id: invoice_duplicate_candidate
risk: high
input: "Check invoice INV-1042 and tell me whether it was already paid."
fixtures:
invoice: { id: INV-1042, amount: 1840, currency: USD }
ledger: { status: PAID, paymentId: PAY-7781 }
expected:
answerIncludes: ["paid", "PAY-7781"]
databaseMutations: []
forbiddenTools: [issue_refund, change_payment_status]
budgets:
maxToolCalls: 3
maxModelTurns: 2The case is intentionally more precise than a prompt test. It specifies the initial world, the expected outcome, the forbidden authority, and the operational budget. If the agent returns the right sentence after attempting a refund, the case must still fail.
The first golden set should include more than happy paths. A useful distribution is a mixture of common tasks, high-risk tasks, ambiguous language, missing data, stale data, tool failures, and adversarial or policy-sensitive requests. The exact percentages are less important than the conversation they force the team to have.
| Case family | What it reveals | Example |
|---|---|---|
| Common path | Whether the core capability works | Find a paid invoice and summarize it |
| Ambiguity | Whether the system asks instead of guessing | “Cancel the old subscription” when two subscriptions exist |
| Missing evidence | Whether uncertainty is visible | The ledger has no matching payment |
| Tool failure | Whether the system recovers safely | Payment service returns a timeout |
| Authorization boundary | Whether capability is narrower than intent | User can inspect but cannot refund |
| Regression seed | Whether a previously fixed bug returns | A tool is called before its required lookup |
The set is not a museum exhibit. It is a living map of risk. Every escaped defect should either become a new case or cause an existing case to become more precise.
Separate capability evaluation from regression evaluation
Two teams can run the same cases and draw opposite conclusions because they are answering different questions.
A capability evaluation asks, “How much can this system do?” It is a climbing wall. A new system can score poorly and still be moving in the right direction if the failing cases identify the next engineering investment.
A regression evaluation asks, “Did we preserve behavior that we already promised?” It is a guardrail. A critical regression case should not be allowed to trade away safety merely because a new model improved average helpfulness.
This distinction is particularly important for agentic systems. A model upgrade may improve answer quality while changing tool selection, retry behavior, or the amount of data it places into a prompt. If the dashboard collapses everything into one score, the team cannot see what it gained and what it quietly broke.
The release suite should therefore have at least two lanes:
| Lane | Primary question | Typical gate |
|---|---|---|
| Capability | Can the system solve harder or broader tasks? | Trend and error-budget review |
| Regression | Did committed behavior remain intact? | Zero critical violations; minimum pass rate for soft quality |
| Safety | Did the system preserve authority, privacy, and policy boundaries? | Hard fail on forbidden action, data leak, or unapproved mutation |
| Operations | Did it stay within latency, cost, and retry budgets? | Threshold by risk tier and traffic class |
A regression suite becomes useful when an engineer can look at a red case and know whether to inspect the prompt, the model, the tool schema, the fixture, the evaluator, or the product contract.
Grade the system at three surfaces
The final answer is only one surface of an AI system. For a tool-using agent, a useful evaluation model separates the run, the trace, and the outcome.
A run is one model call or tool invocation. It is where schema validity, token budgets, provider errors, and local tool decisions can be checked cheaply. A trace is the complete execution path for one user task: model calls, retrievals, tool invocations, guardrails, retries, and final response. The outcome is the state of the world after execution: a database row, a generated file, a ticket transition, or the deliberate absence of any mutation.
| Surface | Grade deterministically | Grade semantically |
|---|---|---|
| Run | JSON schema, allowed tool, argument shape, token count | Whether a local decision was reasonable |
| Trace | Call sequence constraints, forbidden tools, retry count | Groundedness, completeness, clarity of uncertainty |
| Outcome | State diff, emitted event, approval status | Human usefulness and business acceptability |
This separation prevents a common mistake: asking a language model to grade facts that a program can check exactly. If the database changed, compare the database. If the tool was forbidden, match the tool name. If a response must contain a policy version, assert it. Reserve model-based judges for qualities that genuinely require interpretation.
The evaluation itself can have a contract:
type EvaluationResult = {
caseId: string;
hardFailures: string[];
softScore: number;
outcome: "pass" | "fail" | "review";
traceId: string;
costUsd: number;
latencyMs: number;
};
function decide(result: EvaluationResult) {
if (result.hardFailures.length > 0) return "fail";
if (result.softScore < 0.82) return "review";
return "pass";
}A single aggregate score hides too much. A system with 94 percent helpfulness and one unauthorized write is not healthier than a system with 87 percent helpfulness and no authority violations. The gate must reflect risk, not just average sentiment.
Connect evals to business outcomes without pretending causality
Technical teams often stop at “the judge score went from 0.71 to 0.78.” That is useful only if the number changes a decision. Product teams need to know whether the system reduces handling time, improves resolution, lowers escalation, or creates expensive rework.
The bridge is not to force every response into a simplistic revenue label. It is to attach a business observation to the same case or cohort that produced the technical trace.
Suppose a customer-support agent proposes reply drafts. A useful measurement chain might look like this:
case -> trace -> technical graders -> reviewer action -> customer outcomeThe technical graders can check citation coverage, policy compliance, and tool behavior. The reviewer action can record accepted, edited, rejected, or escalated. The customer outcome can record reopened ticket, time to resolution, or satisfaction signal. The chain is not proof that the model caused every business result, but it gives the team a way to investigate whether improvements survive contact with work.
| Technical signal | Operational signal | Business question |
|---|---|---|
| Groundedness score | Reviewer edit rate | Are people correcting the same factual gaps? |
| Tool-contract pass rate | Escalation rate | Is the agent safe enough to handle the intended tier? |
| Median and p95 latency | Handle time | Does the system make work faster, not merely smarter? |
| Cost per successful task | Cost per resolved case | Is the capability economically sustainable? |
| Regression failures | Rollback or hotfix count | Is release quality improving over time? |
There are two traps here. The first is optimizing a proxy because it is easy to measure. The second is waiting for perfect attribution before instrumenting anything. Start with a narrow, decision-relevant cohort and document what the metric can and cannot claim.
Design release gates as a matrix, not a magic number
The right release gate depends on the risk of the behavior. A low-risk internal summarizer and a payment-authorizing agent should not share the same threshold.
Use hard gates for properties that are non-negotiable. Use soft thresholds for qualities that can improve gradually. Send ambiguous results to review instead of turning uncertainty into a false pass.
| Risk tier | Hard gate | Soft gate | Promotion policy |
|---|---|---|---|
| Low | No schema or privacy violations | Helpful score above baseline | Automatic if cost and latency are stable |
| Medium | No forbidden tool or unsupported claim | Quality not below agreed floor | Canary with sampled review |
| High | No unauthorized mutation, leak, or policy bypass | Domain score and human review threshold | Manual approval and rollback plan |
| Critical | Zero critical failures in protected cases | N/A for hard safety properties | Do not promote on aggregate score alone |
A gate should also specify the comparison baseline. “The new model is better” is not a test. “The new model has no critical regression, improves accepted-draft rate by at least three points on the same cohort, and stays within the cost envelope” is a decision rule.
Make evaluation cheap enough to run continuously
A perfect evaluation suite that takes six hours will be bypassed. The practical answer is a tiered suite.
The pull-request lane should run deterministic, low-cost cases: schema validation, tool allowlists, state invariants, prompt-injection fixtures, and a small set of golden traces. The pre-release lane can run broader replay with multiple trials per case and selected judge-based graders. The post-deployment lane should sample real traffic with privacy controls, compare cohorts, and watch for drift.
| Lane | Frequency | Cases | Main purpose |
|---|---|---|---|
| PR | Every change | Small, deterministic, high-risk regressions | Stop obvious breakage early |
| Release | Model/prompt/tool/index change | Full golden set with repeated trials | Decide promotion |
| Production | Continuous sampling | Sanitized real traces and business cohorts | Detect drift and hidden cost |
| Incident | On demand | Replayed failure plus neighboring cases | Prevent recurrence |
The cost of a case is not just model tokens. It includes fixture maintenance, grader maintenance, review time, and the cognitive cost of interpreting a failure. Keep the suite small enough that ownership is explicit. Delete cases only when the underlying contract is no longer meaningful, not because a red test is inconvenient.
The production loop: observe, label, change, replay
An eval system becomes valuable after launch. Production failures reveal language, workflows, and combinations that the original team could not imagine. The loop should turn those discoveries into durable engineering knowledge.
observe -> sanitize -> cluster -> label -> add or refine case
-> change system -> replay -> compare -> promote or revertThe sanitize step matters. Traces often contain customer data, secrets, or proprietary prompts. The evaluation record should preserve the failure signal while minimizing copied sensitive content. OWASP recommends sanitization, least-privilege access, tokenization, and redaction as part of reducing sensitive-information disclosure risk.
The cluster step prevents a hundred similar tickets from becoming a hundred noisy test cases. Group failures by invariant: wrong tenant, stale policy, unsupported action, missing citation, retry storm, or poor clarification. The test suite should encode behavior, not the exact wording of one customer.
Finally, record the reason a case changed. A regression case without a short history becomes hard to trust. A case that says “added after INV-1042 double-refund incident” carries the institutional memory that an aggregate score cannot.
What senior engineers should refuse to ship
There are three warning signs that an AI system is not ready for a meaningful rollout.
First, the team cannot define the outcome independently of the model’s final text. If the system changes a state, creates a file, sends a message, or recommends a decision, the outcome must be observable outside the model response.
Second, the team has only happy-path examples. Without ambiguity, missing evidence, tool failure, authorization and regression cases, the suite is measuring a demo rather than a system.
Third, the release decision is a single score with no risk decomposition. Averages are useful for trends, but they are poor substitutes for authority boundaries, data protection and operational budgets.
The goal is not to make an AI system deterministic. The goal is to make its variability legible, bounded, and actionable.
Closing perspective
Evaluation is often introduced as a testing task owned by an ML engineer. In a production AI system, it is broader: it is the place where product intent, architecture, security, operations, and economics become executable enough to disagree with one another.
Start with a small golden set. Split capability from regression. Grade runs, traces, and outcomes separately. Use code for hard facts and judges for semantics. Attach technical evidence to business observations. Build risk-aware gates. Then let production failures improve the suite instead of disappearing into a support queue.
The most mature AI team is not the one that claims its model rarely fails. It is the one that can show which failures are unacceptable, which are improving, who owns them, and why the next release deserves to go live.
Comments (0)