There's a moment in almost every AI project where someone changes a prompt, the output looks a little different, and the room goes quiet. Is it better? Worse? Nobody actually knows. Someone eyeballs a few examples, decides it seems fine, and it ships. That moment, the shrug followed by the shipment, is the single most under-appreciated risk in building with language models, and the only real defense against it is something most teams treat as a chore: evaluation.

An eval is just a way of answering one question with evidence instead of opinion: did this change make the system better or worse? It sounds too basic to need defending. But the absence of a good answer to that question is, in our experience, the thing that quietly sinks AI products. It isn't a bad model or a clever competitor; it's an accumulation of changes nobody could measure, slowly drifting somewhere worse.

Why this is an AI problem and not a normal software problem

Ordinary software has a property so fundamental we forget it's there: run it twice on the same input and you get the same output. That determinism is the bedrock under nearly every tool engineers trust. Unit tests, regression suites, a red build that blocks a merge: all of that accountability is free, and it's free precisely because the behavior is repeatable. Put a language model in the loop and the bedrock disappears. The same input can produce different outputs, and a reworded prompt, a model version bump, or a slightly different document order each nudges the result in ways you can't fully predict. Traditional testing stops applying, because there's no single “correct” output to assert against.

That doesn't mean reliability is impossible. It means reliability is no longer observed; it has to be constructed. You have to rebuild by hand the testing culture you got for free in deterministic software, and evals are that reconstruction. Teams that skip the step aren't moving faster than everyone else. They've just given up the ability to know what they shipped.

The most expensive sentence in AI

“It seems a little worse lately.”

You cannot do anything with that sentence. You can't bisect a feeling, open a ticket against a vague sense of decline, or tell whether the culprit was last week's prompt edit or last month's model upgrade. By the time the degradation is obvious enough to name, you've lost the thread connecting it to the change that caused it.

This is the failure mode evals exist to prevent, and it's insidious because nothing dramatic happens. There's no outage, no stack trace, no alarm. The product just gets a little less trustworthy with each unmeasured change, and the people relying on it are the last to find out. In a consumer toy that's an annoyance. In a domain where the output informs real decisions about cost or care, “a little less trustworthy” is the whole risk.

The fix is almost embarrassingly simple to state and genuinely hard to practice: decide, in advance and in writing, what “good” means for a task, then measure every change against that definition on data the system hasn't seen. For us, that's more than a labeling project. Our ground truth is the reviewer-corrected contract model: the version a human already went in and fixed on the contracts we'd gotten wrong. Ground truth is what our reviewers already fixed. There's no screenshot of a regression you prevented, which is exactly why the discipline is so consistently undervalued.

Why demos lie, and so do bad evals

The reason teams convince themselves they don't need evals is the demo. A demo is a curated best case: you show the system handling a hard example, everyone's impressed, and the impression substitutes for evidence. But a demo proves the system can work, which is the easy thing to establish. It says almost nothing about whether it reliably does, which is the only thing that matters once real data and real users show up.

The catch is that bad evals lie too. A generic scorecard, say a single pass rate or a model-as-judge dutifully rating “helpfulness” one to five, produces a comforting number that means very little. A high pass rate on a test you designed to be easy isn't a signal; it's a mirror. That's worse than no number at all, because it manufactures confidence you haven't earned, and confidence is exactly what stops you from looking closer.

Good evals come from the opposite direction. The best ones grow out of specific, recurring failures you got tired of rediscovering: you catch one concrete mistake automatically, then generalize only once you have several. An eval done well is less like a report card and more like a debugger, only as good as your understanding of the bug it was built to catch.

This isn't a fringe view. Hamel Husain, in some of the most widely read practitioner writing on the subject, argues that error analysis, patiently reading your system's real outputs, is where most of the value in evals actually lives; his FAQ is worth reading in full. Shreya Shankar and Eugene Yan have written the same lesson from different angles: eval discipline is what separates AI systems that stay reliable from ones that quietly drift, and generic scorecards produce reassuring numbers that mean very little. The through-line among people who have shipped this is consistent. The measurement has to grow out of the mistakes, not the other way around.

To make it concrete, take a system we actually run: an extractor that reads rebate contracts and pulls out dates, rebate amounts, and program structure. A lazy eval asks “did the model return something shaped like a date?” It passes at ninety-nine percent, and everyone relaxes. A useful eval asks “is it the correct date, on the contracts our reviewers have already had to fix?” When we first ran that eval it sat at 76.6% on date fields, 848 of 1,107 correct. It moves every time someone touches the prompt or reorders the input. The first eval measures whether the model can type. The second measures whether it's correct. Only one of them will ever catch a regression, and it's never the one with the reassuring number.

Better, not just safer

Everything so far is defensive: evals keep you from shipping something worse. But the real reason to invest in them is that they're also the fastest way to make a system better. Evaluation and improvement are one loop. You read real failures, find a pattern, change something to fix it, and then measure whether the fix actually helped without breaking three other things. Skip that last step and you aren't improving the system, you're just rearranging its failures.

Two lessons from building that loop are worth passing on, because they're where the naive version breaks. The first is speed. An improvement loop you dread running is one you won't run, and re-evaluating is expensive, since every example is one or more model calls. The obvious optimization, caching each model call, barely helps for anything multi-step, because each step folds the previous step's output into its own prompt, so a small change upstream busts every cache downstream. What worked for us was caching the whole result for an example, keyed on everything that legitimately changes it, including a fingerprint of the prompt itself. Change a prompt and only the affected examples re-run; change nothing and the whole suite is nearly free. The eval invalidates itself, correctly, with no bookkeeping, so running it after every change costs almost nothing, and you actually do.

This matters double for agents. An agent is a cascade of steps, each conditioned on the last, which is exactly where errors compound and a per-step view hides the damage. You have to evaluate the whole trajectory, not the individual call, and you have to make re-running that trajectory cheap enough to do constantly. If you can't, you won't, and the agent quietly drifts.

The second lesson is that “better” and “worse” have direction, and a naive gate gets it wrong. A regression isn't just “a number moved.” Accuracy falling is worse; an error rate rising is worse in the opposite direction; a check that used to pass now failing is worse whatever the number behind it says. Meanwhile some changes only look like regressions: a category that vanished between two runs usually means you reorganized your taxonomy, not that quality dropped. And some comparisons should be refused outright, like grading one model against another model's baseline, where the delta is meaningless. An eval that flags every movement cries wolf until the team mutes it, and a muted eval improves nothing.

“But our data is too sensitive to test on”

In a lot of serious domains there's a real obstacle to all of this: the data you most need to evaluate on is the data you're least allowed to copy around. You can't dump production records into a test fixture and commit it to a repository. For many teams this becomes the excuse that ends the conversation.

It shouldn't, because the constraint has a clean solution. The trick is to separate the structure of an evaluation from the sensitive content it runs on. What you commit to version control is only a set of opaque pointers: identifiers, plus the category of failure each case was chosen to stress, and never the underlying records. At run time those pointers resolve against access-controlled systems to fetch the real content, the evaluation scores against a reviewed ground truth, and only the resulting metrics come back out. The sensitive data never enters the repository; only the shape of the problem does.

In our repo a single case is one JSON object:

{
  "rebate_program_id": "RBP_2f3a91d",
  "internal_contract_id": "CTR_0a1b7c9",
  "cluster_tag": "wrong_dates"
}

{
  "rebate_program_id": "RBP_2f3a91d",
  "internal_contract_id": "CTR_0a1b7c9",
  "cluster_tag": "wrong_dates"
}

{
  "rebate_program_id": "RBP_2f3a91d",
  "internal_contract_id": "CTR_0a1b7c9",
  "cluster_tag": "wrong_dates"
}

Two identifiers and a category tag. Nothing else. What comes back out after a run is a metrics snapshot, also committed, that a reviewer can diff:

A reviewer looking at that pull request sees "this change adds twelve wrong_dates cases" or "date accuracy 0.766 to 0.810," in a normal code review, without ever opening a contract.

That constraint tends to produce better systems anyway, for two reasons. First, an eval that stores only pointers has to fetch and parse a case the same way production does. The two paths cannot drift; a bug in the fetch is a bug in both, and you find out at eval time instead of on a customer call. Second, forcing every case to carry a failure category bakes the error-analysis discipline the post argues for into the file format itself. You can't add a case without deciding what it's supposed to prove.

We adopted that pattern because we had to. “Our data is too sensitive” is a design constraint, not a reason to fly blind.

When evals aren't worth it

None of this is free, and it isn't always worth doing. Building and maintaining an eval is real engineering effort, and for genuinely disposable work, a prototype you'll throw away next week, a one-off script, an internal tool with a single forgiving user, the honest move is to skip the machinery and just look at the output. Rigor you don't need is its own kind of waste.

The trap isn't skipping evals on a toy. It's failing to notice when the toy has quietly become something people depend on. The calculus flips the moment a system is going to live, change hands, and inform decisions that matter, and it flips earlier than almost anyone wants to admit. If you're going to be wrong about the timing, be wrong by asking “how would we know if this got worse?” a few weeks too early, rather than one incident too late.

Evals are how a demo becomes a product

AI is moving from things that demo well toward things people actually depend on, and that transition is an engineering problem long before it's a modeling one. Nobody gets into AI because they're excited to write evals. They're tedious, unglamorous, and they slow you down at exactly the moment you most want to move fast. But they're the difference between a system that happens to work today and one you can stake a decision on tomorrow. In a field crowded with impressive demos, the teams that earn trust won't be the ones with the cleverest prompt. They'll be the ones who can answer, with evidence, a deceptively boring question: how do you know it still works?

Photo by Markus Spiske on Unsplash