Most AI projects are tested the way a magic trick is tested: someone runs it in front of an audience, it works, everyone agrees it works. Then it goes into production, where the inputs are not curated and nobody is watching, and it starts being confidently wrong about 4% of the time. Nobody notices for a month.
This is not a model problem. It is a testing problem, and it is solvable with a discipline that predates AI by decades. Here is what actually works, whether you are building the system or paying someone else to.
Why normal testing does not catch it
Ordinary software is deterministic: the same input gives the same output, so a test asserts equality and either passes or fails. That is why unit tests work.
A language model breaks the assumption. The same question can produce two differently-worded correct answers, or a correct answer today and a subtly wrong one after a model update you did not control. `assert output == expected` is useless here, so a lot of teams conclude AI cannot be tested and fall back on eyeballing it. That conclusion is wrong. The assertion changes; the discipline does not.
Evaluations are test cases with a grader
An evaluation is a stored input, a definition of what a good answer looks like, and a way to score the actual answer against it. Run a few hundred of them and you get a number you can watch over time. In practice they come in a few shapes:
- Golden answers — questions whose correct answer is known and fixed. Scored on whether the key facts are present, not on wording.
- Grounding checks — for anything retrieval-based, does the answer actually follow from the retrieved documents, and does it cite them? This is what catches confident invention.
- Abstention cases — questions the system should refuse. A system that answers everything is a system that will answer wrongly. 'I don't know' is a correct output and deserves a test.
- Adversarial and edge inputs — the empty document, the 400-page one, the two contradictory sources, the prompt-injection attempt hidden in an uploaded PDF.
- Regression cases — every single thing that has ever gone wrong in production becomes a permanent case. This is the highest-value set and it is free; you are just refusing to forget.
They have to run automatically
An evaluation suite that someone runs manually before a big release is barely better than nothing, because the changes that break AI systems are small and constant: a tweaked prompt, a new chunking strategy, a provider upgrading the model underneath you.
So they run in CI, on every change, and the build fails when the score drops below the agreed floor. Same contract as a normal test suite. The only real difference is that the pass mark is a threshold rather than a boolean, and that you re-run the whole thing whenever the model version moves — because that is a dependency upgrade, even though it does not show up in your lockfile.
Keep the model away from anything exact
The most effective reliability technique is not a testing technique at all. It is refusing to let the model do work that has a single correct answer.
Arithmetic, tax rates, date maths, anything with a compliance definition — that goes in ordinary code, which can be unit-tested to certainty. The model gets the work it is genuinely better at: reading unstructured text, classifying, summarizing, explaining, deciding what looks unusual.
On the IFTA filing agent this is the entire architecture. Deterministic Python computes every number that lands on the tax form. The model reads the finished return and writes a review note flagging what a careful human reviewer would flag. A person approves before anything is filed. The result is a system where the probabilistic component cannot produce a wrong number, only a wrong opinion about a right one — and a human is reading the opinion.
That is also why a backtest was possible: because the output is deterministic, it could be run against a real prior filing and compared exactly. It matched to the penny. You cannot do that with a system that lets a model do the arithmetic, which is a good reason not to build one.
Human gates on anything irreversible
Some actions cannot be undone: filing a return, sending a message to a customer, changing a live price, deleting a record. For those, the correct amount of autonomy is none. The system prepares the action and stops.
The engineering detail that matters is where the gate lives. If the check sits in the calling code, the next caller forgets it. It belongs inside the thing performing the action, so that skipping it is not an available option — and an approval should authorize one action, not become a token that a retry can replay.
What to demand from whoever builds yours
If you are commissioning an AI system rather than writing one, most of this reduces to four questions:
- How many evaluation cases does this ship with, and can I see them?
- Do they run automatically on every change, and what happens when the score drops?
- Which parts are deterministic code rather than the model, and why those?
- What can this system do without a human, and what stops for approval?
Vague answers here are the single most reliable predictor of an expensive disappointment. This is not a niche concern, either — it is the difference between a system your team comes to rely on and one they quietly stop trusting after the second bad answer.
If something is already built — by your team or another vendor — and you want to know what it does on the inputs nobody tried, this is that engagement: functional, regression, and edge-case coverage with a written report of what breaks and why.
Independent testing and evaluationAnd if you would rather see the discipline applied end to end on a real system, the IFTA case study documents every one of these decisions on a build that a paying client files taxes through.