Your AI-built internal tools work. How do you make them reliable?

In 2026 a lot of businesses have a new kind of software: tools an owner or a manager built with an AI assistant in a few weeks, that nobody on the payroll could have written a year ago. Dashboards fed from the company's cloud systems. Scheduled syncs between two platforms that never talked. TV boards on the shop floor. A weekly reconciliation that used to take an afternoon. They work, people depend on them, and they were never designed to be depended on.

The short answer first. An AI-built tool becomes reliable when the parts that must run unattended are moved out of the chat window and the browser into plain code on a machine nobody logs into: a service account instead of a personal login, a scheduler instead of a laptop that happens to be on, a status page that says what ran, and a backup you have restored at least once. The AI keeps the work it is good at, which is building the next thing and answering questions about the data. Most of this is days of work, not months, and none of it requires throwing away what already works.

The rest of this article is the longer answer: why these tools break the way they do, what to change, and what to leave alone.

The pattern I keep seeing

When a business shows me what it built with an AI assistant, the setup is almost always the same, because the assistant built it the fastest way that worked:

• Everything runs through one logged-in browser session on one person's laptop. The assistant calls the vendor's API from that session because nobody has an API key yet.
• Each screen on the floor runs its own little local server, started by hand after the last reboot.
• There is a homemade login page in front of the tablet site.
• Credentials live in the browser, in a script, or in the chat history.
• The vendor system has monitoring and alerts for production. Nothing watches the tools built on top of it.
• One person knows how it all fits together, and that person has another job.

None of this is a mistake. It is what a working prototype looks like, and the prototype has already proven the idea is worth having. The question is what happens the first morning the login expires at 6 a.m.

Why it fails, and why nobody notices

AI-built tools fail quietly. A scheduled sync that depends on a browser session does not crash when the session expires; it simply stops, and the board on the wall keeps showing yesterday's numbers. A local server that was not restarted after a Windows update shows a blank screen, and the first person to notice is a customer at will-call. A cleanup script that an assistant ran against the live system removed "stray records" that turned out to be somebody's work.

The vendor's monitoring will not catch any of this. It watches its own system, which is fine. The layer you built is yours to watch.

What to change

Six moves, in the order I do them. The first four are the ones that end the 6 a.m. phone calls.

1. A service account and a stored secret, instead of a person's login

Ask the vendor for an API token or a dedicated integration user. Keep the key in a secrets store on the server, never in a browser, a spreadsheet or a chat. If the vendor has no API yet, say so in writing and plan around it: this one item sets the timeline for everything else.

2. One always-on host, in your own account

A small virtual machine on your network or a small cloud box in the company's own cloud account, not the consultant's. Everything that runs on a schedule moves here. The screens and tablets just open an address; if one goes blank, you refresh the page instead of restarting a server.

3. Scheduled jobs as services

Each job gets a log, a retry, a restart on failure and a known owner. This is where the AI's code usually needs the most work, and it is also where I turn the model's involvement off: anything that has to run every night without a human watching should be plain, tested code. The model is for judgment and language, not for the 2 a.m. sync.

4. A status page and one alert

A page that shows every job green or red, when it last ran, and how fresh the data on each board is. One alert, to one person, when something goes red. Your factory already has a light for machines. This is the same light for your tools.

5. A nightly backup you have restored

Back up the database and the configuration every night, off the host. Then restore it once, on purpose, and time it. A backup that has never been restored is a hope.

6. Version control and a review step

Put the code in Git. When someone builds the next feature with an assistant, it goes through a review before it reaches the floor, and there is a known-good version to roll back to. Write a one-page runbook for the person who will keep it running: what runs where, what the colors mean, who to call.

What to leave alone

The model does not need to come out of the picture, and the person who built the tools should keep building. What changes is where the model sits.

• It keeps doing the planning, the analysis and the one-off cleanups, with a human looking at the result.
• It reads from a database that the plain code keeps current, instead of poking the live vendor system through a browser.
• It proposes changes to live data; it does not make them. Anything that creates, changes or deletes a record goes through a check first, by a person or by a rule.
• It should be swappable. Put the model behind a thin interface so that if the vendor changes prices, limits or terms, you change a setting rather than rebuilding. Keep the prompts and the test cases in the repository with the rest of the code.

Testing without a testing program

You do not need a QA department. You almost certainly already have the first test: the weekly reconciliation someone runs by hand, comparing the numbers in one system with the numbers in the other. That check is the test. Run it every night, automatically, and have it alert only when the numbers do not match or the job did not run.

Three checks cover most of what goes wrong with these tools:

• Did every scheduled job run, and did it finish without errors?
• Is the data fresh? Every board and report should know how old its data is and say so when it is stale.
• Do the numbers match between the systems that are supposed to agree, within the tolerance you decide on?

That is the whole program to start with. Nothing about it is exotic. I came to AI engineering from software testing, and I have found that the discipline transfers directly: the systems that stay reliable are the ones where the checks run on a schedule and somebody is told when they fail. The same idea, applied to a codebase, is what my open-source QA agent, Verdict, does nightly against a production repository. It has found real defects in libraries other people depend on, and it signs its own run history so a copied result cannot pass for a real one. The point is not the tool; it is that checking became a habit instead of an event.

How much work this is

Less than most owners fear. On the projects I quote, the foundation — service account, host, jobs as services, status page and alert, backup and restore test, runbook — is typically days to a couple of weeks of engineering, depending mostly on how quickly the vendor provides API access. A proper data layer with reports and alerts is a second step, scoped after the first one is running.

Prices for this kind of work are published on this site: data and integration work starts at $1,500 as a fixed scope, and AI automation projects at $8,000 as a fixed quote, with the number in writing before anything starts. A review of what you already have is the right first step when the tools are already running the business, because the review tells both of us what the fixed price should be.

Eight questions to ask about the tools you already have

• If the laptop they run on is turned off tonight, what stops?
• Where do the credentials live, and who else could log in?
• How would you find out that a scheduled job did not run?
• How old can the data on the board get before anyone notices?
• Has the backup ever been restored?
• Can anything the assistant does change or delete live records without a person approving it?
• Is the code in version control, and is there a version to roll back to?
• If the person who built it left tomorrow, who could keep it running from the documentation that exists today?

If you can answer all eight, you do not need me. If more than two of them make you pause, the fixes above are the order to do them in.

And if what you need is only the review, say so. Telling you that the foundation is sound and sending you back to building costs me a project and saves you a quarter, which is a trade I will take every time.