Your Automation Runs. How Do You Know It Works?

Imagine you run an accounting firm and a freelancer built you an automation that files client invoices. Or your nephew set up a booking reminder with ChatGPT. The demo was smooth, so you stopped thinking about it.

Three months later a client calls: the invoice you "sent" never arrived. By then there are more of them. Lost clients, late payments and a reputation for being careless are expensive, and none of it came with an error message.

"It runs" and "it works" are two different things

Think of a delivery van that leaves the depot on time every morning. The engine runs, the route is on schedule, the driver waves. Nobody has checked whether the parcels are still in the back when it gets there.

An automation is the same. Running means no step crashed. Working means the right thing arrived at the right place. AI makes this gap wider, because it builds tidy, confident-looking systems very quickly. Even OpenAI describes the errors of its models as plausible but false statements. I wrote about that pattern in why AI sounds sure when it is wrong.

Five checks you can do yourself

None of these needs technical knowledge. They need a pen, a spreadsheet and the willingness to ask.

1. Run 10 real cases through it

Do not use the cases the builder prepared for the demo. Take 10 real ones from your own inbox: last week's enquiries, real invoices, actual bookings. Include one awkward case, such as a client with a long name, an empty field or a message sent twice.

Then follow each one to the end and write down where it landed.

2. Count what goes in and what comes out

This is the cheapest check there is. If 40 enquiries arrived through your website this week, how many are in your customer list? If 100 invoices were created, how many reached a client? The two numbers must match, or you must be able to explain the difference item by item.

A gap with no explanation means something is being lost, quietly.

3. Ask for a one-page map

Ask whoever built it for a single page showing the steps: what goes in, what comes out, and who or what is responsible for each step. Plain boxes and arrows are enough.

If nobody can produce this page, nobody fully knows how the system behaves.

4. Ask where you can see what happened to each item

For one given enquiry or invoice, you should be able to look at a screen, a sheet or a list and see its story: received, processed, sent, when. If the only answer is "we would have to look in the code", you are flying blind.

5. Ask what happens when something goes wrong, and who hears about it

Emails bounce, forms arrive with missing data, a service you depend on has a bad hour. A good system expects this. The question is who gets a message when it happens: you, the builder, nobody?

Many tools can send an alert when a run fails. The documentation of one of the most common automation tools describes an error workflow that runs when an execution fails, for example to send an email alert. It only helps if someone set it up and someone reads it. I describe the other half of this problem in what to do when an automation stops silently.

What we found while building our own system

I am telling this one midway, on purpose, because the system is not finished and I will not claim it works.

What caught it was not a failing test. It was one question: what does this step do with a long message?

What to ask, and what a good answer sounds like

You are the client, so you may ask. Here is how to tell a good answer from a bad one.

Your checklist for this week

1. Pick one automation your business already depends on: invoices, enquiries, bookings, reminders.
2. Pull 10 real cases from last week and follow each to the end.
3. Count what went in and what came out for the same week.
4. Ask the builder for the one-page map and the screen that shows each item.
5. Ask what happens when something fails and who hears about it. Write down the answer.
6. Put the three weakest answers in an email to the builder and ask for dates.

How we help

We start with an analysis: we map one process step by step and look for the places where a result can be lost. Then we automate the whole process, including the checks and the monitoring that tell you when something goes wrong. That is the idea behind the HARDEN method: we try to break the system before it goes live, and we keep watching it afterwards.

The same reasoning, in plain language for managers, is taught on the AI Ready Standard site (in Bulgarian). If you want a map of one of your own processes, the Process X-ray is the place to start. Your next step is smaller: do check number 2 on one real week before you read anything else.