AI gets things wrong: how to check the answer before deciding
Trusting without checking is the fastest route to being confidently wrong. The four error types that show up in practice, how to recognise each, and the ten-second checking habit.
An assistant connected to your data does get things wrong, and the shape of the error is the dangerous part: a wrong answer looks exactly like a right one. Round number, confident sentence, no hesitation. So checking cannot depend on the answer "looking odd".
The four errors that actually occur
| Error | How it shows up | How to catch it |
|---|---|---|
| Different scope | A number close to yours, but not equal | Ask for the filter used and compare criterion by criterion |
| Misread question | Answers something else, confidently | Ask it to restate the question before answering |
| Wrong data at the source | A correct answer over incorrect data | Check against cash, statements or a manual count |
| Incomplete source | A number far too low with no explanation | Ask how far each source has data |
In most first-week discrepancies the model is right and the criterion differs. The common error is not the AI inventing — it is you and it counting different things under the same name.
The ten-second habit
Add one sentence to every question that will drive a decision: "show the filter you used and how many rows went into the calculation". That turns the answer from something to believe into something to check, at no perceptible cost.
- If the row count is far lower than expected, a source is probably missing or the period is wrong.
- If it is far higher, there is probably duplication — the same order counted twice by a repeated sync.
- If the filter names a status you would not use, the criterion diverged and the discussion is about definition, not data.
The check worth doing once
Pick three numbers you already know from another source — last month’s revenue, total orders, spend — and ask the assistant for all three. If all three match, the base is trustworthy for most questions. If one does not, you found a data problem that existed before the AI.
Connected AI does not create data errors. It exposes the error that was already there and that nobody had time to look for.
When to be more suspicious
- 01When the answer is exactly what you wanted to hear. Confirmation bias works with machines too.
- 02When the number is too round. An exact total usually means a result limit, not a coincidence.
- 03When the question involved two sources. The join is where almost every analysis error lives.
- 04When nobody can say where the data comes from. A source with no owner is a source with no maintenance.