The confident wrong answer: how to catch an AI making things up

Here is the failure mode that should worry you, and it isn't the one in the movies. A lawyer in a widely reported 2023 US federal case filed a brief citing six court decisions that supported his argument perfectly. They were exactly on point. They were also entirely invented — an AI had produced them, complete with plausible case names, plausible citations and plausible judges, and the lawyer had trusted them because they read exactly like real law. The problem was never that the model was wrong. The problem was how good it was at being wrong.

That's the whole thing you need to internalise about these tools: a language model is trained to produce text that is *plausible*, not text that is *true*. Most of the time plausible and true overlap, which is why the tools are useful. The skill — the one that separates people who get burned from people who don't — is knowing exactly where that overlap breaks, and having a thirty-second habit for the moments that matter.

Where models bluff (it's predictable)

Hallucination isn't random. It clusters, and once you know where, you can aim your skepticism instead of spreading it evenly over everything and exhausting yourself. The pattern underneath every cluster is the same: the model is reaching for a specific fact it may not actually hold, so it generates the shape of a right answer instead. The fluency stays perfect even when the facts underneath have quietly evaporated. Here is the map, in the order you'll meet them.

Read that list the other way round and it tells you where these tools are safest: vague, general, widely repeated knowledge. Ask a model what a balance sheet is and it will be reliable, because the answer is written in ten thousand places and the model has genuinely absorbed the shape of it. Ask it what a specific company's balance sheet said in a specific quarter and you have moved into the territory where the answer either exists precisely in its weights or gets manufactured to fit the sentence. Sharp and checkable is where models invent. That is not a defect you can prompt your way out of; it is what the training objective produces, and it is why the habit below matters more than any clever wording.

Fluency is not accuracy. The model's confidence is a property of its writing style, not its knowledge — never read one as evidence of the other.

The thirty-second check

You don't need to fact-check everything; that would defeat the point of using the tool at all. You need to check the things that would *matter if they were wrong*, and you need the check to be short enough that you actually run it. Three questions do it, in this order, and the order matters — the first one is a filter that saves you from running the other two on things that don't deserve them.

Step three is the one worth practising, because it is the one that generalises. You are not asking the model to be honest — it has no access to whether it is being honest. You are applying pressure and watching which way the text moves. Grounded answers get narrower under questioning: a range becomes a figure, a vague attribution becomes a document, a claim acquires a stated limitation. Bluffed answers get wider: the specific becomes the general, the assertion becomes a hedge, and somewhere in the third sentence of the reply the original claim quietly stops being made at all. Once you have seen that reversal twice you will recognise it instantly, and it costs one message to trigger.

It helps to know what you're looking for before you go looking, because both replies are fluent and both are polite, and on a fast read they feel similar. They aren't. The difference shows up in what the reply *contains*, not in how confident it sounds, and that is the whole trick: you're reading for named objects — documents, dates, numbers, stated limits — rather than for tone. The table below is what a decade of this looks like compressed into five rows, and it is worth reading once before you need it rather than in the moment you do.

Two habits that make it automatic

First, demand sources for anything factual and then actually open one. The single most common mistake isn't trusting AI — it's trusting AI that provided a link nobody clicked. Real citations are checkable; invented ones dissolve the moment you look. Second, for anything genuinely important, ask a second, different model the same question and watch where they disagree. Agreement isn't proof, but disagreement is a bright flare pointing exactly at the sentence you need to verify yourself. This is, more or less, how our own newsroom's Verification Agent works: nothing precise ships unless it's confirmed against a primary source, and any figure that lives in only one place gets labeled or cut. You can run the same discipline in your own head for free.

What this does not protect you from

Be clear about the ceiling here, because a check you over-trust is worse than no check at all. This habit catches sloppy invention — the fabricated citation, the too-round number, the quote nobody said. It does not catch a wrong answer that is confidently wrong in a way that happens to survive one round of questioning, and it does not catch an error in reasoning that is built on facts which are each individually true. It also does nothing about omission: a model that leaves out the one consideration that changes your decision has not hallucinated anything, and no amount of asking it for sources will surface what it never mentioned. Those failures remain unverified by anything in this guide, and the honest answer is that catching them requires knowing the subject yourself.

One more piece of housekeeping, in the spirit of the thing. This guide links no external source for the court case in its opening, because we do not have a primary document in front of us to point you at, and a guide about fabricated citations is the last place on earth to attach a citation we have not personally opened. It was widely reported at the time and it is easily found; treat the detail as reported rather than as verified here, and apply step four of the check to it exactly as you would to anything else. The two references below are internal pages, labelled as such, and neither is offered as evidence for that case.

None of this makes AI less useful. It makes it *safe* to use for the things that matter — which is the only way to actually rely on it. The habit is small on purpose: one sentence per answer, three questions, one click. It survives contact with a busy week, which is the only property that matters in a habit, and it scales down gracefully — on a day when you have thirty seconds you still run step one, and step one alone catches the worst of it. The people who get value from these tools long-term aren't the ones who trust them most or least. They're the ones who know precisely which sentence to double-check, and never skip it.