People say an AI "lied" to them when it invents a fact. That's the wrong word. Lying means you know the truth and choose to hide it, but the model has no idea what's true. It isn't deceiving you.

To use these tools well, you have to understand what it’s doing. Because the reason a chatbot makes things up has nothing to do with it being broken. It's doing exactly what it was built to do.

A language model is a prediction engine. You give it some text, and it predicts the next word, then the next, then the next, based on patterns it absorbed from an enormous pile of data. There is no internal library of facts it looks things up in, no database it checks your question against. It is generating the most statistically likely continuation of your prompt, one word at a time.

Most of the time, the likely continuation is also the true one. "The capital of France is" reliably produces "Paris" because that pairing shows up constantly in the training data. Plausible and correct line up, and you get a right answer.

The trouble starts when they come apart. Ask something obscure, something with thin coverage in the data, and the model still produces the most plausible-sounding answer it can assemble. It doesn't know that answer is shaky. It has no sense of its own certainty. It just generates fluent, confident text, because generating fluent, confident text is the only thing it does. When that text happens to be wrong, we call it a hallucination.

Models get trained and graded on tests. Thousands of benchmark questions, scored on how many they get right. And on almost every one of those tests, a confident guess is worth more than "I don't know." Guess, and you might get a point. Say you're unsure, and you score zero, same as being wrong. So over millions of rounds of this, the model learns the optimal strategy. When you don't know, guess anyway. Never leave a question blank.

It's the exact behavior of a student guessing an exam answer they're unsure of. Bluffing gets rewarded, so bluffing becomes the habit. Recent research from OpenAI showed that models hallucinate in part because the way we grade them punishes honesty about uncertainty. We built a machine that guesses because we only ever rewarded guessing.

Knowing the mechanism tells you exactly where the danger lives. It's worst precisely where you're least equipped to catch it.

The thinner the training data on a topic, the more the model improvises, so the risk climbs with obscurity. Things like specific numbers, direct quotes, citations, names, or edge-case details in a niche field pose more risk. Ask it for a source and it may provide a realistic title, a plausible author, and a page number for a paper that doesn't exist. Ask for a specific clause in a standard and it may invent one, formatted perfectly, section number and all. The answer looks the same whether it's solid or fabricated, because confidence is not a signal the model can vary. It sounds equally sure of everything.

Give it the source instead of trusting its memory. Paste in the document, upload the standard, hand it the data. A model working from material in front of it is far more reliable than one dredging up a remembered pattern, for the same reason a person reading off the page beats one reciting from memory.

Tell it that "I don't know" is allowed. Explicitly giving the model permission to admit uncertainty cuts down the bluffing, because you're removing the pressure to fill every gap.

Make it show its work. Ask for the reasoning or the citation, so you have something to check rather than a bare claim to trust.

And verify anything load-bearing. Names, numbers, quotes, sources, anything with a safety or legal edge. Use the model freely for the work where plausible is useful, like drafting, restructuring, and thinking out loud. Verify it for the work where you need ground truth.

Got a second? Give some feedback on today’s article so we can keep making improvements to The Manifold.

PS — Growing up, I was told Wikipedia is an untrustworthy source of information and everything should be double-checked. Now, AI seems to be in the same place.