The term is unfortunate — it suggests a rare, pathological error. In reality, the mechanism that produces a correct answer and the one that produces an invented claim are identical.
Why they happen
A model generates the most plausible text in context. When the context calls for information the model does not have well represented, the most plausible text is still one that looks like an answer: a citation with author, year and title, an article number, a figure to two decimal places.
The form has been learned correctly. The content is filled in with whatever fits statistically.
Several factors raise the risk:
- questions about rare, recent or highly specific facts;
- requests for references, quotations, exact figures;
- questions phrased as if the answer certainly exists;
- long conversations, where the relevant context has slipped out of the model's window.
What works
Give it the sources. If the model receives the relevant document in context and is asked to answer only from it, the invention rate drops substantially. That is essentially what a RAG architecture does. It is not a guarantee — the model can still misread — but it shifts the task from \u201crecall\u201d to \u201cread\u201d, where it performs far better.
Require verifiable citations. Ask the model to point to the passage it relies on. A quotation that does not appear in the document is an automatic red flag and can be checked programmatically.
Leave it an exit. A model explicitly told that \u201cI do not know\u201d is an acceptable answer will use it more often than one pressed to produce something.
Check what can be checked. File names, URLs, identifiers, formulas — all can be validated automatically before reaching a human.
What does not work
Asking the model to assess itself does not work. \u201cAre you sure?\u201d produces a rephrasing, not a verification. The same limitation applies to \u201cgive me a confidence score\u201d: the score is generated, not measured.
Nor does assuming a newer model solves the problem. Larger models invent less often, but invent more convincingly — which, for an organisation, can be worse.
What follows for architecture
If the output reaches an end user without checking, the system must be designed so that an invented claim does no damage: show the sources next to the answer, mark generated content visibly, keep an obvious way to report an error.
If the output triggers an action — a payment, a rejection, an official communication — there must be a check between generation and action. Not because the model is often wrong, but because when it is wrong, it gives no sign.