Almost every misunderstanding about large language models starts from one silent assumption: that the system looks up an answer somewhere and then phrases it. That is not what happens.
Prediction, not retrieval
A language model is trained on a single task: given some text, predict what comes next. Repeated across an enormous quantity of text, that task produces a system which — in order to predict well — has to capture regularities in language: grammar, but also patterns of argument, conventions of genre, relationships between concepts.
When you ask it a question, the model does not open a database. It continues the text in a statistically plausible way, drawing on what it saw in training. The correct answer and the wrong one are produced by exactly the same mechanism.
Why the distinction matters
Three practical consequences follow directly.
There is no \u201csource\u201d for the answer. If you ask the model how it knows something, it will generate a plausible explanation — which may or may not have anything to do with what actually happened. The justification is generated just like the answer.
Displayed confidence does not measure correctness. A model phrases a verified statement and an invented one with equal fluency. Fluency is what it was optimised to produce; truth was not.
Knowledge is frozen at training time. Without an explicit mechanism for reaching external data, the model cannot know anything about what happened afterwards. And when it does not know, it usually does not stay silent — it generates something.
What this does not mean
It does not mean these systems are useless or that they \u201cjust imitate\u201d. Text prediction at scale produces real capability: summarising, rephrasing, extracting information from a given text, translating, generating code. In those tasks the working material is the text the model is handed — that is its home ground.
Problems begin when we treat the model as a source of truth about the world rather than a processor of text.
The practical rule
Split tasks into two categories.
- Transform text you give it. Summarise, rephrase, extract, classify, translate. Here the model is strong, and checking is cheap: you have the original in front of you.
- Produce information you do not have. Ask for a figure, a reference, a date, a legal provision. Here the model is unreliable, and checking is mandatory.
The difference between an organisation that gets value from these systems and one that ends up with embarrassing incidents usually runs exactly along that line.