The idea is simple: instead of hoping the model \u201cknows\u201d something, you give it the information in context and ask it to answer only from that.
How it works
The flow has four steps.
- Preparation. Documents are split into fragments, and each fragment gets a numerical representation capturing its meaning.
- Retrieval. For a question, the fragments closest in meaning are found.
- Building the context. The retrieved fragments go into the request to the model, alongside the question.
- Generation. The model composes an answer from the fragments it received.
Where it fails in practice
Almost every problem in a RAG system is a retrieval problem, not a generation problem. If the right fragment does not reach the context, no model can produce the right answer.
Bad chunking. Cutting at a fixed character count breaks tables, separates a definition from its example, and leaves fragments that make no sense on their own. Chunking along the document's real structure — sections, subsections — works far better.
Questions phrased differently from the document. A user asks \u201chow much holiday do I get\u201d; the document says \u201cduration of annual leave\u201d. Semantic search helps, but does not solve everything; combining it with keyword search improves results noticeably.
Questions that require aggregation. \u201cHow many contracts did we sign last year\u201d is not answered by retrieving a few fragments. Those are questions for a database, not for RAG.
Contradictory documents. When three versions of the same procedure exist, retrieval brings all of them, and the model picks one — usually without flagging the conflict.
Ignored metadata. Date, department, version, access level are often more important than semantic similarity. A correct answer from an expired procedure is still a wrong answer.
Worth building from the start
- Mandatory citations. Every claim in the answer must point to its source fragment. Without that you can verify nothing.
- Metadata filtering before search. Narrow to valid documents and to those the user is allowed to see. Access control must be applied at retrieval, not at display.
- A set of test questions. Real questions with known correct answers, used as a regression test on every change.
- An honest \u201cnot found\u201d. \u201cI could not find this in the available documents\u201d is a valid result, and far more useful than an approximation.
When RAG is not the answer
If the information is structured, a database query is faster, cheaper and exact.
If the questions require calculation or aggregation, you need a calculation tool, not retrieval.
If the documents are few and fit in the model's context, hand them over directly.
Where the effort goes
Effort goes, in order, into: the quality and cleanliness of the documents, the chunking strategy, retrieval quality, and only last the choice of model. Reversing that order is the most common cause of a disappointing RAG project.