The question \u201cshould we fine-tune?\u201d comes up often and the answer is usually \u201cnot yet\u201d.
What each one solves
Prompting. You tell the model what to do and how. Solves: output format, tone, structure, the steps to follow. Cost: minutes. Change: instant.
Retrieval (RAG). You give the model the information it needs. Solves: answers grounded in your documents, information that changes, the need for citations. Cost: days to weeks. Change: update the documents.
Fine-tuning. You adjust the model's parameters on your examples. Solves: a very specific style hard to describe in words, a narrow task repeated at high volume, cost reduction by using a smaller model. Cost: weeks, plus labelled data. Change: retrain.
The most expensive confusion
Fine-tuning does not reliably add new knowledge. It teaches the model how to answer, not what to answer.
If you want the model to know your organisation's internal procedures, fine-tuning is not the solution — RAG is. A model fine-tuned on your documents will pick up their style and carry on inventing details, only in a more familiar tone.
The recommended order
- Try prompting first. Many problems considered \u201ctoo complex\u201d are solved by a well-written instruction and a few in-context examples.
- Add retrieval if information is missing. If the model errs because it does not have the data, give it the data.
- Consider fine-tuning only if: you have a few hundred to a few thousand quality examples, the desired behaviour is hard to describe in words, the volume justifies the effort, and the task is stable over time.
What is easy to overlook
A fine-tuned model becomes your responsibility: it has to be hosted or managed, versioned, re-evaluated and retrained when requirements change. A model reached through prompting and retrieval is updated by changing a text or a document.
For a small team, the difference in operational effort often matters more than the difference in performance.
The short rule
If the problem is how it answers → prompting. If the problem is what it knows → retrieval. If the problem is cost or latency at high volume, on a narrow and stable task → fine-tuning.