Every AI project eventually hits the same fork: the base model is not doing what you need. Three roads lead out of it - better prompting, retrieval-augmented generation (RAG), and fine-tuning. Teams routinely reach for the most expensive one first. The right instinct is the opposite: start cheap, and only climb when the cheaper option provably cannot get there.
What Each One Actually Changes
Prompting changes the instructions. You are working entirely within the model's existing knowledge and abilities, just steering them. Zero training, instant iteration.
RAG changes the knowledge. You inject relevant facts into the prompt at query time so the model can answer questions about data it never saw in training. It does not teach the model new skills or behaviors - it hands it a reference sheet.
Fine-tuning changes the model. You continue training on your own examples so new behavior, format, or tone is baked into the weights themselves. It teaches skills and styles, but it is the poorest way to teach facts.
The Decision Framework
Is the problem that the model lacks KNOWLEDGE
(your docs, current data, private info)?
-> RAG. Fine-tuning is the wrong tool for facts;
facts change, and retraining every time is madness.
Is the problem that the model lacks a SKILL or won't follow
a specific FORMAT/STYLE consistently, even with good prompts?
-> Fine-tuning. You are teaching behavior, not facts.
Is the problem just that you haven't told it clearly enough
what you want?
-> Prompting. Try this first, every time. It's free and instant.The Test: Could a New Employee Do It?
A useful mental model. Imagine handing the task to a sharp new hire.
If they could do it well with a clear instruction sheet, you need better prompting. If they could do it well but only with access to your internal wiki and databases, you need RAG. If they would need weeks of training to internalize a specialized skill or a very particular house style, you need fine-tuning.
Why Fine-Tuning Is Usually Premature
Fine-tuning is seductive because it feels like "really" teaching the model. But it carries costs the other two avoid: you need a curated dataset of hundreds to thousands of high-quality examples, a training and evaluation loop, a place to host the custom model, and a re-training plan every time the base model improves or your requirements shift. Worse, if you fine-tune to inject facts, those facts go stale and there is no easy way to update a single one - you retrain.
In practice, a large majority of "we need to fine-tune" instincts are solved by a better prompt plus RAG, at a fraction of the cost and with far faster iteration.
When Fine-Tuning Genuinely Wins
It earns its cost when: you need a consistent output format or voice that prompting cannot reliably enforce; you have a narrow, high-volume task where a smaller fine-tuned model can replace a large expensive one and cut cost dramatically; you need to reduce latency or prompt length by moving instructions and examples into the weights; or you are teaching a genuinely specialized skill (a domain classification, a structured extraction) for which you have good labeled data.
The Hybrid Answer
These are not mutually exclusive, and mature systems combine them. A common production shape: a lightly fine-tuned model for consistent format and tone, wrapped in RAG for current facts, driven by a carefully engineered prompt for control. Each layer does the job it is best at - fine-tuning for behavior, RAG for knowledge, prompting for steering.
The discipline is to add each layer only when the cheaper ones demonstrably fall short. Start with a prompt. Add retrieval when it needs your data. Fine-tune only when you have proven a skill or format gap that neither can close - and when you have the data and the appetite to maintain a model of your own.