RAG vs fine-tuning vs prompting: which one does your problem need?

The five approaches, cheapest first

Each step up the list costs more to build, more to run, and more to change later. Move to the next one only when the previous one has demonstrably failed on your test set.

1. Prompting: try this first, almost always

Modern models can already do most business tasks: summarise, classify, extract, draft. When they fall short, the cause is usually the instruction, not the model. Clear task framing, a few worked examples, an explicit output format, and the right context in the window solve a surprising share of problems that arrive labelled "we need a custom model".

Prompting is also the cheapest thing to change. A better prompt ships in minutes, and you can test it against the same examples as everything else.

2. Retrieval (RAG): when the model doesn't know your information

If the model needs to know your refund policy, your product catalogue, or last quarter's contracts, that's retrieval. At the moment it answers, the system searches your documents, puts the relevant passages in front of the model, and the model answers from them, ideally citing which passage it used.

Updating a RAG system means updating a document. That's why it's the default answer for anything involving your own knowledge: the information stays current without retraining anything. It's what sits behind a support chatbot that answers from your docs.

3. Tool use: when the answer lives in a live system

Some answers aren't in any document. "Where is order 4471?" lives in your order system; "book me a slot on Thursday" needs a calendar. Tool use lets the model call your systems, read the result, and act on it. This is the step where a chatbot becomes an AI agent, and where guardrails and permissions start to matter as much as the model.

4. Fine-tuning: a behaviour, not a body of facts

Fine-tuning does not teach a model your facts. It teaches a behaviour, a style, or a narrow task. Fine-tuning your refund policy into a model makes it harder to update, not easier, and the model can still state it wrongly. This single misunderstanding is behind most wasted fine-tuning budgets.

Fine-tuning is often the cost answer, not the capability answer. The strongest real case for it is taking a task a large model already does well and moving it onto a smaller, cheaper, faster model tuned for exactly that task. That's an economics decision, and it only pays once you have volume.

5. Training from scratch: effectively never

For a business application, training a model from scratch solves nothing the first four can't, at a cost several orders of magnitude higher. If someone proposes it, ask which problem the previous four failed to solve, and how they measured the failure.

You need evaluation before you need any of them

Without a test set, nobody can tell whether a fine-tune helped, whether a retrieval change improved anything, or whether last month's model update broke something. Teams skip this and then argue about whether the system is getting worse based on anecdotes.

Before choosing an approach, collect 50 to 200 real examples with the answers you'd accept. Every option above is then judged against the same set, and the decision stops being a matter of opinion.

A quick way to decide

• The model can do the task but does it inconsistently: improve the prompt.
• The model doesn't know something it needs: add retrieval.
• The answer depends on live data or needs an action: add tool use.
• A proven task is too slow or costly at your volume: consider fine-tuning a smaller model.
• Someone suggests training a model: ask what the other four failed to do.