Fine-Tuning vs RAG vs Prompting: A Decision Matrix

When a customer says they want to use AI, there is a decision they need to make before anyone touches a model. It is not which model. It is which method.

In production, three methods cover almost everything: prompting, retrieval-augmented generation (RAG), and fine-tuning. Most teams reach for the most expensive one by default. Here is how we choose.

The three methods, briefly

Prompting: you write a careful instruction, possibly with a few examples, and send it to a base model. No external data, no training, no infrastructure beyond an API call.

RAG: you retrieve relevant documents at query time, hand them to the model alongside the user question, and have the model reason over them. The model does not learn. Your knowledge base updates instantly.

Fine-tuning: you train a model on a curated dataset of your domain, baking the knowledge and the response style into the weights themselves. Cheap at inference, expensive up front, requires real engineering discipline.

The wrong default

When teams hear that production AI is hard, they assume the answer is to fine-tune everything. It is the most technically impressive move, so it feels like the right one.

It usually is not. Fine-tuning is the most expensive, slowest, and least reversible of the three. It is also the right move surprisingly often, but only after you have ruled out the others.

The decision matrix

We use a rough matrix when deciding. Two axes:

• How often does the underlying knowledge change?
• How specialised is the response style or domain?

If the knowledge changes constantly and the style is generic, prompting plus RAG. If the knowledge is stable but the style or domain is highly specialised, fine-tuning. If both, hybrid: fine-tune the style, RAG the facts.

Where prompting wins

You should always try prompting first. It is free of infrastructure, instant to iterate, and surprisingly capable. We have shipped customer support, summarisation, classification, and lightweight extraction with a careful prompt and zero further engineering.

Prompting wins when:

• The task is well within frontier-model capability
• The knowledge is general or fits comfortably in the prompt
• Cost per call is acceptable
• Latency is not the bottleneck

If a well-crafted prompt does the job, you do not need RAG or fine-tuning. We have seen teams burn months on fine-tuning what should have been a 200-word prompt.

Where RAG wins

RAG is the right move when the knowledge that matters is yours, large, and changes regularly. Customer documents, support tickets, regulatory updates, internal wikis, product catalogues: anywhere the corpus is bigger than a prompt and more dynamic than a training run.

RAG wins when:

• The corpus is too large for a context window
• The corpus updates on a cadence faster than fine-tuning cycles
• You need citations and source attribution
• You can afford slightly higher per-call latency

Most customer-facing knowledge systems we ship are RAG-first. The model does not need to know the corpus, it needs to reason over it accurately.

Where fine-tuning wins

Fine-tuning is the right move when the task itself is unusual. The response style is specific, the domain is technical, the format constraints are tight, or you need to compress a large complex system down into a faster smaller model.

Fine-tuning wins when:

• Style or format matters as much as content
• The domain has its own vocabulary that base models butcher
• Latency or cost demand a smaller deployed model
• The task is high-volume and stable

The Teyseer Motors deployment is fine-tuned because the customer experience needs to sound like Teyseer, every time, across every brand they distribute. A fresh prompt or a RAG layer cannot guarantee that.

Hybrid is usually the answer

In practice, most production systems blend two or three of these methods. We will fine-tune the style, RAG the live knowledge, and prompt the orchestration logic. The methods compose. The method matrix is a starting question, not the final architecture.

“Fine-tuning is the most expensive, slowest, and least reversible of the three. It is also the right move surprisingly often, but only after you have ruled out the others.”