"Should we use RAG or fine-tuning?" is usually asked before anyone has stated what the system is getting wrong. That ordering is the problem. The two techniques repair different faults, and picking one without naming the fault produces an expensive project that leaves the original complaint intact.
The distinction is narrow enough to state in one line. Retrieval-Augmented Generation changes what the model can see when it answers. Fine-tuning changes how the model behaves when it answers. A model that does not know your refund policy has a visibility problem. A model that knows the policy but replies in three paragraphs when you need a JSON object has a behaviour problem. Applying the wrong remedy to either one wastes months.
This article covers how to identify which fault you actually have, what each approach costs to build and to keep running, when both are justified, and when the answer is neither.
What Each Technique Actually Changes
RAG leaves the model untouched. At answer time the system searches a corpus, selects passages, and places them in the prompt as evidence. The model reasons over text it was handed rather than text it memorized. Update a document and the next answer reflects it, because nothing about the model encoded the old version.
Fine-tuning changes the weights. You supply examples of the input-output behaviour you want, and training shifts the model toward that pattern. The result is a model that responds in a particular style, follows a specific format, applies a house convention, or handles a task type more reliably - without those instructions occupying prompt space on every call.
The consequence that matters operationally: RAG makes knowledge current, fine-tuning makes behaviour consistent. Facts belong in retrieval because facts change. Conventions belong in weights because conventions are stable.
A common misreading is that fine-tuning is a way to "teach the model your data." It is not a reliable way to store facts. Trained-in facts cannot be updated without retraining, cannot be cited, cannot be permission-filtered per user, and are recalled approximately rather than exactly. If someone must be able to see the source sentence behind an answer, that answer has to come from retrieval.
Diagnose the Fault Before Choosing the Fix
Collect thirty real failures - actual questions from actual users, with the answer the system gave and the answer it should have given. Then sort them.
Knowledge failures. The answer is wrong or absent because the necessary information never reached the model. Somebody can point at a document containing the correct answer. This is a retrieval problem.
Behaviour failures. The correct information was present in the context and the model still produced the wrong shape: too long, wrong tone, wrong schema, ignored an instruction, inconsistent between similar cases. This is a behaviour problem.
Reasoning failures. The evidence was present and correctly formatted, but the model combined it incorrectly - a total that does not add up, a condition applied in the wrong order. Neither technique reliably fixes this; the answer is usually deterministic code for the part that must be exact.
Specification failures. Two reviewers disagree about what the correct answer even is. No technique fixes this. The requirement has to be decided first.
The proportions tell you where to spend. In enterprise deployments the first category usually dominates, which is why RAG is the more common starting point - not because it is fashionable, but because most complaints are about the system not knowing something.
When RAG Is the Right Answer
Choose retrieval when any of the following is true, and note that most enterprise cases hit several at once.
The information changes. Prices, policies, product specifications, contract terms, and inventory move on their own schedule. Retraining a model each time a document is edited is not a workable operating model.
Answers must be traceable. Regulated or high-stakes work requires showing which passage supports a claim. Retrieval produces that link naturally; weights do not.
Access differs by user. Two employees asking the same question may be entitled to different answers. Retrieval can filter by permission before the model ever sees a passage. A fine-tuned model has no per-user view of what it learned.
The corpus is large or long-tailed. Thousands of documents, most of them rarely needed, are cheap to index and expensive to train into a model.
Content must be removable. If a document is withdrawn or a customer exercises a deletion right, removing it from an index is immediate. Removing it from weights is not.
When Fine-Tuning Is the Right Answer
Fine-tuning earns its cost when the requirement is about form or behaviour rather than facts.
You need a strict output format at high volume, and prompt instructions plus schema validation still produce drift on edge cases. Training removes the instruction overhead from every request and reduces variance.
You need a consistent voice or house convention that is tedious to specify - a particular way of structuring a clinical note, a legal summary style, a support tone your team has agreed on. These are easier to demonstrate with examples than to describe in rules.
You have a narrow, repetitive classification or extraction task where a smaller fine-tuned model matches a large general model's quality at a fraction of the cost and latency. This is often the strongest economic case: not better quality, but the same quality much cheaper.
Your prompts have grown to thousands of tokens of instructions and examples, and that overhead is now the dominant cost per call.
The precondition is data. You need examples that are consistent with each other. A few hundred well-curated pairs typically outperform tens of thousands of inconsistent ones, because training amplifies whatever pattern is in the data - including a contradictory one.
When Both Are Justified
The combination is legitimate when a system needs current facts and disciplined output.
A claims assistant is a clear case: retrieval supplies the policy wording, the customer's contract, and the incident record, while a fine-tuned model produces the assessment in the exact structure the downstream system expects, in the tone the regulator has accepted.
Sequence matters. Build retrieval first and measure. Fine-tune afterwards, on the behaviour that retrieval could not fix. Doing both at once makes it impossible to attribute an improvement or a regression, and you will end up retraining to compensate for a retrieval bug.
When the Answer Is Neither
Several problems presented as "RAG or fine-tuning" are neither.
If the answer lives in one structured database and can be produced by a query, write the query. Wrapping a deterministic lookup in a language model adds cost, latency, and a failure mode that did not previously exist.
If the failure only occurs on a handful of prompts, try prompt work first. It is hours rather than weeks, and it is reversible.
If the exact part of the answer must be exact - a total, a date calculation, an eligibility rule - move it out of the model entirely. Compute it in code and let the model present the result.
If the source documents are outdated, contradictory, or unlabelled, no technique compensates. Retrieval will faithfully return the wrong document. This is the most common hidden blocker in enterprise projects, and the honest response is to fix the corpus before building anything on top of it.
Cost, Latency, and Operational Weight
The comparison people expect is training cost. The comparison that decides projects is what each approach demands after launch.
RAG's build cost sits in ingestion, chunking, embedding, index infrastructure, permission handling, and evaluation. Its per-query cost is higher, because retrieved context enters the prompt on every call and adds both tokens and retrieval latency. Its ongoing work is keeping the index synchronized with the source systems and watching retrieval quality - a pipeline that runs forever.
Fine-tuning's build cost is dataset preparation, which is usually the largest line and is mostly human effort, plus a comparatively small training run. Its per-query cost can be lower, since prompts shrink and a smaller model may suffice. Its ongoing work is episodic but heavier when it arrives: each behaviour change means a new dataset, a new training run, evaluation against the previous version, and a deployment with a rollback path. You also inherit a dependency on a base model version that the provider may deprecate.
Neither is cheap. RAG spreads its cost thinly and continuously; fine-tuning concentrates it into periodic events. Teams without a data pipeline discipline usually find RAG's ongoing demand easier to absorb than fine-tuning's periodic spikes.
Deciding in Practice
A sequence that avoids most wasted effort:
Establish a baseline. Build an evaluation set from the real failures you collected, and measure the plain model with a careful prompt. You cannot claim an improvement without this number, and occasionally the baseline is already sufficient.
Add retrieval if knowledge failures dominate, and measure retrieval separately from generation so you know which layer moved.
Exhaust prompt and schema work on whatever remains, since it is the cheapest lever and the fastest to undo.
Fine-tune only for behaviour that survives all of the above, and only when you can produce a consistent dataset for it. Evaluate against the same set, and keep the previous model deployable.
The failure mode this ordering prevents is the common one: a team fine-tunes a model on documents to "teach it the business," discovers the answers are unciteable and already stale, and rebuilds with retrieval six months later. The technique was not wrong. The diagnosis was skipped.