The position, stated up front
The default order is retrieval first, then prompt work, then fine-tuning for form and unit cost once you have an eval set. Most teams reverse it, because training sounds like the more serious engineering.
What each one actually changes
The consequence that decides most cases: facts learned in weights cannot be cited, cannot be permissioned and cannot be edited. They are blended into a distribution rather than stored as records. To correct one fact you retrain, and you still cannot show the reader where the answer came from. For anything a customer, an auditor or a regulator might question, that alone settles it.
Why retrieval degrades in month six
This is the failure nobody warns you about, and it explains why a system that launched well quietly stops being trusted. Four mechanisms, all of which compound.
The fixes are all boring, which is why they are skipped. Treat recency and authority as ranking signals rather than metadata. Deprecate superseded documents formally, so the index knows they lost. Run hybrid search with a re-ranking stage instead of trusting one similarity score. Chunk per document type. And keep a scored retrieval set that you rerun on a schedule, so decay shows up as a number falling rather than as a complaint from a customer. That maintenance discipline is most of what RAG implementation means once the first version ships.
None of these are fine-tuning problems. Training a model on a corpus with two contradictory procedures in it produces a model that is confidently wrong with no citation to check.
When to fine-tune an LLM
Four cases where it earns its cost.
What all four have in common: they are about how, not what. Ask which of the two your failure is, and the choice usually makes itself.
RAG vs fine tuning: the seven questions
Work down the list. Stop at the first one that fires.
1. Would a competent person with the right document in front of them have got this right? If yes, your problem is knowledge access. Fix retrieval. Training is the wrong tool and you will spend a quarter learning that. 2. Do you have an eval set? If not, neither option is available to you yet, because you will not be able to tell whether anything you did helped. Build a scored set from real cases first. This step is skipped more than any other and it is why so much AI work is argued about instead of measured. 3. Does the knowledge change more often than you are willing to retrain? Prices, policies, inventory, personnel and anything with a date. Retrieval, without further discussion. 4. Must the answer be cited, permissioned or auditable? Retrieval, and it is not a preference. Weights have no provenance and no access control. 5. Is the failure about shape rather than substance? Format, length, tone, schema adherence, consistency. Try prompt work first, because it costs an afternoon. If prompts get you close but not reliable, or the instruction block is now large enough to be a cost line of its own, fine-tune. 6. Are you paying for a large model to do a narrow, high-volume job? Distill to a smaller one and verify on the eval set. Keep the large model as the escalation path for the hard cases. 7. Is it both? Usually. Retrieval supplies the facts, a tuned model supplies the shape, and they compose cleanly because they operate at different layers.
The costs of fine-tuning that get left off the slide
Five, in rough order of how often they surprise people.
A retrieval system’s costs are real too, and they are ongoing rather than upfront: index maintenance, the scored retrieval set, permission synchronization, and the deprecation discipline described above. The honest comparison is not build cost. It is the cost of keeping each one correct through a year of change.
- Dataset construction - Several hundred to several thousand examples that are actually correct. Somebody with domain knowledge produces or reviews them. This is the real project and it is mostly human time.
- Evaluation - You need the eval set before you train, not after, or you cannot demonstrate the gain.
- Retraining on base-model change - Providers deprecate and upgrade. A tuned model is pinned to a base version, so their roadmap becomes your maintenance schedule.
- Serving - A custom model has its own deployment, its own scaling behavior and often a different price structure from the shared endpoint you were using.
- Reduced flexibility - A tuned model is committed to one behavior. Changing what the system does means training again, where a retrieval system means editing a document.
The third option nobody puts on the slide
Both sides of this debate skip the cheapest experiment, which is fixing what the model is given.
Context design is its own discipline and it accounts for a large share of the quality problems attributed to model choice. What gets retrieved, in what order, with what said about its source and its date. What is summarized rather than passed whole. What is deliberately left out because it competes with the passage that matters. How much of the budget goes to instructions versus evidence.
An example that recurs on almost every project. A system retrieves ten passages and presents them as an undifferentiated block, so the model has no way to tell the current policy from a two-year-old draft. Label each passage with its source, its effective date and its authority, and instruct the model to prefer the current one, and a class of confident wrong answers disappears. No training, no new index, an afternoon of work.
Prompt work sits in the same bucket. It is not durable, it changes with every model version, and it is still the correct first experiment because it costs hours rather than weeks. The rule we apply: try it, measure it against the eval set, and move to fine-tuning only when the prompt gets you close but not reliable, or when the instruction block has grown large enough to appear in your cost model.
Hybrid retrieval belongs here too. Semantic search alone fails on exact identifiers, part numbers, error codes and names, because embeddings are built to blur precisely those distinctions. Running keyword search alongside it and re-ranking the combined set fixes a category of failure that people routinely attribute to the model being weak.
Where the model boundary sits in an agent
For an agent, this decision shows up in a specific place. Retrieval determines what the agent knows about your business at the moment it acts, and fine-tuning determines how consistently it produces the structured output the next step depends on.
That second point is easy to miss. In an agent, a malformed output is not a cosmetic problem. It breaks the handoff to the next stage. A model that produces the right decision in the wrong shape one time in thirty is a system that fails one time in thirty, which is a strong argument for tuning form once the volume justifies it. That work is fine-tuning and evaluation, and we do not sell one without the other.
The honest limits
The short version
- Retrieval changes what the model knows. Fine-tuning changes how it behaves. Match the tool to the failure.
- Facts in weights cannot be cited, permissioned or edited. That settles most regulated and customer-facing cases.
- Retrieval decays in month six through near-duplicates, superseded content, new document types and drifting questions. All four are fixable and all four need maintenance.
- Fine-tune for form, for behavior you cannot write down, and to distill a cheaper model for a narrow job.
- Without an eval set, neither option is open to you yet.
- The most common expensive mistake is training a model to learn facts that should have been retrieved.
Keep reading
- Your AI pilot worked. Here’s why production won’t. - The gaps that appear between a good demonstration and a system under real load.
- What is an AI agent? (And why it isn’t a chatbot) - Where these choices land inside a working agent.
- Questions, answered - Cost, safety, timelines and what stays human.
