Why Retrieval
By the end of this lesson
Explain why supplying context beats expecting a model to know your data.
Somebody asks your assistant how much notice they need before booking a flight. The answer is in paragraph four of a handbook that lives on your intranet. The model has never seen that handbook, and no amount of asking will change that.
This lesson is about the gap between those two facts and the three ways teams try to close it. Only one of them holds up, and the wrong choice is expensive enough to be worth a lesson of its own.
What a model knows, and what it cannot know
A hosted model's knowledge came from its training data. Your documents were not in it, and could not have been, for reasons that do not go away:
- They are private. They sit behind your authentication, which is the whole point of them.
- They change. A policy revised last Tuesday cannot be inside a model whose training finished months earlier.
- They are specific to you. Your grade bands, your approval thresholds, your client names. There is no general version of them to learn.
- Even public material has a cut-off. Anything published after training stopped is absent, and the model has no way to notice the absence.
That last point is the sharp one. A model asked about a document it has never seen does not return an error or say it lacks the material. It produces likely-looking text, because producing likely-looking text is the only operation it has. A plausible clause number and a confident tone come out the same way a correct answer would.
Three ways to get your facts into an answer, and what each one actually does:
- Paste the whole corpus into the prompt
- Honest and simple, and fine for one short document. Beyond that it costs input tokens on every question, adds latency before the first word appears, and dilutes the one relevant clause among hundreds of irrelevant ones. The tokens lesson covered why a bigger context window does not rescue this.
- Fine-tune a model on your documents
- Continue training on your own examples so the resulting model behaves differently. This is a real technique with real uses. Teaching it facts is not one of them, for reasons in the next section.
- Retrieve the relevant passages and supply them
- Search your own store for the few passages that bear on the question, put those in the prompt, and instruct the model to answer from them and nothing else. The facts arrive at query time, from a source you control and can point at.
Fine-tuning and retrieval get confused constantly because both are described as "training the AI on our data". They are not alternatives to each other; they solve different problems:
| Fine-tuning | Retrieval | |
|---|---|---|
| What it changes | The model's internal numbers | The prompt sent for one question |
| What it is genuinely good at | Style, tone, output format, following a house convention without being reminded | Supplying specific facts the model has never seen |
| Adding a revised document | Prepare examples, run training again, evaluate, redeploy | Re-index that document. Minutes, and no model change |
| Correcting one wrong fact | No targeted mechanism. You cannot reach in and edit a statement | Edit the source document and re-index it |
| Showing where the answer came from | Nothing to show. The fact is spread across weights | The passages you supplied, with links a reader can open |
| Cost shape | A training run per update, plus a per-token rate to run the result | An embedding call per document and per question, plus normal inference |
| How it fails | Fluent answers in your house style that state facts you never taught it | The right passage is not retrieved, and a well-built pipeline says so |
def answer_from_documents(question: str, caller: User) -> dict:
passages = knowledge_store.search(question, caller=caller, top_k=5)
if not passages:
return {"answer": "Not covered by the supplied documents.", "sources": []}
context = "\n\n".join(
f"[{n}] {p.document_title} / {p.heading_path}\n{p.text}"
for n, p in enumerate(passages, start=1)
)
answer = chat(
system=(
"Answer only from the CONTEXT block. Cite the bracketed number of "
"every passage you relied on. If the CONTEXT does not answer the "
"question, reply exactly: Not covered by the supplied documents."
),
user=f"CONTEXT\n{context}\n\nQUESTION\n{question}",
)
return {
"answer": answer,
"sources": [{"n": n, "url": p.source_url} for n, p in enumerate(passages, start=1)],
}- The search happens in your code, against your store, before the model is involved. Whatever the model says next, the material it was given is a list of rows you can print out and read.
- caller is passed into the search so a passage the person is not allowed to read is never retrieved in the first place. Filtering after retrieval is too late — by then the text is in a prompt you are about to send.
- The empty-result branch is not a nicety. Without it, no passages means an empty context block, and an empty context block plus a question is an invitation to answer from the weights.
- Each passage is numbered, and the numbers are the citation handles the model is asked to use. Numbering the context is what turns "cite your sources" from a hope into something you can check mechanically.
- The instruction to answer only from the block, and the exact wording to use when it cannot, do a lot of work here. They are covered properly in the pipeline lesson, because retrieval without them is the failure mode that catches most teams.
- The returned sources travel with the answer, so the interface can show links. A reader who can check the claim is worth more than a model that sounds certain.
Summary
- Your documents were never in the training data, and a model asked about them produces plausible text rather than an error
- Fine-tuning shapes style and format; it has no mechanism for storing a fact you can rely on or point at
- Retrieval supplies facts at query time from a store you control, so answers stay current and can be cited
- A retrieval failure is debuggable — you can see which passages were supplied; a wrong fine-tuned answer offers nothing to inspect
- Retrieval moves quality onto your search, and grounding reduces ungrounded answers without eliminating them
Practice
Attempt each one before opening the solution. Getting it wrong first is how the idea sticks.
Think about it
Fine-tune, retrieve, or neither
Four requests land on your desk in the same week. Every rejection email should follow the finance team's fixed wording and structure. The assistant should answer questions about a policy that changes roughly monthly. It should also answer questions about a fifteen-page onboarding guide that has not changed in two years. And it should tell an employee their remaining leave balance.
For each, say which approach you would use and why. One of them is a trap.
Show solution
The fixed rejection wording is a format problem, so it is the one case here where fine-tuning is a candidate. Try a careful prompt with two or three examples first — that often gets you there, and it costs nothing to maintain. Reach for fine-tuning if prompting plateaus and the volume justifies it.
The monthly-changing policy is retrieval, and it is the clearest case. Anything that changes on a schedule you do not control should never be baked into weights.
The stable fifteen-page guide is the trap. It is tempting to say fine-tuning is fine because the document never changes, but stability was never the objection — fine-tuning does not reliably store facts whether they change or not. Fifteen pages is also small enough that you could send the whole thing, which is worth pricing before you build anything. Retrieval is still the better default, because the corpus will grow.
The leave balance is neither. It is a database row, and the answer must be exactly right for one named person at this moment. That is tool calling, and the previous module covered it. Notice how easily it hides inside a list of "questions the assistant should answer" — part of designing one of these is spotting which questions are documents and which are data.
Think about it
The answer that was never in a document
Your assistant tells an employee that claims must be submitted within 45 days. Your policy says 30. The 45-day figure appears in no document you hold.
Work out where it came from, then say what you would change so this class of failure becomes visible rather than silent.
Show solution
Most likely the retrieval step returned nothing useful — or nothing at all — and the model answered from its weights. Submission windows in the tens of days are common in expenses policies generally, so 45 is a high-probability number in a sentence of that shape. Nothing malfunctioned. It produced likely text, which is the only thing it does.
The first change is the one from the code sample: when no passage clears your relevance bar, return a fixed message and do not call the model at all. An empty context block plus a confident instruction is the worst combination in this whole design.
The second is citations, surfaced in the interface. An answer with no source link attached is an answer nobody can check, and "45 days, source: nothing" is a failure a user spots immediately.
The third is logging what was retrieved for every question, with scores. Without it you cannot tell whether search missed the passage or the model ignored it, and those two need different fixes.
Worth being blunt about the remaining gap: all three make the failure visible. None of them make it impossible. Grounded generation reduces ungrounded answers substantially and does not eliminate them, which is why anything consequential keeps a human reading the source.
Knowledge check
Nothing is recorded and there is no score. The explanation appears either way.
Saved in this browser only.