Retrieval (RAG)

A model knows what it learned in training and nothing about your private or current data — your codebase, your internal docs, what changed today. Retrieval-Augmented Generation (RAG) closes that gap: before the model answers, you fetch the documents relevant to the question and put them in its context, so it works from your actual material rather than from memory. As a pattern it is two things together — a “search the knowledge” tool , and the habit of retrieving before answering.

It is worth doing for two reasons. First, it supplies knowledge the model does not have (the current, the private, the specific). Second, it grounds the answer in sources you can point to, which reduces the confident guessing a model does when it is working from memory alone.

How it works

1. Index your documents. Split them into chunks and store them so they can be searched — commonly by embedding each chunk into a vector store, so “find the relevant ones” means “find the nearest ones to the question.”
2. Retrieve at query time. Given the question, pull the handful of chunks most relevant to it.
3. Put them in the prompt and answer. The model responds from the retrieved text, ideally citing which chunk each claim came from.

Underneath, this is the other patterns combined: retrieval is a tool call, and the document store is long-term memory the agent reads on demand.

Getting it right

• Retrieval quality is the whole game. If the right chunk is not retrieved, the answer cannot use it — no model fixes a bad fetch. How you chunk the documents, phrase the query, and build the index matters more than which model writes the final answer.
• Show the sources. Have the model cite the chunks it used. A grounded answer you can check beats a fluent one you cannot, and citations are how you check.
• Retrieve enough, not everything. Flooding the context with marginally related chunks crowds out reasoning and costs tokens — the same restraint as memory : relevant, not all.

What RAG does not fix

RAG reduces wrong answers; it does not remove them. The model can still misread or ignore a retrieved passage, so retrieval makes an answer checkable, not automatically correct — which is again why you show the sources. And the answer is only ever as fresh and accurate as the index behind it: a stale index gives stale answers with full confidence.

Where to go next

• Retrieval is memory read on demand: Memory .
• Tell whether the answers are any good: Evaluation .