Why RAG, and the canonical pipeline

Three gaps in a frozen model

A language model's weights encode what it saw during pretraining (8.1 The LLM lifecycle and model landscape) — its parametric knowledge. That knowledge has three structural gaps. It is stale: anything after the training cutoff does not exist. It is public: your company's wiki, your lab's notes and yesterday's support tickets were never in the corpus. And it is ungrounded: when the model does not know, it still produces fluent text, the behaviour we call hallucination (8.12 Safety, risks and responsible AI). Fine-tuning (8.2 Fine-tuning and PEFT) can inject some facts, but it is slow, expensive to update, hard to attribute, and surprisingly bad at reliably storing specific facts.

Retrieval-augmented generation (RAG) attacks all three gaps with one move: at query time, fetch the most relevant pieces of text from an external store (non-parametric knowledge) and place them in the prompt so the model answers from evidence it can quote. Lewis et al. (2020) formalized this as marginalizing over retrieved passages $z$:

with a trainable retriever $p_\eta$ (a dense bi-encoder, 9.2 Embeddings deep dive) and a generator $p_\theta$ (BART, 7.7 Encoder–decoder: T5, BART and choosing an architecture). Modern RAG keeps the shape but simplifies the math: a frozen LLM reads the concatenation of the top-$k$ chunks, and the weighting happens implicitly through attention (7.2 Scaled dot-product and multi-head attention, fully derived).

Intuition. Fine-tuning is studying for a closed-book exam; RAG is an open-book exam with a good index. The model still has to read and reason, but it no longer has to remember everything.

RAG vs fine-tuning vs long context

Rules of thumb: knowledge problems → RAG; behaviour problems → fine-tuning; a document set that fits comfortably in context and is queried a handful of times → just put it in the prompt, ideally with prompt caching (9.9 Production: latency, cost, caching, guardrails, security and privacy). Serious systems combine them: a fine-tuned small model for formatting, RAG for facts, and a long-context read of a few whole documents at the end (9.7 Advanced RAG patterns).

The canonical pipeline

RAG has an offline half that prepares knowledge and an online half that answers questions:

1. Ingest — collect sources (PDFs, HTML, tickets, code, databases) with their metadata and permissions.
2. Parse — convert to clean text or Markdown while preserving structure (headings, tables).
3. Chunk — split into retrieval units of a few hundred tokens (9.4 Chunking and ingestion).
4. Embed — map each chunk to a vector with an embedding model (9.2 Embeddings deep dive).
5. Index — store vectors (and a lexical index) for fast nearest-neighbour search (9.3 Vector search and indexes).
6. Retrieve — embed the query and fetch the top-$k$ candidates, usually from dense and sparse indexes (9.5 Retrieval quality: hybrid search, reranking, query transforms and metrics).
7. Rerank — re-score candidates with a cross-encoder and keep the best few (9.5 Retrieval quality: hybrid search, reranking, query transforms and metrics).
8. Assemble — build a prompt with instructions, numbered sources and the question inside a token budget (9.6 Generation side: context assembly, citations, structured outputs and tool calling).
9. Generate — the LLM answers, grounded in the sources, often as structured output (9.6 Generation side: context assembly, citations, structured outputs and tool calling).
10. Cite — map claims back to sources so users can verify.
11. Evaluate — measure retrieval and answer quality continuously (9.8 Evaluating and observing LLM apps).

Steps 1–5 run when documents change; 6–10 run per query; 11 runs always. Every step is a place where quality is won or lost, which is why most of them get their own lesson in this phase.

Where the pipeline breaks: the seven failure points

Barnett et al. (2024) studied three production RAG systems and catalogued seven recurring failures. They make an excellent debugging checklist:

Math check. Retrieval quality is a ceiling on answer quality. If the relevant chunk is in the top-$k$ for only 70% of queries (recall@$k$ = 0.7, 9.5 Retrieval quality: hybrid search, reranking, query transforms and metrics), no prompt can make the system right on more than roughly 70% of the questions that need that chunk. Measure retrieval on its own before touching the prompt.

What changed between 2020 and 2026

The original RAG was a research recipe: a jointly trained retriever and generator. Today "RAG" names a design pattern: frozen frontier models, off-the-shelf embedding models, hybrid retrieval with rerankers, LLM-written chunk context (9.4 Chunking and ingestion), structured outputs, and increasingly agentic retrieval, where the model decides when and what to search, iterates, and uses other tools such as SQL or web search (9.7 Advanced RAG patterns, 10.2 Tool use and function calling). Context windows of hundreds of thousands of tokens and prompt caching moved the boundary — more systems retrieve whole documents or sections instead of tiny fragments — but did not remove the need to choose which documents to read. Retrieval is also the backbone of agent memory (10.4 Memory and context engineering).

Where this shows up later

• Embeddings and vector search — 9.2 Embeddings deep dive, 9.3 Vector search and indexes; chunking — 9.4 Chunking and ingestion; retrieval quality — 9.5 Retrieval quality: hybrid search, reranking, query transforms and metrics.
• Grounded generation, citations and tool calling — 9.6 Generation side: context assembly, citations, structured outputs and tool calling; advanced patterns — 9.7 Advanced RAG patterns; evaluation and production — 9.8 Evaluating and observing LLM apps, 9.9 Production: latency, cost, caching, guardrails, security and privacy.
• Agents treat retrieval as one tool among many — 10.2 Tool use and function calling; memory is retrieval over an agent's own past — 10.4 Memory and context engineering.