Advanced RAG patterns
Grading, hierarchies, graphs, agents and structured data
Retrieve-top-five-and-answer fails in predictable ways: it trusts irrelevant chunks, cannot see themes that span a corpus, cannot chain facts, and cannot read tables, databases or slides. Each failure has a named pattern — Self-RAG and corrective RAG, RAPTOR, GraphRAG, agentic and multi-hop retrieval, text-to-SQL, multimodal RAG — and each costs something. This lesson maps symptom to pattern so you reach for complexity only when a measurement says you need it, and it is your first real contact with agent loops (10.3 Reasoning and planning for agents).
You'll be able to
- Map RAG failure symptoms to the pattern that targets them
- Explain Self-RAG and corrective RAG as grade-then-generate loops and implement a lightweight version
- Describe RAPTOR and GraphRAG indexing and the global questions they answer
- Design an agentic, multi-hop retrieval loop with routing and budgets
- Handle structured and visual data with text-to-SQL, table tools and multimodal retrieval
Key ideas
- Diagnose before adding complexity: each naive-RAG symptom maps to a specific pattern with a specific cost.
- Self-RAG and corrective RAG grade retrieved chunks before generating and fall back (rewrite, search elsewhere) when too few pass — the adaptive-RAG loop.
- RAPTOR clusters and recursively summarizes chunks into a tree so thematic questions retrieve summaries while detail questions retrieve leaves.
Show 4 more key ideas
- GraphRAG extracts an entity–relation graph, detects communities and summarizes them; global questions become a map–reduce over community summaries, at high indexing cost.
- Agentic RAG wraps retrieval-as-a-tool in a plan–retrieve–read–decide loop with routing and budgets; multi-hop questions need chained, dependent queries.
- Text-to-SQL needs schema linking, a semantic layer, read-only execution and verification; pandas tools often beat SQL generation for small tables.
- Multimodal RAG uses captions, joint image–text embeddings or page-image retrieval; long-context models shift retrieval from chunks toward whole documents; agent memory is RAG over history.
Suggested path through this lesson
When naive RAG is not enough
"Retrieve top-5, stuff, answer" fails in predictable ways. Diagnose first, then reach for the pattern that targets the symptom:
| Symptom | Likely cause | Pattern |
|---|---|---|
| Confident answers built on irrelevant chunks | No relevance check before generation | Self-RAG / corrective RAG grading |
| Questions about themes, trends or "the whole corpus" | Chunk-level retrieval cannot see the forest | RAPTOR, GraphRAG |
| Questions that chain two or more facts | A single retrieval step | Decomposition, iterative or agentic retrieval |
| Answers need numbers from tables or databases | Text retrieval over structured data | Text-to-SQL, table tools |
| Slides, diagrams, scanned pages | Text-only index | Multimodal RAG |
| Follow-up questions in a conversation | Query lacks context | Rewriting, memory (10.4 Memory and context engineering) |
| Some queries need no retrieval at all | Always-retrieve pipeline | Routing / adaptive retrieval |
Self-RAG and corrective RAG
Self-RAG (Asai et al., 2023) trains the generator to emit reflection tokens: whether to retrieve at all, whether each passage is relevant, whether a generated segment is supported by its passage, and how useful the answer is. Generation becomes a search over segments guided by these critiques. You will rarely train such a model yourself, but the idea ported directly to frozen models: grade before you generate. Corrective RAG (CRAG) adds a lightweight retrieval evaluator that labels retrieved documents correct, incorrect or ambiguous; correct documents are refined (split into strips, irrelevant strips dropped), incorrect ones trigger a fallback such as web search, and ambiguous ones get both. In practice both become a small graph: retrieve → grade each chunk with a cheap LLM prompt → if too few pass, rewrite the query or search elsewhere → generate → check groundedness → answer or loop, with a step budget. This "adaptive RAG" is the canonical example in orchestration frameworks (9.10 Frameworks and SDKs).
Intuition. Naive RAG trusts its librarian blindly. Self-RAG and CRAG add a reader who skims each book before quoting it, and sends the librarian back when the pile is useless.
RAPTOR: retrieval across levels of abstraction
A question like "what recurring risks appear across these forty quarterly reports?" has no single supporting chunk. RAPTOR (2024) builds a tree: embed chunks, cluster them (Gaussian mixtures over dimensionality-reduced embeddings, 3.11 Clustering: k-means, GMMs/EM, DBSCAN and hierarchical), summarize each cluster with an LLM, embed the summaries, and repeat until one root remains. At query time retrieve from all levels at once (the "collapsed tree"), so a thematic question pulls high-level summaries while a detail question pulls leaves. The cost is an indexing pass of LLM calls proportional to corpus size, repeated when the corpus changes.
GraphRAG and knowledge graphs
GraphRAG (Microsoft, 2024) goes further: an LLM extracts entities and relationships from every chunk into a knowledge graph; community detection (the Leiden algorithm) groups the graph into hierarchical communities; an LLM writes a summary for each community. Global questions ("what are the main themes?") are answered by a map–reduce over community summaries; local questions start from matched entities and expand through their neighbourhoods and source chunks. It shines on sense-making over large corpora and produces answers naive RAG cannot, but indexing is expensive (LLM calls for every chunk, then every community) and the graph must be rebuilt as documents change. Lighter alternatives — entity tags as metadata, a graph database queried by an agent, or "vector search plus a few hops" — capture much of the benefit for most teams.
Agentic RAG: iterative and multi-hop retrieval
Make retrieval a tool (9.6 Generation side: context assembly, citations, structured outputs and tool calling) and wrap it in a reasoning loop (10.3 Reasoning and planning for agents): plan (what do I need to know?), retrieve, read, decide (enough? which gap remains?), refine the query, repeat, then answer with citations. Multi-hop questions ("who directed the film that won Best Picture the year the company was founded?") decompose into chains where the second query depends on the first answer; interleaving reasoning and retrieval in a ReAct-style trace (10.3 Reasoning and planning for agents) handles this where single-shot retrieval cannot. Add routing so the agent chooses among a vector index, a SQL database, a web search tool, or no retrieval. The "deep research" products of 2025–26 are agentic RAG over the open web with long budgets. Guard against loops with step budgets and stop conditions, and log every trajectory — agent evaluation is its own craft (10.9 Evaluating agents).
Structured data: text-to-SQL and table question answering
Numbers live in databases. Text-to-SQL gives the model the schema (table and column names, types, a few sample rows and descriptions), few-shot examples of question→SQL pairs, and asks for a query; you execute it read-only with row and time limits and feed the result back for the final answer. The hard parts: large schemas (retrieve the relevant tables first — "schema linking" is RAG over the schema), ambiguous business terms (a semantic layer that defines "active customer" once), and verification (execute, inspect errors, let the model self-correct, cross-check with a second formulation). For spreadsheets and small tables, a pandas code-execution tool is often simpler and more reliable than SQL generation.
Multimodal RAG
Documents are not only text. Three routes: (1) caption everything — describe images, charts and slide visuals with a vision-language model (8.10 Multimodal models) and index the captions as text; (2) joint embeddings — CLIP-style models embed images and text in one space so a text query retrieves images directly; (3) page-image retrieval — models that embed rendered PDF pages (late interaction over image patches) retrieve the right page without OCR, and a vision-language model then reads the page. Audio goes through transcription, and timestamps become citations.
Long context plus retrieval, and memory
With windows of hundreds of thousands of tokens and prompt caching (9.9 Production: latency, cost, caching, guardrails, security and privacy), "retrieve, then read the whole document" is often better than reading five fragments: retrieval selects documents or sections, and the model reads them in full. Caching makes repeated reads of the same large context cheap, and lets a corpus of a few hundred thousand tokens skip retrieval entirely for a session. Finally, an agent's memory (10.4 Memory and context engineering) is RAG over its own history — summaries, files and embeddings of past interactions retrieved when relevant — so every technique here transfers.
Choosing a pattern
| Pattern | Indexing cost | Query cost | Gains | Reach for it when |
|---|---|---|---|---|
| Grading (Self-RAG / CRAG) | none | one or two cheap LLM calls | fewer ungrounded answers | hallucinations traced to bad retrieval |
| RAPTOR | LLM summaries proportional to corpus | low | thematic, multi-level questions | reports, books, long manuals |
| GraphRAG | high (extraction plus summaries) | medium | global sense-making | analysis over large corpora |
| Agentic / multi-hop | none | several LLM calls | complex, chained questions | research assistants |
| Text-to-SQL | schema documentation | one to three LLM calls | exact numbers | analytics questions |
| Multimodal | captions or image embeddings | medium | visual content | slides, scans, diagrams |
Start with the cheapest pattern that addresses the measured failure, and prove the gain on your test set (9.8 Evaluating and observing LLM apps) before adding the next.
Where this shows up later
- These patterns are the first real "agents" you build — 10.1 What is an agent? Loops, autonomy and workflows vs agents, 10.3 Reasoning and planning for agents, 10.5 Agentic workflow patterns; agent memory — 10.4 Memory and context engineering.
- Each pattern must prove it pays for itself — 9.8 Evaluating and observing LLM apps; orchestration frameworks implement them — 9.10 Frameworks and SDKs.
- Clustering and Gaussian mixtures (3.11 Clustering: k-means, GMMs/EM, DBSCAN and hierarchical) power RAPTOR; vision-language models (8.10 Multimodal models) power multimodal RAG.
Common pitfalls (4)
- Adopting GraphRAG for a corpus of FAQ pages; the indexing bill dwarfs the benefit when questions are local and factual.
- Running an agentic retrieval loop without a step budget or stop condition — it can loop, spend, and still not answer.
- Letting generated SQL run with write permissions or no row limits.
- Grading chunks with the same large model you use to answer; a small, fast grader keeps the loop affordable.
Recap: all 7 key points
- Diagnose before adding complexity: each naive-RAG symptom maps to a specific pattern with a specific cost.
- Self-RAG and corrective RAG grade retrieved chunks before generating and fall back (rewrite, search elsewhere) when too few pass — the adaptive-RAG loop.
- RAPTOR clusters and recursively summarizes chunks into a tree so thematic questions retrieve summaries while detail questions retrieve leaves.
- GraphRAG extracts an entity–relation graph, detects communities and summarizes them; global questions become a map–reduce over community summaries, at high indexing cost.
- Agentic RAG wraps retrieval-as-a-tool in a plan–retrieve–read–decide loop with routing and budgets; multi-hop questions need chained, dependent queries.
- Text-to-SQL needs schema linking, a semantic layer, read-only execution and verification; pandas tools often beat SQL generation for small tables.
- Multimodal RAG uses captions, joint image–text embeddings or page-image retrieval; long-context models shift retrieval from chunks toward whole documents; agent memory is RAG over history.
Watch
The routing and query-structuring parts, plus the RAPTOR and ColBERT indexing parts, implement patterns from this lesson in code.
Read
Exercises
- CRAG-lite
Extend the minimal pipeline from 9.6 Generation side: context assembly, citations, structured outputs and tool calling: after retrieval, ask a small, fast model to label each chunk relevant/irrelevant to the question. If fewer than two chunks pass, rewrite the query once and retrieve again; if still fewer than two, refuse. Measure, on your test set, how often the loop triggers and whether faithfulness (9.8 Evaluating and observing LLM apps) improves.
- Text-to-SQL over a sample database
Load a small SQLite database (any public sample schema with 5–10 tables). Build a prompt with the schema, three sample rows per table and five question→SQL examples. For 15 natural-language questions, generate SQL, execute read-only with a row limit, feed results back for the final answer, and allow one self-correction on error. Report accuracy and the failure types.
- RAPTOR-lite
Take 100 chunks from one long document. Cluster their embeddings into about 10 groups (3.11 Clustering: k-means, GMMs/EM, DBSCAN and hierarchical), summarize each group with an LLM, and index both chunks and summaries. Ask five detail questions and five thematic questions; record which level each answer's best source came from.
- Multi-hop with and without decomposition
Write ten two-hop questions over your corpus. Answer them with single-shot retrieval and with a loop that decomposes into sub-questions whose second query uses the first answer. Compare correctness and total tokens.
Quiz
Flashcards
These cards are now in your spaced-repetition queue. Review all due cards →