Project: a cited RAG assistant over your own documents, with evals
Put the whole phase into one system you can defend
Everything in this phase becomes real when you build one system end to end: ingest a corpus you care about, measure retrieval, ground and cite answers, refuse honestly, and prove it all with an evaluation suite that runs from one command. This is a portfolio piece (11.3 Portfolio project ladder), the base for the agent project in 10.12 Project: build a research agent with tools, an MCP server and an eval harness, and the fastest way to discover which lessons you only thought you understood.
You'll be able to
- Assemble a complete RAG system with hybrid retrieval, reranking, grounded generation and citations
- Build a golden set and an evaluation harness that reports retrieval and answer metrics from one command
- Run controlled experiments (chunking, hybrid search, reranking, contextual retrieval) and attribute improvements
- Instrument latency and cost and apply one caching optimization
- Write a concise technical report with failure analysis and next steps
Key ideas
- Build on a corpus you care about and can legally use; motivation and domain knowledge make error analysis possible.
- Create the golden set before optimizing anything, with unanswerable, multi-hop and exact-identifier questions and a held-out slice.
- Change one thing at a time and record every configuration in the same results table on the same set.
Show 4 more key ideas
- Zero fabricated citations and a working refusal path are non-negotiable acceptance criteria.
- One command for ingestion, one for indexing, one for evaluation — reproducibility is part of the deliverable.
- Measure latency and cost and apply one caching optimization with before/after numbers.
- The report explains what moved the needle and analyses the worst failures; that analysis is what makes the project credible.
Suggested path through this lesson
The brief
Build an assistant that answers questions about a corpus you care about — your own notes, a project's documentation, an open-source codebase's docs, a textbook you have the right to use — with every factual sentence cited, honest refusals when the corpus does not contain the answer, and an evaluation suite that proves it works and catches regressions. Aim for a system you could demo and defend in an interview (11.4 ML and LLM system design and interviews).
Architecture
Keep the stack thin (9.10 Frameworks and SDKs):
- Ingestion: convert sources to Markdown, chunk structure-aware at 300–500 tokens with overlap, attach metadata (source, section path, page) — 9.4 Chunking and ingestion.
- Indexes: an open embedding model (9.2 Embeddings deep dive) in a NumPy/FAISS or pgvector index (9.3 Vector search and indexes) plus a BM25 index.
- Retrieval: hybrid search fused with reciprocal rank fusion, top-50 reranked by a cross-encoder to top-5 — 9.5 Retrieval quality: hybrid search, reranking, query transforms and metrics.
- Generation: grounded prompt with numbered sources, structured output (answer, citations, answerable flag), citation validation and an abstain path — 9.6 Generation side: context assembly, citations, structured outputs and tool calling.
- Evaluation and observability: golden set, retrieval metrics, RAGAS-style metrics, a calibrated judge, tracing — 9.8 Evaluating and observing LLM apps.
- Optional: contextual retrieval on the chunks (9.4 Chunking and ingestion), retrieval as a tool (9.6 Generation side: context assembly, citations, structured outputs and tool calling), prompt caching (9.9 Production: latency, cost, caching, guardrails, security and privacy).
Milestones
Each milestone has an acceptance criterion; do not move on until it is met.
- Corpus and golden set. 30–100 documents. 50 questions: at least 10 unanswerable from the corpus, 10 multi-hop, 10 containing exact identifiers (names, codes, versions). For each answerable question record a reference answer and the relevant chunk ids. Accepted when the set is in version control with a held-out slice of 10.
- Baseline. Dense-only top-5 with the minimal pipeline. Accepted when recall@5, faithfulness, answer correctness, p50 latency and cost per query are recorded in a results table.
- Hybrid and rerank. Add BM25 plus RRF, then the reranker. Accepted when both deltas on recall@5 and MRR are measured on the same set, with per-query diffs for the five largest changes.
- Chunking experiments. Two chunk sizes plus the structure-aware splitter; contextual retrieval on a 50-chunk subset. Accepted when a chosen configuration is justified by recall@5 and context precision.
- Grounded generation. Structured output, citation validation (every cited index exists and supports its sentence), refusal path. Accepted when there are zero fabricated citations on the golden set and at least 8 of 10 unanswerable questions are refused.
- Evaluation harness. RAGAS-style metrics plus a pairwise judge calibrated on 30 items. Accepted when one command produces a versioned results file and the judge agrees with your labels at least 80% of the time.
- Tracing and cost. Traces for every request; p50/p95 latency and cost per query; one caching optimization with measured effect. Accepted when the before/after numbers are in the report.
- Report. What moved the needle most, the five worst questions and why, what you would do next.
Suggested evaluation table
| Configuration | recall@5 | MRR | Faithfulness | Correctness | Refusal accuracy | p50 latency | Cost / query |
|---|---|---|---|---|---|---|---|
| Dense top-5 | |||||||
| + BM25 and RRF | |||||||
| + Reranker | |||||||
| + Better chunking | |||||||
| + Contextual retrieval | |||||||
| + Structured output and abstain |
Fill every cell on the same golden set so differences are attributable to one change at a time.
Rubric
| Criterion | Target |
|---|---|
| Retrieval | recall@5 of at least 0.8 on answerable questions |
| Faithfulness | at least 0.9 on answered questions |
| Refusals | at least 80% of unanswerable questions refused, with sources offered |
| Citations | 100% of citations resolve to real chunks that support the sentence |
| Reproducibility | ingestion, indexing and evaluation each run from one command |
| Write-up | numbers, failure analysis and a clear next-step list in under two pages |
Report template
- Problem and corpus (what, size, why it matters).
- Architecture diagram and the stack, with one sentence of justification per component.
- Golden set construction (sources, counts, held-out slice).
- Results table and the two or three changes that mattered most.
- Failure analysis: five worst questions, the stage that failed, the fix or why none exists.
- Latency and cost, before and after caching.
- Limitations and next steps.
Stretch goals
- Retrieval as a tool in a loop with a step budget, compared against the fixed pipeline on the multi-hop questions (9.7 Advanced RAG patterns).
- A RAPTOR-lite or GraphRAG-lite index for thematic questions (9.7 Advanced RAG patterns).
- Page-image retrieval for a corpus of slides or scanned PDFs.
- A public demo on Hugging Face Spaces with Gradio, with the evaluation report linked from the README.
- A semantic cache with a measured false-hit rate (9.9 Production: latency, cost, caching, guardrails, security and privacy).
Where this shows up later
- The agent capstone extends this system with tools, an MCP server and an agent eval harness — 10.12 Project: build a research agent with tools, an MCP server and an eval harness.
- The report and demo are portfolio material — 11.3 Portfolio project ladder; the architecture is a standard interview case — 11.4 ML and LLM system design and interviews.
Common pitfalls (4)
- Optimizing before the golden set exists, which makes every later improvement unattributable.
- Choosing a corpus you do not understand; you cannot label relevance or judge answers without domain knowledge.
- Letting the judge grade without calibration against your own labels.
- Skipping the report; the write-up is what interviewers and teammates actually read.
Recap: all 7 key points
- Build on a corpus you care about and can legally use; motivation and domain knowledge make error analysis possible.
- Create the golden set before optimizing anything, with unanswerable, multi-hop and exact-identifier questions and a held-out slice.
- Change one thing at a time and record every configuration in the same results table on the same set.
- Zero fabricated citations and a working refusal path are non-negotiable acceptance criteria.
- One command for ingestion, one for indexing, one for evaluation — reproducibility is part of the deliverable.
- Measure latency and cost and apply one caching optimization with before/after numbers.
- The report explains what moved the needle and analyses the worst failures; that analysis is what makes the project credible.
Watch
Revisit the indexing, query translation and routing parts as you decide which upgrades to test.
Read
Exercises
- Milestones 1–2: golden set and baseline
Assemble the corpus and the 50-question golden set, then run the dense-only baseline and fill the first row of the results table. Expected: a clear list of the questions the baseline gets wrong, sorted by stage of failure.
- Milestones 3–4: retrieval bake-off
Add BM25 with RRF, then a cross-encoder reranker, then the chunking variants and contextual retrieval on a subset. Fill the table one row at a time and keep the per-query diffs. Expected: hybrid plus reranking gives the largest recall@5 gain; chunking changes show up mostly on context precision.
- Milestones 5–6: grounded generation and the harness
Implement structured output with citation validation and the abstain path, then the evaluation command with RAGAS-style metrics and a calibrated pairwise judge. Expected: zero fabricated citations, at least 8 of 10 refusals, judge agreement above 80%.
- Milestones 7–8: trace, cache, report
Add tracing, measure p50/p95 latency and cost per query, apply prefix caching (or a component cache) and measure again. Write the two-page report using the template, including the five worst questions and what you would try next.
Quiz
Flashcards
These cards are now in your spaced-repetition queue. Review all due cards →