Generation side: context assembly, citations, structured outputs and tool calling

From chunks to a grounded answer

Retrieval hands you $k$ chunks; generation must turn them into an answer the user can trust. The prompt does four jobs: set the role and rules (answer from the sources, cite them, admit when they do not contain the answer), present the sources in a machine-readable layout (numbered, with titles and metadata), pose the question, and specify the output format. A template that works:

You are a support assistant for ACME. Answer using ONLY the numbered sources.
Cite every factual sentence with its source number like [2]. If the sources do
not contain the answer, reply exactly: "I don't have that information."

Sources:
[1] (Installation guide › Linux, p. 4) ...chunk text...
[2] (Release notes 3.2, 2026-03-10) ...chunk text...

Question: ...

Frontier models follow such rules well; smaller models need few-shot examples (8.4 Prompting as programming) and tighter formats. Put the stable instructions first and the question last — that ordering also makes prompt caching effective (9.9 Production: latency, cost, caching, guardrails, security and privacy).

Budgeting and ordering the context

Context is finite and not free: $\text{budget} = \text{window} - (\text{system} + \text{question} + \text{reserved output})$. Decide $k$ by tokens, not by count; de-duplicate near-identical chunks; group chunks from the same document under one header; include the heading path so each chunk is interpretable on its own. Order matters: Liu et al. (2023) showed that models use information at the beginning and end of long contexts far better than information in the middle — the "lost in the middle" effect — so place the top-ranked chunks first (or first and last) rather than in an arbitrary order. When sources are long, compress: keep only sentences a small model marks as relevant, or summarize low-ranked chunks. More context is not better once the relevant material is in; extra chunks add distraction and cost (failure point FP4 in 9.1 Why RAG, and the canonical pipeline).

Intuition. The prompt is an exam paper you hand the model: the question, the allowed reference material, and the marking scheme. Ambiguous instructions produce ambiguous answers.

Citations and the "no answer" path

Numbered sources make citations cheap: map [2] back to the chunk's URL, page and section and render it as a link. For higher assurance, verify after generation: split the answer into claims and check each against the cited chunk with a small natural-language-inference model or an LLM judge — the same computation as the faithfulness metric in 9.8 Evaluating and observing LLM apps. Some providers now return citations natively, as character spans into the documents you passed, which removes the parsing step. Design the abstain path deliberately: a reranker score below a threshold, or an explicit "insufficient information" verdict, should produce a helpful refusal plus the closest sources and a clarifying question. A confident wrong answer is the worst outcome a RAG system can produce; an honest "not in my documents" is a feature.

Structured outputs

Downstream code should never parse prose. Define the answer shape as a JSON schema (or a Pydantic model) and use the provider's structured output mode, which constrains decoding so the output is always valid JSON that matches the schema (8.5 Decoding and sampling). A good RAG answer type carries the text, the citation ids it used, an "answerable" flag or confidence, and optional follow-up questions. Libraries such as Instructor and Pydantic AI validate the model's JSON against the schema and retry with the validation error when it fails.

from pydantic import BaseModel

class GroundedAnswer(BaseModel):
    answer: str
    citations: list[int]      # indices into the numbered sources
    answerable: bool

Validate the citations too: every index must exist, and ideally the cited chunk must support the sentence that cites it.

Retrieval as a tool

Instead of always retrieving before generating, expose search_docs(query) as a tool and let the model call it when — and as often as — it needs (10.2 Tool use and function calling). The loop: the model emits a tool call with arguments; your code runs the search and returns results as a tool result; the model continues, possibly calling again with a refined query; finally it answers. This is agentic RAG in miniature (9.7 Advanced RAG patterns): it handles follow-up questions, multi-hop queries and "no retrieval needed" turns naturally, at the cost of extra round-trips and less predictability. Modern APIs support parallel tool calls and streaming of both text and tool arguments; tool descriptions are prompts, so write them as carefully as the system prompt.

A complete minimal RAG in plain Python

Local embeddings (no API key needed) with sentence-transformers, exact cosine search in NumPy, and a grounded prompt sent to an LLM. Everything else in this phase is an upgrade of one of these lines.

import numpy as np
from sentence_transformers import SentenceTransformer
import anthropic

docs = open("chunks.txt").read().split("\n\n")        # your chunker's output (9.4)
embedder = SentenceTransformer("all-MiniLM-L6-v2")
E = embedder.encode(docs, normalize_embeddings=True)  # (N, 384) unit vectors

def retrieve(query, k=5):
    q = embedder.encode([query], normalize_embeddings=True)[0]
    scores = E @ q                                    # cosine: all vectors are unit length
    top = np.argsort(-scores)[:k]
    return [(int(i), float(scores[i]), docs[i]) for i in top]

def build_prompt(query, hits):
    sources = "\n\n".join(f"[{n}] {text}" for n, (_, _, text) in enumerate(hits, start=1))
    return ("Answer using ONLY the numbered sources and cite them like [1]. "
            "If they do not contain the answer, say you don't know.\n\n"
            f"Sources:\n{sources}\n\nQuestion: {query}")

client = anthropic.Anthropic()                         # reads ANTHROPIC_API_KEY

def answer(query):
    hits = retrieve(query)
    msg = client.messages.create(
        model="claude-sonnet-5-5", max_tokens=600,
        messages=[{"role": "user", "content": build_prompt(query, hits)}],
    )
    return msg.content[0].text, hits

text, hits = answer("How do I reset the device to factory settings?")
print(text)
for n, (i, score, chunk) in enumerate(hits, start=1):
    print(f"[{n}] score={score:.3f} {chunk[:80]}...")

Upgrade path, one measured step at a time: hybrid retrieval and a reranker (9.5 Retrieval quality: hybrid search, reranking, query transforms and metrics) replace retrieve; a vector index (9.3 Vector search and indexes) replaces the NumPy matrix when $N$ grows; structured output replaces the free-text answer; a tool-calling loop replaces the fixed retrieve-then-answer order; tracing and evals (9.8 Evaluating and observing LLM apps) wrap the whole thing.

Streaming and the user experience

Stream tokens as they arrive — time to first token dominates perceived latency (9.9 Production: latency, cost, caching, guardrails, security and privacy). Show the retrieved sources immediately (they are ready before generation starts), render citations as links, and make "was this helpful?" a single click, because those clicks become your evaluation data (9.8 Evaluating and observing LLM apps).

Where this shows up later

• The retrieval scores behind the abstain threshold — 9.5 Retrieval quality: hybrid search, reranking, query transforms and metrics; faithfulness checking — 9.8 Evaluating and observing LLM apps; latency and cost of long contexts — 9.9 Production: latency, cost, caching, guardrails, security and privacy.
• Tool calling is the core mechanic of agents — 10.2 Tool use and function calling; structured outputs rely on constrained decoding — 8.5 Decoding and sampling; prompt design — 8.4 Prompting as programming.