Chunking and ingestion

Ingestion is where RAG projects actually fail

Everything downstream reads the text you extract, so parsing quality sets the ceiling. PDFs lose reading order across columns, interleave headers and footers into paragraphs, flatten tables into word soup, and hide scanned pages that need OCR. HTML carries navigation menus and cookie banners. Slides have titles without sentences; spreadsheets have structure without prose; code has syntax that naive splitters break mid-function. The practical pipeline converts every source to Markdown as a common intermediate — headings, lists, tables and code fences survive, and they carry the structure you will exploit when chunking. Tooling matured quickly: layout-aware converters (open-source and commercial) recover tables and reading order from most PDFs, and vision-language models (8.10 Multimodal models) handle the rest. Budget real time here, inspect outputs by eye, and keep the original bytes and page numbers for citations.

Why chunk at all

Three constraints force chunking. Embedding models (9.2 Embeddings deep dive) have a maximum input (often 512 tokens, 8k for newer ones) and, more importantly, one vector cannot represent ten topics: the longer the text, the more its embedding becomes a blurry average and the worse it matches a specific question. The generator has a context budget (9.6 Generation side: context assembly, citations, structured outputs and tool calling), so retrieving an 80-page manual to answer one question is wasteful and invites "lost in the middle". And the chunk is the unit of evaluation and citation — you can only measure recall@$k$ (9.5 Retrieval quality: hybrid search, reranking, query transforms and metrics) and point a user to a passage if there is a passage.

The trade-off is sharp. Small chunks match precisely but lose the context needed to interpret them (a pronoun, a table header, the product the paragraph is about); large chunks keep context but dilute the embedding and burn tokens. There is no universal best size — typical sweet spots are 200–800 tokens with 10–20% overlap — so the only correct answer is measure on your data.

Five chunking strategies

1. Fixed-size by tokens with overlap. Count tokens with a tokenizer (6.1 Text to numbers: tokenization) and cut every $n$ tokens, repeating the last $o$ tokens so a sentence straddling a boundary appears whole in one chunk. Dumb, fast, and a strong baseline.
2. Recursive splitting. Try to split on the largest separator first (section breaks, then paragraphs, then sentences, then words) until pieces fit the size limit. Respects natural boundaries most of the time.
3. Structure-aware. Use the document's own structure: Markdown headings, HTML sections, slide boundaries, functions and classes in code (via a parser). Attach the heading path (Manual › Installation › Linux) as metadata and often as text prepended to the chunk.
4. Semantic chunking. Embed consecutive sentences and start a new chunk where the cosine similarity between neighbouring sentences drops sharply — topic shifts become boundaries. Costs an embedding call per sentence.
5. LLM-based chunking. Ask a model to rewrite a passage into self-contained propositions or to decide boundaries. Highest quality, highest cost; reserve it for small, high-value corpora.

Greg Kamradt's "five levels" video walks through exactly this ladder with code. Start at level 1 or 2, add structure awareness, and climb further only when your evaluations demand it.

import tiktoken

def chunk_by_tokens(text, size=400, overlap=60):
    enc = tiktoken.get_encoding("cl100k_base")
    ids = enc.encode(text)
    chunks, start = [], 0
    while start < len(ids):
        end = min(start + size, len(ids))
        chunks.append(enc.decode(ids[start:end]))
        if end == len(ids):
            break
        start = end - overlap        # step back so boundaries are covered twice
    return chunks

Small-to-big: index small, return big

Match on precise units, hand the model rich context. Sentence-window retrieval embeds single sentences but returns the sentence plus its neighbours. Parent–child chunking embeds small child chunks (100–200 tokens) that point to a parent (a section or page); retrieval ranks children, then de-duplicates and returns parents. Multi-representation indexing embeds an LLM-written summary (or a hypothetical question) for each section and returns the full section. All three decouple what you match on from what the model reads.

Contextual retrieval

Chunks lose their context by construction: "the company's revenue grew 3%" does not say which company or which quarter. Anthropic's contextual retrieval (2024) fixes this by asking an LLM to write a short, 50–100 token context for each chunk — this chunk is from ACME Corp's Q2 2023 filing; the previous section covered… — and prepending it before computing both the embedding and the BM25 index. In Anthropic's experiments this cut the top-20 retrieval failure rate by 49%, and by 67% when combined with a reranker (9.5 Retrieval quality: hybrid search, reranking, query transforms and metrics). Generating a sentence for every chunk of a corpus is affordable because the whole document can sit in a cached prompt (9.9 Production: latency, cost, caching, guardrails, security and privacy) while the model is asked about each chunk in turn.

Metadata, deduplication and incremental ingestion

Every chunk should carry: source id and URL, title, section path, page or line numbers, timestamps (created, modified, ingested), language, access-control labels, and the embedding model version. Metadata powers filters (9.3 Vector search and indexes), citations (9.6 Generation side: context assembly, citations, structured outputs and tool calling) and permission checks (9.9 Production: latency, cost, caching, guardrails, security and privacy). Deduplicate: exact duplicates by hash, near-duplicates (boilerplate, re-posted articles) by MinHash or embedding similarity — duplicates waste top-$k$ slots and make the model repeat itself. Make ingestion incremental: hash each document and chunk, re-parse and re-embed only what changed, propagate deletions to the index, and keep a manifest so you can answer "which version of which document produced this chunk?".

Evaluating chunking

Chunking is tuned with the retrieval test set from 9.5 Retrieval quality: hybrid search, reranking, query transforms and metrics: for each candidate strategy and size, index, run the queries and compare recall@$k$ and context precision; then confirm with a small end-to-end answer evaluation (9.8 Evaluating and observing LLM apps), because a strategy that retrieves "correct" chunks too short to answer from is not a win. The interactive demo below lets you feel the effect: paste a few documents, move the chunk-size slider and watch which chunks a TF-IDF retriever (6.2 Classical NLP and n-gram language models) pulls for a query.

Intuition. A chunk is a search result snippet you design in advance. If a human could not tell what a chunk is about without the rest of the page, neither can the embedding model.

Where this shows up later

• Retrieval quality and the test set you need to tune chunking — 9.5 Retrieval quality: hybrid search, reranking, query transforms and metrics; assembling chunks into a prompt — 9.6 Generation side: context assembly, citations, structured outputs and tool calling.
• Hierarchical and graph indexes are chunking taken further — 9.7 Advanced RAG patterns; metadata becomes the security boundary in production — 9.9 Production: latency, cost, caching, guardrails, security and privacy.
• Tokenization (6.1 Text to numbers: tokenization) decides what "400 tokens" means; embeddings (9.2 Embeddings deep dive) decide how chunk length hurts.