Retrieval quality: hybrid search, reranking, query transforms and metrics

Two kinds of relevance

Dense retrieval (9.2 Embeddings deep dive) matches meaning: "how do I reset my password" finds "credential recovery procedure". But it is weak at exact tokens that carry little semantics: product codes, error strings, names, legal clause numbers, function names. Lexical retrieval is the mirror image: exact matches are perfect, paraphrases are invisible. Because real queries contain both, hybrid search — run both, fuse the rankings — beats either alone on most benchmarks, and the gap is largest on domain data full of jargon.

BM25, term by term

BM25 is the lexical workhorse (the default in Elasticsearch and OpenSearch, Lucene, and the rank_bm25 library). For a query $q$ with terms $t$ and a document $D$:

• $f(t,D)$ is the term frequency in $D$. The fraction grows with $f$ but saturates at $k_1 + 1$: the tenth occurrence of "password" adds much less than the first. $k_1 \in [1.2, 2.0]$ sets how fast.
• $|D|/\text{avgdl}$ is the document's length relative to the average; $b \in [0,1]$ (usually 0.75) controls length normalization so long documents do not win merely by containing more words.
• $\text{IDF}(t)$ rewards rare terms: $N$ documents in total, $n_t$ containing $t$. A term in every document scores near zero; a term in one document scores near $\ln N$.

Compared with TF-IDF (6.2 Classical NLP and n-gram language models), BM25 adds saturation and length normalization — two small changes that made it the 25-year champion of lexical search.

Math check. $N = 1{,}000$, $n_t = 10$ ⇒ $\text{IDF} = \ln\left(\frac{990.5}{10.5}+1\right) \approx 4.56$. For $f = 3$, $k_1 = 1.5$, $b = 0.75$ and $|D| = \text{avgdl}$: the fraction is $\frac{3 \cdot 2.5}{3 + 1.5} \approx 1.67$, so the term contributes $\approx 7.6$. With $f = 30$ the fraction is $\frac{30\cdot 2.5}{31.5} \approx 2.38$ — ten times the occurrences, only 1.4× the score.

Fusing rankings: reciprocal rank fusion

Dense scores (cosines near 0.8) and BM25 scores (unbounded, 7.6 above) live on different scales, so adding them is meaningless without calibration. Reciprocal rank fusion sidesteps scales by using ranks:

Example: dense ranks $[A, B, C]$, BM25 ranks $[C, A, D]$. $\text{RRF}(A) = \frac{1}{61} + \frac{1}{62} \approx 0.0325$, $\text{RRF}(C) = \frac{1}{63} + \frac{1}{61} \approx 0.0323$, $\text{RRF}(B) = \frac{1}{62} \approx 0.0161$, $\text{RRF}(D) = \frac{1}{63} \approx 0.0159$. Final order: A, C, B, D — documents both rankers like float to the top. The constant $k$ dampens the dominance of top ranks; 60 is the conventional value and rarely needs tuning. The alternative — min-max normalize each score list and take $\alpha\, s_{\text{dense}} + (1-\alpha)\, s_{\text{bm25}}$ — can beat RRF when you have a test set to tune $\alpha$.

def rrf(rankings, k=60):
    scores = {}
    for ranked_ids in rankings:                 # each list: doc ids, best first
        for rank, doc_id in enumerate(ranked_ids, start=1):
            scores[doc_id] = scores.get(doc_id, 0.0) + 1.0 / (k + rank)
    return sorted(scores, key=scores.get, reverse=True)

Reranking

First-stage retrieval optimizes recall with cheap scores; a reranker spends more compute on fewer candidates to optimize precision. The classic choice is a cross-encoder (9.2 Embeddings deep dive) that reads query and candidate together and emits a relevance score — 50–100 candidates in, the best 5–10 out, typically 50–300 ms on a GPU-backed service. ColBERT late interaction scores $\sum_i \max_j \mathbf{q}_i\cdot\mathbf{d}_j$ over token vectors and is fast enough to rerank thousands. LLM rerankers show a model the query and a list of passages and ask it to order them (listwise) or to judge each one (pointwise); they are the most accurate and the most expensive. Rule: if your latency budget allows it, rerank — it is the most reliable improvement after hybrid search, and it is what let contextual retrieval reach its 67% error reduction (9.4 Chunking and ingestion).

Query transformations

The user's words are not always the best search key. Cheap LLM calls before retrieval fix that:

• Rewriting: fix typos, expand abbreviations, resolve references to earlier turns ("what about the Pro plan?" → "Pro plan pricing").
• Multi-query: generate 3–5 paraphrases, retrieve for each, fuse with RRF — covers vocabulary mismatch and raises recall.
• HyDE (hypothetical document embeddings): ask the model to write a plausible answer, embed that, and search with it — answers look like passages, questions do not.
• Step-back: first ask a more general question ("what are the rules for refunds?") when the specific one is too narrow.
• Decomposition: split multi-hop questions into sub-questions, retrieve for each, then compose (9.7 Advanced RAG patterns).
• Routing: classify the query to pick an index, a tool (SQL, web search) or to skip retrieval entirely for small talk (10.5 Agentic workflow patterns).

Each adds an LLM round-trip of latency; apply them where your failure analysis shows vocabulary mismatch or multi-hop questions, not everywhere.

Measuring retrieval: recall@k, MRR and nDCG

Let the set of relevant chunks for a query be $R$ and the ranked list the system returns be $(d_1, d_2, \dots)$.

• Recall@k $= \dfrac{|R \cap \{d_1,\dots,d_k\}|}{|R|}$ — the fraction of relevant chunks that made the cut; the ceiling on answer quality (9.1 Why RAG, and the canonical pipeline).
• Precision@k $= \dfrac{|R \cap \{d_1,\dots,d_k\}|}{k}$ — how much of the context is useful.
• MRR $= \dfrac{1}{|Q|}\sum_{q \in Q} \dfrac{1}{\text{rank of the first relevant result}}$ — rewards putting a right answer on top.
• nDCG@k $= \dfrac{\text{DCG@k}}{\text{IDCG@k}}$ with $\text{DCG@k} = \sum_{i=1}^{k} \dfrac{\text{rel}_i}{\log_2(i+1)}$ — graded relevance, discounted by position; IDCG is the DCG of the ideal ordering, so nDCG $\in [0,1]$.

Worked example: $R = \{A, B\}$, the system returns $[C, A, D, B, E]$, binary relevance. Recall@3 $= 1/2$, recall@5 $= 1$, precision@5 $= 2/5$, and this query contributes $1/2$ to MRR (first relevant at rank 2). $\text{DCG@5} = \frac{1}{\log_2 3} + \frac{1}{\log_2 5} \approx 0.631 + 0.431 = 1.062$; $\text{IDCG@5} = \frac{1}{\log_2 2} + \frac{1}{\log_2 3} = 1 + 0.631 = 1.631$; $\text{nDCG@5} \approx 1.062 / 1.631 = 0.651$.

Report recall@k at the $k$ you actually pass to the generator, MRR for "is the best chunk on top", and nDCG when you have graded labels.

Building the retrieval test set

You need 50–200 queries with labelled relevant chunks. Sources, best first: real user questions from logs; questions written by domain experts; synthetic questions generated by an LLM from sampled chunks, then filtered by a second pass asking "would a user really ask this?". Beware synthetic questions that copy the chunk's wording — they make lexical search look perfect. Store the set in version control, add every production failure to it, and keep a held-out slice so you do not tune to your own test. Every change — embedding model, chunk size, hybrid weights, reranker — gets a before/after on this set and a look at the five worst queries.

Where this shows up later

• Chunk design is tuned with these metrics — 9.4 Chunking and ingestion; prompt assembly consumes the reranked top-$k$ — 9.6 Generation side: context assembly, citations, structured outputs and tool calling.
• Query transformations grow into agentic retrieval — 9.7 Advanced RAG patterns; retrieval metrics join end-to-end metrics in 9.8 Evaluating and observing LLM apps.
• BM25 is TF-IDF (6.2 Classical NLP and n-gram language models) refined; cross-encoders are BERT-style encoders (7.5 Encoder-only: BERT and friends (and sentence embeddings)).