Our RAG pipeline got worse before it got better

The first time I put a cross-encoder reranker in front of our retrieval stack, answer quality went down. Not by a rounding error. Our grounding score fell about four points and the on-call channel noticed before the dashboard did.

The reranker was doing exactly what it was supposed to do. The problem was that I had been measuring the wrong thing for two months.

What we were measuring

The pipeline was the ordinary shape. Embed the query, pull the top k from a vector index, stuff the results into the prompt, generate. To evaluate it we used recall@k against a labelled set: for each question, does the gold passage appear anywhere in the retrieved set?

Recall@20 was 0.94. That number felt like permission to stop working on retrieval.

The trouble is that recall@k treats position as free. A gold passage sitting at rank 19 counts exactly the same as one at rank 1. That assumption is fine when the consumer is a human scanning a result list. It is wrong when the consumer is a language model with a context budget and a documented tendency to weight the beginning and end of its context more heavily than the middle.¹

So the reranker did its job, reordered the set, and in doing so shortened it. Fewer passages, better ordered. Recall@20 barely moved. Answer quality moved a lot, in the wrong direction, because I had also cut k from 20 to 8 in the same change and the labelled set could not see the difference.

Fixing the measurement first

I stopped touching the pipeline and rewrote the evaluation. Three changes:

1. Rank-aware retrieval metrics. nDCG@k and MRR instead of recall@k, so moving a gold passage from rank 12 to rank 2 registers as an improvement.
2. End-to-end grounding, scored separately. Whether the generated answer is supported by the passages it was given, which is a different question from whether the right passage was retrieved.
3. One change per run. Obvious. I still got it wrong.

Splitting retrieval quality from generation quality was the part that mattered. Before the split, a regression could come from either half and the metric could not tell me which. After it, the reranker showed a clear nDCG gain and the k reduction showed a clear grounding loss, in the same run, as two separate numbers.

Fusing sparse and dense

With honest metrics in place, the change that actually helped was unglamorous: keep the vector search, add BM25 alongside it, and fuse the two ranked lists with reciprocal rank fusion.

RRF scores a document by summing over the ranked lists it appears in:

$$\mathrm{RRF}(d) = \sum_{r \in R} \frac{1}{k + r(d)}$$

where $r(d)$ is the rank of document $d$ in list $r$, and $k$ is a smoothing constant that stops rank-1 results from dominating. The literature suggests 60 and I have never found a reason to move it.

What I like about RRF is what it does not need. No score normalisation between two retrievers whose scores are not on the same scale, no tuned weights per retriever, no training. It only reads ranks.

from collections import defaultdict


def reciprocal_rank_fusion(ranked_lists, k=60, top_n=8):
    """Fuse several ranked lists of document ids into one.

    Each ranked list is ordered best-first. Only ranks are used, so the
    retrievers do not need comparable scores.
    """
    scores = defaultdict(float)

    for ranked in ranked_lists:
        for rank, doc_id in enumerate(ranked, start=1):
            scores[doc_id] += 1.0 / (k + rank)

    ordered = sorted(scores.items(), key=lambda pair: pair[1], reverse=True)
    return [doc_id for doc_id, _ in ordered[:top_n]]


fused = reciprocal_rank_fusion([dense_hits, bm25_hits])

Sparse retrieval earns its place on the queries dense retrieval is worst at: exact identifiers, ticker symbols, a specific defined term. An embedding model will happily return three passages that are about the right concept while missing the one that contains the literal string the user typed.

Where it landed

Numbers from our internal set, same corpus, same generator, one change per row:

Configuration	nDCG@10	Grounding	p95 latency
Dense only, k=20	0.61	0.78	240 ms
Dense only, k=8	0.61	0.71	190 ms
Dense + reranker, k=8	0.74	0.83	410 ms
Hybrid + reranker, k=8	0.79	0.88	445 ms

The reranker costs about 220 ms. That is a real price and on a latency-sensitive path it would be the wrong trade. Here it buys enough grounding to be worth it, and semantic caching takes most of it back on repeat queries.

The part I would tell myself

Retrieval work fails quietly. A generation bug produces something visibly wrong. A retrieval bug produces a fluent, confident answer built on the wrong passage, and it looks fine until someone who knows the domain reads it closely.

So the ordering matters. Build the measurement before the improvement, make it rank-aware, split retrieval from generation, and change one thing at a time. None of that is clever. All of it would have saved me two months.

Footnotes

1. Liu et al., Lost in the Middle: How Language Models Use Long Contexts (2023). The effect is strong enough that retrieval order is a real hyperparameter, not a presentation detail. ↩