Take the user's query. Before you embed it, append a handful of synonyms and domain anchor terms. For a query about a retry interval, you bolt on "timeout, backoff, authentication, serialisation, config". Then embed the enriched string and search. The reasoning feels airtight: more relevant terms means a richer query means better recall. It is a classic, respected move.

It is also, for a dense encoder, close to sabotage. Appending terms before embedding does not enrich the query vector. It averages it — dragging the single point that represented the user's intent toward the centroid of whatever you bolted on. The encoder exists precisely to place that point well. Expansion moves it somewhere worse and then asks the encoder to find the answer from the wrong location.

The thesis of this post is one word: asymmetric. Query expansion is a lexical technique. It is genuinely good for lexical and reasoning-based retrievers, and genuinely bad for the dense one. Same user intent, two differently-prepared queries. If you run a hybrid stack (and the first post in this series, Your Hybrid Search Is a Voting Machine, Not a Ranker, argued you probably do), you must prepare the query differently for each arm. Sending one expanded string to both is a category error dressed up as a best practice.

What "expansion" means, and why the encoder hates it

Term expansion comes from the sparse-retrieval world. BM25 and its relatives score a document by how many query terms it contains, weighted by rarity. If the user types "timeout" and the document says "deadline", BM25 scores zero on that term. The tokens do not match. So you expand the query: add "deadline, expiry, TTL", and now the document lights up. Expansion buys you lexical coverage. It papers over the fact that a bag-of-words model has no idea two different strings mean the same thing.

A dense encoder has the opposite problem, or rather, it does not have that problem at all. Its entire job is to map "timeout" and "deadline" to nearly the same point in vector space. Synonymy is handled inside the model. When you hand it a pre-expanded string, you are not adding information it lacked. You are adding tokens it will fold into a single pooled representation. Many sentence encoders mean-pool their token vectors, and for those, every token you append literally gets a vote in where the final point lands. (Some encoders instead read out a learned [CLS]-style summary vector, which is not a plain arithmetic mean, but it is still a single point computed from all the tokens, and adding off-topic tokens still drags it. The mean-pooled case is just the one where the geometry is exact and easy to show.)

That is the mechanism. A query embedding is a point. Concatenating anchor terms and re-embedding moves the point toward the region those terms occupy. If the anchors sit near the user's intent, no harm. If they sit anywhere else (and "authentication", "serialisation", "config" almost always do, relative to a question about a retry interval), you have pushed the query away from the passage that answers it.

The geometry, made concrete

Let me build a tiny vector space by hand so you can watch the effect happen. Fixed vectors, no model, fully deterministic. Run it and you will get the same numbers I did.

I will use a 5-dimensional space where each axis is a made-up "concept". Think of the dimensions as: retry, timing, auth, serialisation, general-config. A user asks what the default retry interval is. The passage that answers them scores high on retry and timing and low on everything else.

import numpy as np

def unit(v):
    return v / np.linalg.norm(v)

def cos(a, b):
    return float(np.dot(unit(a), unit(b)))

# dims: [retry, timing, auth, serialisation, general-config]
query  = np.array([0.9, 0.8, 0.1, 0.2, 0.3])  # "what is the default retry interval"
target = np.array([1.0, 0.7, 0.0, 0.1, 0.2])  # the passage that answers it

# Anchor terms a well-meaning engineer appends to "help" the retriever.
auth   = np.array([0.2, 0.0, 0.9, 0.1, 0.6])
serial = np.array([0.1, 0.1, 0.2, 0.9, 0.7])
config = np.array([0.3, 0.0, 0.4, 0.4, 0.9])  # generic config noise

print("baseline    :", round(cos(query, target), 4))

# Naive expansion = average the query in with the anchor vectors.
expanded = np.mean([query, auth, serial, config], axis=0)
print("3 anchors   :", round(cos(expanded, target), 4))

one_anchor = np.mean([query, auth], axis=0)
print("1 anchor    :", round(cos(one_anchor, target), 4))

Output:

baseline    : 0.9842
3 anchors   : 0.5918
1 anchor    : 0.7782

The bare query sits at cosine 0.984 to the passage that answers it, nearly on top of it. Add a single anchor term and it falls to 0.778. Add three and it collapses to 0.592. Nothing about the target changed. The passage is exactly as relevant as it was. We moved the query, and we moved it by averaging in vectors that point elsewhere.

The averaging is the whole story. Mean-pooling k token vectors is literally computing a centroid. The more anchors you add and the further they sit from the intent, the harder the query is dragged toward their mutual middle and away from the one passage you wanted. "Serialisation" and "config" are perfectly sensible words to see near this document. They are also poison here, because the user did not ask about serialisation or config in general. They asked about a retry interval's default, and the encoder had already nailed that.

You can make this worse or milder by choosing friendlier anchors. If every appended term sat near the intent, cosine would barely move. But that is the trap: you cannot know in advance which anchors are safe without already knowing the answer. In production the anchors come from a synonym list or a generative expander that has no idea where the intent lives in the space. So on average, they pull you off.

Why nobody catches it

This regression has a particular signature: it is invisible in exactly the way that keeps it alive for months.

First, it looks like it should help. Query expansion is a named, respected, textbook technique with a decade of IR papers behind it. When you add it, you are not doing something reckless. You are applying a known-good method. Nobody reviews the PR and says "wait, is expansion even valid for a dense retriever?" The phrase carries its own credibility.

Second, the metric moves the wrong way by a little, not a lot. You do not go from working to broken. Recall@10 drifts from, say, 0.71 to 0.66 — a hypothetical trajectory, but a realistic one. That is well inside the range of normal week-to-week noise from a corpus change, a re-chunk, a model version bump. A five-point drop does not scream "revert the expansion". It mutters "hmm, retrieval's been a bit soft lately".

Third, and this is the killer, nobody attributes the drop to the expansion, because expansion is the thing that is supposed to be helping. Causal blame flows toward suspicious-looking changes. The expansion step is not suspicious. It is the reassuring part of the pipeline. So the team goes hunting in the chunker, the encoder, the reranker: everywhere except the one stage everyone "knows" is beneficial. I have watched people spend a fortnight tuning top_k and chunk overlap to claw back points that a two-line change to the query-prep function would have handed back instantly.

The tell, if you are looking for it: the regression correlates with query length after expansion. Short queries that got few anchors are fine. The ones the expander went to town on are the ones that miss. If your worst-performing queries are also your most-expanded, you have found it.

The fix is asymmetric: that is the entire point

The correct architecture does not remove expansion. It routes it. You have one user intent and two retrievers that want opposite things, so you prepare two queries. The sketch below stands in embed, dense_index, bm25_index, and fuse for whatever you already run:

def prepare_queries(raw_query, expander):
    lexical_query = expander(raw_query)   # synonyms, anchors, morphological variants
    dense_query   = raw_query             # clean intent, untouched
    return dense_query, lexical_query

dense_q, lexical_q = prepare_queries(user_query, my_expander)
dense_hits   = dense_index.search(embed(dense_q), k=50)
lexical_hits = bm25_index.search(lexical_q,       k=50)
results = fuse(dense_hits, lexical_hits)   # RRF or your fusion of choice

• Lexical arm (BM25, SPLADE, anything token-matching): expand freely. This is expansion's home turf. Synonyms and anchors buy real coverage against a bag-of-words scorer that cannot infer them.
• Dense arm (any embedding retriever): send the clean query. Let the encoder do the semantic generalisation it was trained for. Do not pre-average its input.
• Reasoning / generative arm (an LLM that rewrites or decomposes the query): this is a different operation entirely. It produces new intents (sub-questions), each of which you then embed cleanly. That is not term expansion. It is query decomposition, and it is fine.

The asymmetry is not a limitation to apologise for. It is the correct reading of what each retriever is. Lexical retrieval wants surface-form coverage. Dense retrieval wants a faithful point. A technique that adds surface forms helps the first and corrupts the second — by construction, because adding off-topic tokens to a pooled encoder moves the point.

When this is wrong: the honest nuance

I have stated this cleanly, so let me concede where the clean version bends.

Not all "expansion" is centroid-dragging. If your expander appends terms that genuinely co-locate with the intent (tight, on-topic synonyms rather than scattershot domain anchors), the query barely moves and you lose almost nothing. The damage scales with how far the anchors sit from the intent and how many you add. A single well-chosen synonym is nearly free. A dozen "related" nouns is a wrecking ball. The technique is not uniformly bad; it is bad in proportion to the semantic spread of what you append.

Some encoders resist it better than others. Models trained with heavy query-augmentation, or asymmetric query/document encoders tuned for exactly this, can absorb a modest amount of appended text without much drift. If your encoder was trained to expect messy, keyword-stuffed queries, my warning softens. And a [CLS]-pooled model does not average as cleanly as the toy (its readout is nonlinear), so the exact geometry differs even where the direction of the effect does not. Test yours. Do not assume.

Pseudo-relevance feedback is a real, separate thing. Vector-space PRF (pulling top results, then nudging the query embedding toward the ones that look relevant) genuinely helps, and it also moves the point. The difference is direction: PRF moves the query toward vectors that retrieval already suggests are on-target, not toward an arbitrary synonym list. Moving the point is not the sin. Moving it toward a centroid that has nothing to do with the answer is.

So the precise claim is narrower than "never expand for dense retrieval". It is: do not blindly append lexical expansion terms to a dense query and expect the encoder's semantics to survive the pooling. Everything above is a corollary of that.

Do this yourself

A ten-minute experiment that will tell you whether you have this bug, on a public corpus so you can share it:

1. Grab any open text collection: the CFR bulk data, the RFC archive, a Wikipedia dump. Chunk it and embed the chunks with any off-the-shelf sentence encoder.
2. Hand-write 30 natural questions with a known answer chunk each. This is your gold set.
3. Run retrieval three ways: (a) bare query, (b) query + 3 synonyms, (c) query + 5 anchor nouns from elsewhere in the corpus. Same encoder, same index, only the query string changes.
4. Report Recall@10 and, critically, the per-query delta between (a) and (c).
5. Bucket the deltas by post-expansion query length. If the longest queries own the biggest drops, you have reproduced the centroid effect on real data.

Then flip to the fix: keep expansion on the lexical arm only, and confirm the fused result beats every single-arm variant. On a public corpus, expect the qualitative pattern to hold even if your exact numbers differ from mine. The bare dense query typically ties or beats the expanded one, and the win comes from fusion, not from cramming everything into one string.

Three takeaways

1. A dense query is a point, and appending terms averages it. Pre-embedding expansion drags the query toward the centroid of the appended terms. The numpy toy shows the mechanism exactly: 0.984 to 0.592 with three off-topic anchors and nothing else changed.
2. The regression is invisible because expansion is "known good". The metric slips a few points, the drop hides in normal noise, and blame never lands on the reassuring step. Correlate your worst queries with their expansion length to catch it.
3. Route expansion, do not delete it. Expand for the lexical and reasoning arms, but send the clean query to the encoder. Same intent, two prepared queries.

The broader lesson: two pipeline stages both called "retrieval" are not therefore compatible. Lexical expansion and dense encoding want opposite inputs, and a technique proven on one paradigm can silently corrupt the other. Portability across retrieval methods is an assumption, not a property. Check it before you ship it.

Part of Retrieval, Honestly, a series on the unglamorous mechanics of getting the right chunks back. Previous: We Doubled Retrieval Recall Without Touching the Model. Next: You Probably Don't Need a Vector Database.

Here is the result, stated plainly and stripped of anything that could identify the system: on a corpus of long structured documents, section-level recall@5 sat at 28%. We changed no embedding model, no index, no chunker, no query rewriting, no re-ranker. We fixed two defects in how the source documents were extracted into text. Recall@5 went to 66%. That is a 2.36x improvement (\(66 \div 28 \approx 2.36\)), and every hour we had previously spent auditioning better embedding models produced nothing by comparison.

Treat those two numbers as one team's result on one private corpus, not a law of nature. I cannot hand you the corpus, so you cannot rerun them. What you can rerun is the method: the audit below works on any structured corpus you have, public ones included.

This post is about why that happens, why your evaluation harness actively hides it from you, and the checklist that finds it in an afternoon.

The reflex is wrong

When retrieval underperforms, the reflex is universal and it is misdirected. Recall is low, so the retriever must be weak, so we need a better embedding model. Swap the 2023-vintage model for this quarter's leaderboard topper. Add a re-ranker. Turn on query expansion. Buy the bigger index.

Look at the arithmetic before you spend. The published gap between a mediocre general-purpose embedding model and a strong one is real but bounded. Pull up a public retrieval leaderboard like MTEB and compare a competent general-purpose model against the current top entry: on tasks where the weaker model was already reasonable, the gap is typically a handful of nDCG points, not a doubling. Call it generously a 1.2x multiplier on a good day, and you pay for it in latency, in a migration, in re-embedding the whole corpus.

We got a 2.36x multiplier by reading our own documents. The model was never the bottleneck. The bottleneck was that a large fraction of the corpus, as the retriever saw it, did not say what the source document said.

If the text in your index is wrong, no embedding model can save you. The best retriever on earth cannot return a passage that was deleted during extraction, and it cannot cleanly rank a passage that has a foreign appendix stapled to the middle of it. Garbage in is not a cliché here; it is the entire mechanism.

Two defects, both silent

The corpus was long structured documents. Think contracts, RFCs, tax publications, technical standards, product catalogues: anything with a nested hierarchy of numbered sections, sub-items, and appendices. Extraction turned each source file into a tree of nodes. Two things were wrong with that tree, and both survived undetected for months.

Defect one: cross-boundary contamination. Many of these documents carry a nested schedule or appendix: a block of tabular or densely formatted material that belongs logically at the end, under its own heading. During PDF extraction, the reading-order heuristic lost the boundary. The appendix's text was spliced into the body of whichever main section happened to sit near it on the page. So a node that should have read as one clean topic instead read as that topic followed, mid-sentence, by three hundred words of unrelated schedule.

What that does to an embedding is exactly what you would predict. The vector for that node is now a blend of two unrelated things. It sits in the wrong place in the space, and it matches queries for neither the real section nor the appendix cleanly. In recall@5, that node quietly stops showing up.

Defect two: dropped children. The hierarchy had parents with child sub-items (a section with its enumerated sub-clauses beneath it). For a class of documents, the extractor attached the parent heading and then silently dropped the children. The node existed. It had a title. It had a plausible length, because the parent's own lead-in text was there. It just did not contain the sub-items that held the actual answer to most queries.

Neither defect throws. Neither produces an empty string, a null, or a stack trace. The pipeline runs green end to end. You get a corpus that is the right shape, the right size, with the right number of nodes and sensible-looking titles, and a meaningful slice of it is quietly lying about its contents.

The only reason we found it: a human sat down and read the extracted text of a few dozen nodes next to the original documents. That is it. That is the whole discovery mechanism. No dashboard surfaced it because no dashboard was measuring the one thing that was broken.

Your evaluation cannot see this

This is the uncomfortable part, and it is the reason the reflex persists.

A garbled corpus produces low retrieval scores that are indistinguishable from a weak-model signal. Both look like: query goes in, right passage does not come back, recall@k is low. The number on your evaluation dashboard is identical in both worlds. Nothing about a low recall number tells you whether the cause is upstream (the corpus does not contain what you think) or in the retriever (the model cannot find what is there).

So you form a hypothesis consistent with the evidence ("the model is weak") and every experiment you run from there is downstream of a corrupt input. You swap the model: recall barely moves, because the passages the eval wants are still contaminated or still missing. You add a re-ranker: it re-ranks the same broken text. You conclude the task is just hard. You were measuring a data defect the entire time and attributing it to modelling.

There is a deeper trap. If your evaluation set was built from the same broken extraction, if the "gold" passages were themselves selected from the garbled corpus, then the eval can be internally consistent and completely wrong. It will happily reward a model for retrieving contaminated nodes, because those are the nodes it knows about. The evaluation is only ever as trustworthy as the corpus underneath it, and almost nobody audits that layer.

The lesson generalises past retrieval: an evaluation harness measures the gap between your system and your labels. It says nothing about the gap between your labels and reality. That second gap is where extraction defects live, and it is dark to every metric you have.

The audit, as a checklist

Budget corpus verification first, before a single modelling experiment, and keep spending there until the well is dry. Here is what that looks like concretely. None of it needs a GPU.

1. Structural round-trip. Re-serialise your parsed tree back toward the source format and diff it against the original. You are not chasing byte-equality. You are looking for whole blocks that vanished or moved. If re-serialising the parse cannot approximately reproduce the source document, your parse is not a faithful representation of it, full stop.

2. Child-count assertions per parent. For every parent node, assert its child count against an independent source of truth: the document's own numbering, a table of contents, a structural heading pass. A section that announces sub-items 1 through 8 and yields a node with zero children is a defect you can catch with an inequality, not a human read.

3. Cross-boundary contamination checks. Hunt for boundary markers that leaked across nodes: an appendix heading, a schedule caption, a page-footer string, a "continued from" marker appearing in the middle of a node's body rather than at its edge. Their presence mid-text is a near-certain sign two regions were fused.

4. Length-outlier triage. Compute the node-length distribution and inspect both tails. Nodes far above the median are candidates for contamination (something foreign got merged in). Nodes far below, or empty, are candidates for truncation or dropped content. Outliers are not proof, but they are a ranked worklist.

5. Spot-diff N random nodes against the authoritative source. Sample, say, 30 nodes uniformly at random, and have a human read each one beside the original. This is the step everyone skips and the step that actually found our bug. Thirty reads is an afternoon. It is cheaper than one model migration and it is the only check that sees what the others cannot describe.

A tiny, runnable flagger

You cannot fully automate step five, but you can automate the triage that tells you where to look first. Here is a deterministic, dependency-free flagger over a list of parsed nodes. It encodes the cheap heuristics (empty or missing children, length outliers, and boundary-marker leakage) and returns a ranked worklist. It needs Python 3.10+ for the int | None annotation; drop to Optional[int] on older versions.

import statistics
from dataclasses import dataclass, field

@dataclass
class Node:
    id: str
    title: str
    text: str
    children: list = field(default_factory=list)
    declared_children: int | None = None  # from ToC / numbering, if known

# Markers that should sit at a node's EDGE, never buried in its body.
BOUNDARY_MARKERS = ("appendix", "schedule", "annex",
                    "continued from", "end of section")

def flag_nodes(nodes, body_len_key=lambda n: len(n.text)):
    lengths = [body_len_key(n) for n in nodes] or [0]
    med = statistics.median(lengths)
    # Robust spread: median absolute deviation, floored so it never divides by zero.
    mad = statistics.median([abs(l - med) for l in lengths]) or 1.0

    report = []
    for n in nodes:
        flags = []
        L = body_len_key(n)

        # 1. Dropped children: parent claims sub-items but has none.
        if n.declared_children and not n.children:
            flags.append(f"missing_children(declared={n.declared_children})")

        # 2. Empty / near-empty body.
        if L < max(1, 0.1 * med):
            flags.append(f"short_body(len={L}, median={int(med)})")

        # 3. Length outlier (possible contamination). ~3.5 MAD is a far tail.
        if (L - med) / mad > 3.5:
            flags.append(f"length_outlier(len={L}, median={int(med)})")

        # 4. Boundary marker leaked into the interior of the body.
        low = n.text.lower()
        interior = low[20:-20] if len(low) > 40 else ""
        for m in BOUNDARY_MARKERS:
            if m in interior:
                flags.append(f"boundary_leak('{m}')")

        if flags:
            report.append((n.id, flags))

    # Most-flagged nodes first — that's your read-it-by-hand worklist.
    report.sort(key=lambda r: len(r[1]), reverse=True)
    return report


if __name__ == "__main__":
    nodes = [
        Node("s1", "Overview", "A normal section. " * 20),
        Node("s2", "Definitions", "Def. ", declared_children=8, children=[]),
        Node("s3", "Payment terms",
             "Net thirty days. " * 20 + "Appendix C tariff table " * 40),
        Node("s4", "Notices", ""),
    ]
    for node_id, flags in flag_nodes(nodes):
        print(node_id, "->", ", ".join(flags))

Run it and s2 flags as missing children (and a short body), s3 as a length outlier with a boundary leak, and s4 as an empty body. Those are precisely the three failure shapes from the war story, and the flagger sorts the most-flagged suspects to the top of your reading queue. It does not decide anything: a long section legitimately can run long, and some documents really do have empty placeholder headings. It ranks suspects so a human spends their thirty reads where the payoff is highest.

When this is the wrong diagnosis

Corpus-first is a strong default, not a law. It is the wrong place to spend when:

• Your corpus is already clean text. If you ingest well-formed Markdown, a curated JSON export, or a database dump (no PDF, no OCR, no layout heuristics between you and the words), extraction defects are rare and the round-trip audit finds nothing. Here the model or the chunking genuinely may be your ceiling.

• Your failures are semantic, not structural. If the right passage is in the index, intact, and the retriever still misses it because the query and the passage share no vocabulary, that is a modelling and query-understanding problem. Later posts in this series live in that territory.

• Recall is already high and you are chasing precision. Corpus audits mostly buy back recall. If recall is fine and irrelevant passages are crowding the top ranks, look at re-ranking, not extraction.

The honest framing: extraction defects impose a ceiling on recall that no downstream component can lift. Audit first to find out whether you are fighting the ceiling or the floor. Once the audit is clean, the reflex to reach for a better model becomes the correct one.

Three takeaways

1. A low recall number is ambiguous by construction. It looks identical whether the model is weak or the corpus is corrupt, and your metrics cannot tell you which. Read the text before you trust the score.

2. Corpus verification is the highest-value hour in retrieval, and the cheapest. A structural round-trip, child-count assertions, contamination checks, and thirty hand-read nodes cost less than one model migration. On our corpus they recovered more recall than any model swap we tried.

3. Automate the triage, not the judgement. A deterministic flagger ranks suspicious nodes so your scarce human attention lands where it pays. The final call still needs eyes on the source.

The broader lesson holds well beyond retrieval: your evaluation measures the distance from your system to your labels, never the distance from your labels to the truth — and the most expensive bugs always hide in that second gap.

Retrieval, Honestly — a series on the unglamorous mechanics that decide whether retrieval works. Previous: Your Hybrid Search Is a Voting Machine, Not a Ranker. Next: The Query Expansion That Made Retrieval Worse.

fused_score = sum(1.0 / (k + rank) for rank in ranks_of(doc))  # k = 60

That k = 60 is doing far more than it appears to. On the candidate lists your pipeline actually fuses (five documents, maybe ten), it flattens rank so aggressively that your carefully-ordered retrievers stop mattering and something else takes over entirely.

Your hybrid search is not blending two rankings. It is holding an election. And the election has rules you never wrote.

This post is arithmetic you can do with a calculator, so I'm going to do it in the open, and then hand you a one-afternoon experiment to run against your own system. There's a runnable notebook at the end.

Where 60 comes from

Reciprocal Rank Fusion (RRF) is the standard way to combine two or more ranked lists into one. Given a document d returned by several sources, its fused score is

$$\text{RRF}(d) = \sum_{s ,\in, \text{sources}} \frac{1}{k + \text{rank}_s(d)}$$

where rank_s(d) is where source s placed the document (1 = best), and k is a constant. The method is simple, has no weights to fit, and works well enough that it ships as the default fusion strategy in LangChain, LlamaIndex, Weaviate, Elasticsearch, and roughly everything else.

The constant k = 60 traces back to the paper that introduced RRF: Cormack, Clarke, and Büttcher's 2009 SIGIR paper, Reciprocal Rank Fusion outperforms Condorcet and individual Rank Learning Methods. They evaluated on TREC runs: candidate lists hundreds to thousands of documents long, and they reported the method to be fairly insensitive to the exact value of k in that setting.

Hold onto both of those facts, because they are the whole story:

1. k = 60 was chosen against lists of length ~1000.
2. It was insensitive in that regime.

Your RAG pipeline fuses the top 5. Neither fact survives the move from 1000 to 5.

The 6.6% spread

Here is what k = 60 does to a five-item list. I'll compute the contribution of each rank position:

Look at the top and the bottom. The document your retriever was most sure about scores 0.01639. The document it was least sure about, dead last of the five it returned, scores 0.01538.

$$\frac{0.01639 - 0.01538}{0.01538} \approx 6.6%$$

The entire spread from first place to fifth place is 6.6%. At k = 60, rank position across a top-5 list is worth almost nothing. Your first-stage retriever spent real compute ordering those five results by relevance, and the fusion step just threw most of that ordering away.

That alone should make you suspicious. But it gets sharper when a second source enters.

Agreement beats rank by 1.88×

The whole point of RRF is fusing two retrievers — say a dense embedding search and a lexical or structural one. So ask the question that actually matters: when the two disagree, who wins?

Compare two documents:

• Document A is returned at rank 5 by both sources. Its fused score is 2 × 1/65 = 0.03077.
• Document B is returned at rank 1 by one source, and not at all by the other. Its fused score is 1 × 1/61 = 0.01639.

$$\frac{0.03077}{0.01639} \approx 1.88\times$$

A document both retrievers ranked worst beats a document one retriever ranked best, by nearly two to one.

Sit with what that means. Within-source rank, the thing your retriever is actually trying to optimise, has been rendered nearly irrelevant. What k = 60 fusion actually rewards is agreement between sources. It is not a ranking blend. It is a voting scheme, where being named by two voters beats being named first by one.

Where the crossover actually is

This isn't hand-waving; you can pin the exact point where voting takes over from ranking. Two sources at rank 5 tie with one source at rank 1 when

$$\frac{2}{k + 5} = \frac{1}{k + 1} ;\Longrightarrow; 2(k+1) = (k+5) ;\Longrightarrow; k = 3$$

For any k greater than 3, cross-source agreement at the worst rank beats single-source placement at the best rank. Below 3, within-source rank still wins. The default of 60 is not near that boundary — it is twenty times past it, deep in pure-voting territory.

Here is the same system swept across values of k. Two quantities: the top-5 spread (how much rank position is worth, first to fifth) and the agreement ratio (dual-source-rank-5 versus solo-source-rank-1):

The two closed forms, if you want them: the spread is 4 / (k + 1), and the agreement ratio is 2(k + 1) / (k + 5). As k → ∞ the ratio approaches exactly 2×: every source contributes the same infinitesimal amount regardless of rank, and fusion becomes counting how many lists a document appears on. k = 60 is already 94% of the way to that limit.

Small k restores rank sensitivity. Large k is a vote. You are picking where your system sits on that line every time you accept the default, whether you know it or not.

Is voting actually wrong?

No. And this is the part the confident version of this post would skip.

For a lot of pipelines, agreement is exactly the signal you want. If your two retrievers have genuinely uncorrelated failure modes (a dense encoder that finds semantic paraphrase and a structural retriever that finds things by position in a document tree, say), then two of them independently surfacing the same passage is a strong precision signal. Stronger, often, than one of them ranking it first. In a precision-critical setting, where a wrong passage downstream is expensive, voting is defensible. Arguably correct.

But that argument has a precondition, and the precondition is where teams get hurt: the two retrievers have to be independent, and comparably good.

Pair a strong retriever with a weak one and k = 60 does something you almost certainly did not intend. It hands the weak retriever a veto. Because within-source rank barely counts, the weak source's mere presence on a list is enough to pull a document up past a strong source's top pick. You didn't build a fusion where the better retriever leads and the worse one assists. You built one where a document needs two votes, and your worst voter is holding one of them.

The honest summary: k = 60 is a bet that your sources are equally trustworthy and independent. Sometimes that bet is right. The problem is that it's being placed silently, on your behalf, by a default copied from a paper about a different problem.

The experiment

The good news is that this is one of the cheapest things in your entire stack to check, because it needs no training, no labels beyond a small relevance set, and no GPU. Here is the whole thing.

def rrf_fuse(rankings, k=60):
    """
    rankings: {source_name: [doc_id, ...]}  — each list in rank order, best first.
    returns: [(doc_id, score), ...] sorted best-first.
    """
    scores = {}
    for ranked in rankings.values():
        for rank, doc_id in enumerate(ranked, start=1):
            scores[doc_id] = scores.get(doc_id, 0.0) + 1.0 / (k + rank)
    return sorted(scores.items(), key=lambda kv: kv[1], reverse=True)


def recall_at_k(fused, relevant, k=5):
    top = {doc for doc, _ in fused[:k]}
    return len(top & relevant) / len(relevant) if relevant else 0.0


def sweep(queries, ks=(1, 5, 10, 20, 60)):
    """queries: [{'rankings': {...}, 'relevant': {doc_id, ...}}, ...]"""
    for k in ks:
        r = sum(recall_at_k(rrf_fuse(q["rankings"], k=k), q["relevant"])
                for q in queries) / len(queries)
        print(f"k={k:>3}  recall@5={r:.3f}")

And the arithmetic from this whole post, so you can confirm every number above yourself:

for k in (1, 3, 5, 10, 20, 60, 1000):
    spread = 4 / (k + 1)                  # first-to-fifth spread on a top-5 list
    agreement = 2 * (k + 1) / (k + 5)     # dual-source rank 5  vs  solo-source rank 1
    print(f"k={k:>4}  spread={spread:6.1%}  agreement={agreement:.2f}x")

To run the sweep for real, you need a modest golden set: a handful of queries, each with the set of document IDs that are actually relevant. Don't have one? Build it against a public corpus so your results are reproducible and shareable: the EU AI Act text, a slice of the US CFR, or the RFC archive all work. Fifty queries is enough to see the shape; a couple of hundred makes the comparison decisive.

Then read the sweep:

• Recall is flat across k. The constant doesn't matter for your corpus. Good. You've closed the question with evidence, and you can stop wondering.
• Recall moves. Then 60 is leaving accuracy on the table, and the peak of that curve is a free improvement you get by changing one number.

Either way you now know, instead of inheriting. And if you do this on a public corpus, you have a publishable result and a notebook, which is worth more than the accuracy bump.

The takeaway

k = 60 is not wrong. It is unexamined: a default tuned for thousand-document TREC lists, quietly repurposed to fuse your top five, where it converts a ranking blend into a vote and hands equal weight to unequal retrievers.

Three things to take:

1. On short lists, RRF measures agreement, not rank. At k = 60, cross-source agreement outweighs within-source position by 1.88×, and the crossover is all the way down at k = 3.
2. That behaviour is a bet on your retrievers being independent and comparably good. If they aren't, you've built a veto, not a blend.
3. It's an empirical question with a one-afternoon answer. Sweep k, look at recall@5 and MRR, and let your own corpus decide.

The broader lesson, which I'll keep coming back to in this series: the most consequential numbers in a machine learning system are often not the ones anybody tuned. They're the ones somebody copied.

Retrieval, Honestly — a series on the parts of RAG that everyone ships and almost nobody measures. This is the first post. Next: We Doubled Retrieval Recall Without Touching the Model. If you want the arithmetic-first take on retrieval and evaluation as it goes out, the newsletter below is where the k-sweep in this post lands as a runnable notebook.