Reranking Explained: When a Cross-Encoder Reranker Pays Off, With Recall and Latency Numbers

A reranker is a second, slower model that re-reads the top handful of chunks your retriever found and puts them in a better order. I measured whether that is worth the extra step on 2026-10-08. On a deliberately confusing set of product manuals it lifted recall@1 from 53.1 to 85.3 percent. On a real document where the first stage was already good, it moved recall@1 from 84.4 to 87.5, which is one question out of 32. The cost was between 0.25 and 1.3 seconds per query for the small model I used, on one CPU thread.

This post shows the script, the numbers, and the rule I'd use to decide whether to add one. The test sets are described in the chunking comparison; the BM25 half of the "hybrid" rows is explained in the hybrid search post.

What a reranker does, in one picture of the pipeline

1. First stage: embed the question, take the top 20 chunks by cosine similarity (and optionally merge in BM25 results). Fast, because chunk vectors were computed in advance.
2. Rerank: feed (question, chunk) pairs to a cross-encoder, which reads both together and outputs one score per pair. Sort by that score.
3. Keep the top 3 to 5 for the LLM prompt.

The cross-encoder can notice things a vector comparison cannot, such as that "LS-310" in the question matches "LS-310" in the chunk rather than "LS-210". The price is that it runs once per candidate, per query.

Here is a runnable toy version. I ran it as written; it uses fastembed for both models. The output was identical before and after reranking, which is a fair reminder that a reranker only helps when the first stage gets something wrong.

import time
import numpy as np
from fastembed import TextEmbedding
from fastembed.rerank.cross_encoder import TextCrossEncoder

chunks = [
    "Kestrel LS-210 Ultrasonic level sensor > Calibration\nRe-zero the LS-210 every 365 days.",
    "Kestrel LS-310 Ultrasonic level sensor > Calibration\nRe-zero the LS-310 every 90 days.",
    "Kestrel LS-210 Ultrasonic level sensor > Maintenance\nReplace the filter element of the unit after 2000 hours of operation.",
    "Kestrel Calibration Service FAQ > Intervals\nMost Kestrel devices should be recalibrated once a year.",
    "Kestrel Warranty Policy > Coverage\nThe standard Kestrel warranty lasts 12 months from the date of shipment.",
]
question = "How frequently must I recalibrate the LS-310?"

embedder = TextEmbedding("BAAI/bge-small-en-v1.5")
reranker = TextCrossEncoder("Xenova/ms-marco-MiniLM-L-6-v2")

vecs = np.array(list(embedder.embed(chunks)))
vecs /= np.linalg.norm(vecs, axis=1, keepdims=True)
q = np.array(next(iter(embedder.query_embed(question))))
q /= np.linalg.norm(q)

# stage 1: cheap retrieval of a wide pool
pool = list(np.argsort(-(vecs @ q))[:4])
print("stage 1:", [int(i) for i in pool])

# stage 2: the cross-encoder reads question and chunk together
t = time.perf_counter()
scores = list(reranker.rerank(question, [chunks[i] for i in pool]))
ms = (time.perf_counter() - t) * 1000
final = [pool[i] for i in np.argsort(-np.array(scores))]
print("stage 2:", [int(i) for i in final], f"({ms:.0f} ms for {len(pool)} candidates)")
print(chunks[final[0]])

stage 1: [1, 3, 0, 2]
stage 2: [1, 3, 0, 2] (149 ms for 4 candidates)
Kestrel LS-310 Ultrasonic level sensor > Calibration
Re-zero the LS-310 every 90 days.

How I measured it

• Set A (synthetic, mine): 22 Markdown files (16 invented product manuals with look-alike models, plus 6 policy files), 213 heading-aware chunks of up to 600 characters, 245 templated questions (181 paraphrases, 64 identifier questions such as error codes).
• Set B (real, public domain): the US Constitution from Project Gutenberg, 60 chunks, 32 hand-written questions.
• Hit rule: a chunk counts only if it is from the right file and holds the complete answer sentence. The reranker sees the chunk text including its heading path.
• First stage: BAAI/bge-small-en-v1.5 through fastembed 0.9.0 (the cache folder it downloaded is named bge-small-en-v1.5-onnx-Q, so it serves a quantized ONNX build). "Hybrid" means dense plus BM25 merged by reciprocal rank fusion.
• Rerankers: Xenova/ms-marco-MiniLM-L-6-v2 (small; the name suggests training on MS MARCO ranking data, which I didn't verify) and BAAI/bge-reranker-base (larger). Both are in fastembed's list of supported cross-encoders.
• Pool: the first-stage top 5, 10 or 20 candidates, reranked, then scored.

Set A is a stress test I built to be confusing, and its questions come from a few templates. Set B has 32 questions, so one question is 3.1 points. Read set A as "what happens when the first stage is bad" and set B as "what happens when it's already good".

Does a reranker improve recall@k?

Set A, 245 questions, recall in percent. Baseline rows have no reranker.

The R@1 gain from dense pool 20 (53.1 to 85.3) has a 95 percent paired-bootstrap interval of +26.1 to +38.8 points, so it is not noise, though the questions are templated and the true uncertainty for your corpus is larger.

Set B, 32 questions, recall in percent.

Two different stories here. Reranking dense results on set B changed one question at recall@1 (interval 0 to +9.4 points). But hybrid fusion had hurt set B (62.5 at R@1, as the hybrid post shows), and the reranker pulled it back to 87.5, level with reranked dense. In other words, a reranker also made the pipeline robust to a poor merge.

When does a reranker pay off?

Putting both sets together, I'd use these rules.

1. It pays when the answer is in the pool but not at the top. On set A the dense top 20 contained the answer for 99.2 percent of questions (top 10: 92.2, top 5: 84.5) but rank 1 only 53.1 percent of the time. That gap, about 46 points, is the headroom a reranker can use. Compute your own pool recall and your own recall@1 before buying anything.
2. It does not pay when the answer isn't in the pool. A reranker reorders; it cannot retrieve. With a pool of 5, reranking raised dense R@1 from 53.1 to 77.1 but R@5 stayed at 84.5, because nothing outside the top 5 was ever seen.
3. It pays most on look-alike content and identifiers. Within set A, dense plus MiniLM at pool 20 went from 54.7 to 81.8 percent R@1 on paraphrase questions and from 48.4 to 95.3 on identifier questions (R@3: 76.2 to 95.0 and 68.8 to 100.0). The cross-encoder is good at "does this chunk talk about the exact thing I named".
4. It pays little when the first stage is already good (set B dense: +3.1 points at R@1, one question).
5. It protects you from a mediocre merge. If you add BM25 and your fused ranking is noisy, a reranker on top cleaned it up in both sets.

A bigger reranker was worse here

I also ran BAAI/bge-reranker-base on every third set-A question (82 questions) and all of set B. Same candidates, pool 20, recall in percent:

On set B the two rerankers were within a question or two of each other (bge-base R@1 90.6 against MiniLM's 87.5 for both candidate sources). I don't know why the larger model lost on set A. I didn't investigate input truncation, scoring scale or the templated text, so please don't conclude that "small beats big" in general. The takeaway is practical: try the cheap model first and measure.

What does reranking cost in latency?

Reranker time per query for the top-N candidates, measured on 4 vCPU of a shared cloud VM with every ONNX model restricted to one thread. CPU time and wall time agreed within about 2 percent; the load average was around 3 at the start of the run (the machine was shared with other jobs, which is why I report CPU time too). Chunks averaged 441 characters including the heading prefix.

For comparison, embedding the query and searching the 213 chunks took about 81 ms of CPU in the same single-thread setup, so reranking 20 candidates with the small model costs roughly 15 times the first stage. That is the real price: a pipeline that answers in 80 ms retrieval can become one that spends over a second reranking. These numbers are slower than I'd expect on a modern laptop with all cores available, so measure on your hardware; I'd trust the ratios (cost grows linearly with candidates, the larger model is about seven times slower) more than the absolute values.

If latency matters, the middle row is the sweet spot from these measurements: hybrid candidates, pool 10, small reranker gave R@3 of 95.9 percent for about half a second.

Should you add a reranker?

• Measure first. Write 30 questions with known answer chunks. Compute recall@1 and recall@3 with no reranker, and the share of questions where the answer is in the top 10 and top 20.
• If recall@20 is high and recall@1 is low, add one. If recall@20 is low, fix chunking or retrieval first (chunking and hybrid search are the two measured here).
• Rerank 10 to 20 candidates, keep 3 to 5. More candidates cost time linearly and, in my data, gave small extra gains after 10.
• Budget the latency and consider caching reranker scores for repeated questions.
• Don't skip evaluation. Building golden test sets covers how to keep the question set honest, and prompt engineering for RAG pipelines covers what happens after retrieval. For the basics, see how RAG works.

What I did not test

• Hosted reranking APIs, LLM-based rerankers, and other open cross-encoders (I ran two).
• GPU inference, batching, or multi-threaded CPU latency.
• Corpora larger than 213 chunks, or longer chunks than 600 characters.
• Answer quality from an LLM after reranking. These are retrieval numbers only.
• Why bge-reranker-base underperformed on set A.