Hybrid Search in Python: BM25 Plus Embeddings With Rank Fusion, Measured

A user pastes E-4417 into your search box and the dense retriever returns three chunks about other error codes. That is the textbook failure of embeddings, and hybrid search is the textbook fix: run BM25 and a dense retriever side by side, then merge the two ranked lists. I measured it on 2026-10-08. BM25 found the right chunk for 64 of 64 identifier questions in the top 3 while dense found 68.8 percent of them. But the merged list was worse than dense at the top of the list on paraphrased questions (it only helped at recall@10), and clearly worse on a real document, so "just turn on hybrid" is not the whole answer.

Everything below uses free local models and runs on a laptop CPU. The test sets and the full question list are described in the chunking comparison; I'll summarise here.

The test, and how small it is

• Set A (synthetic, mine): 22 Markdown files (16 invented product manuals plus 6 policy files), chunked heading-aware at 600 characters into 213 chunks. 245 questions: 181 paraphrases ("How frequently must I recalibrate the LS-310?") and 64 identifier questions ("What does error E-4417 indicate?", part numbers, firmware versions). Questions are templated, so the variety is lower than the count suggests.
• Set B (real, public domain): the US Constitution from Project Gutenberg, 60 chunks, 32 hand-written natural-language questions with no identifiers.
• Hit rule: a chunk counts only if it is from the right file and contains the complete answer sentence.
• Dense model: BAAI/bge-small-en-v1.5 via fastembed 0.9.0 (a quantized ONNX build, judging by its cache folder name). BM25: the rank_bm25 package (BM25Okapi, default parameters) with a lowercase \w+ tokenizer. Fusion: reciprocal rank fusion with the usual constant of 60 over each retriever's top 100.

Set B is one document with 32 questions, so one question is 3.1 points.

The hybrid search script

import re
import numpy as np
from fastembed import TextEmbedding
from rank_bm25 import BM25Okapi

embedder = TextEmbedding("BAAI/bge-small-en-v1.5")
tok = lambda s: re.findall(r"\w+", s.lower())


class HybridIndex:
    def __init__(self, texts):
        self.texts = texts
        v = np.array(list(embedder.embed(texts)))
        self.vecs = v / np.linalg.norm(v, axis=1, keepdims=True)
        self.bm25 = BM25Okapi([tok(t) for t in texts])

    def dense(self, q, k=100):
        qv = np.array(next(iter(embedder.query_embed(q))))
        qv = qv / np.linalg.norm(qv)
        return list(np.argsort(-(self.vecs @ qv))[:k])

    def sparse(self, q, k=100):
        return list(np.argsort(-self.bm25.get_scores(tok(q)))[:k])

    def hybrid(self, q, k=10, bm25_weight=1.0, c=60):
        score = {}
        for ranks, w in ((self.dense(q), 1.0), (self.sparse(q), bm25_weight)):
            for r, i in enumerate(ranks):
                score[i] = score.get(i, 0.0) + w / (c + r + 1)
        return [i for i, _ in sorted(score.items(), key=lambda x: -x[1])][:k]


idx = HybridIndex(["Error E-4417 means the flash memory has failed its self-test.",
                   "Error E-2208 is raised when the signal cable is disconnected.",
                   "Re-zero the sensor every 90 days."])
print([idx.texts[i] for i in idx.hybrid("What does E-4417 mean?", k=1)])

That is the same logic as my experiment code (idx.dense, idx.sparse and the rrf function with weights), condensed into one class, and I ran this exact block separately from the experiment. The demo prints ['Error E-4417 means the flash memory has failed its self-test.'], which proves little beyond "it runs", since the toy index has three documents. Reciprocal rank fusion needs only ranks, so you never have to make BM25 scores (unbounded) comparable to cosine similarities.

One gotcha: the tokenizer splits E-4417 into e and 4417. That works here because 4417 is a rare token with a high IDF, and e is harmless. I did not test identifiers whose digits are common across your corpus, such as SKUs that share long prefixes, so check yours.

Where does each retriever win?

Recall in percent, heading-aware 600-character chunks. "Fused" is equal-weight RRF.

Set A, identifier questions (n=64)

Set A, paraphrase questions (n=181)

Set B, natural-language questions on a real document (n=32)

Reading across the tables:

• Exact identifiers belong to BM25. It was near perfect at rank 1 (98.4). Dense was wrong at rank 3 for almost a third of them. Fusing recovered most of the loss but still left R@3 at 89.1 against BM25's 100.
• Paraphrases belong to dense. BM25 got 16.6 percent at rank 1 on paraphrases because the question and the document use different words ("recalibrate" against "re-zero").
• Fusion with equal weights is a compromise, and it can cost you. On set B it lost 15.6 points at R@3 against dense alone. A paired bootstrap on the 32 questions put the difference between roughly -28 and -3 points at 95 percent, so it's not just luck of the draw. On the mixed set A, the R@3 gain over dense (74.3 to 75.9) was inside noise (interval -4.5 to +7.8 points), while R@10 improved reliably (92.2 to 98.0; interval +2.4 to +9.4).

That last point is the practical one: fusion is good at putting the answer somewhere in the top 10, which is what you want when a reranker will sort the list afterwards. See the reranking measurement for that combination.

Real misses from the run

From the printed examples (illustrative selection, first three of each kind):

The left column is dense retrieval confusing look-alike model numbers, which a keyword match settles. The right column is a vocabulary mismatch: the Constitution says "attained to the Age of twenty five Years", not "how old".

Do the weights matter?

Yes. I swept the BM25 weight (dense fixed at 1) and recorded R@3. These sweeps use the same questions I report on, so they are optimistic. Treat the pattern, not the numbers, as the finding.

No weight is best everywhere. Identifiers want BM25 loud; prose wants it quiet. The mixed set A peaks around 0.5, but that peak depends on my 26 percent identifier share (64 of 245). If half your traffic is part numbers, your best weight is different.

A natural idea is to route: detect identifier-shaped tokens in the query and weight BM25 higher only then. I haven't tested that, so treat it as a hypothesis worth a small experiment on your own query log.

What does hybrid cost in latency?

Measured on 82 queries from set A (213 chunks) on a 4-core shared cloud VM, with the ONNX models restricted to one thread. The machine was busy with other jobs (load average about 11 on 4 cores), so I report CPU time as well as wall time; the two were within a few percent.

The query embedding dominates, and you pay it already. BM25 adds about a millisecond at this size. rank_bm25 scores every chunk on each query, so that number grows linearly with the corpus; at large scale you'd want an inverted index, which I did not test. Treat these as order-of-magnitude figures for this machine.

Should you add BM25 to your pipeline?

1. Look at your queries first. If users type error codes, SKUs, function names, ticket numbers or legal clause numbers, add BM25. If they only ask conversational questions, my real-text test says leave dense alone, or give BM25 a low weight.
2. Evaluate the two query types separately. A single blended recall number hid all of this. Write 20 identifier queries and 20 paraphrases, and report both.
3. Start with weight 1, then look at R@10 and your reranker. Equal-weight fusion improved the top-10 on set A even where it hurt the top 3.
4. Keep an eye on chunk identity. Part of the dense failure on identifiers was look-alike manuals. Carrying the heading into each chunk helped both retrievers; that is measured in the chunking post.

For the concepts behind both retrievers, see how RAG works and the RAG lesson. If you want a first working pipeline before any of this, the PDF chatbot walkthrough builds one in 55 lines.

What I did not test

• Larger corpora, other embedding models, other BM25 implementations or parameters, and stemming or stopword removal.
• Messy real identifiers (SKUs with shared prefixes, mixed-case codes, typos).
• Score-based fusion (normalising and adding scores) against rank fusion.
• Answer quality from an LLM. These are retrieval numbers.