Semantic Caching for LLM Apps: Embedding Threshold Pitfalls

A semantic cache is only as good as the line you draw between "same question" and "similar question", and embeddings do not draw that line where you'd want it. I ran a threshold sweep on 30 support questions with two local embedding models. At every threshold, the cache either missed most real paraphrases or served wrong answers to near-identical questions. That is the pitfall, with numbers.

If you want the architecture (GPTCache, Redis, TTLs), read semantic caching with Redis and GPTCache. This post is about the one setting that decides whether the cache is safe.

The experiment

On 2026-10-30 I wrote 30 FAQ questions for an invented SaaS product. For each, I wrote:

• a paraphrase that should return the cached answer ("I forgot my password, how can I change it?")
• a near-miss that needs a different answer ("How do I reset my two-factor authentication?")

Both were embedded and matched against the 30 cached questions by cosine similarity, taking the top-1. A paraphrase counts as a hit only if top-1 is the right question. A near-miss counts as a false hit whenever its score clears the threshold, because nothing in the cache is correct for it. Models: BAAI/bge-small-en-v1.5 and sentence-transformers/all-MiniLM-L6-v2, both run locally through fastembed.

The data is small and written by me, so the exact percentages won't transfer. The shape of the problem will.

Result 1: the score ranges overlap

The medians for right and wrong matches sit within 0.01 (bge) and 0.06 (MiniLM) of each other. A threshold can't separate groups that overlap this much.

Result 2: the sweep

Counts out of 30 each. "Good hits" are correct paraphrase matches; "false hits" are near-misses served a cached answer.

Read the bge column at 0.90, the number many tutorials suggest: 6 good hits and 6 false hits. Half the cache hits would have been wrong. At 0.95 the false hits nearly vanish but so does the cache. For MiniLM, 0.90 is far too strict while 0.70 already lets 12 wrong answers through. The same number means different things on different models, so thresholds copied between blog posts (including this one) are a trap. Switching embedding models silently resets your safety margin.

Result 3: what the false hits look like

Highest-scoring near-misses on bge, each served the wrong cached answer:

Every one differs by a single decisive word: files versus size, web versus API, desktop versus mobile. Those words change the answer and barely move the vector. On MiniLM, the monthly versus annual refund pair scored 0.920 and "export to Trello" matched "import from Trello" at 0.866.

The misses go the other way. Genuine paraphrases that bge scored lowest: "Which region are your servers in?" for "Where is my data stored?" (0.624) and "Do you have an SLA for availability?" for the uptime question (0.668). Different words, same meaning, low score.

What to do about it

Nothing below was benchmarked here; these are design choices that follow from the failure shapes.

1. Start at a strict threshold and measure. Begin where false hits are zero on your labelled pairs, accept a low hit rate, and loosen only with evidence. A cache with a 10 percent hit rate and no wrong answers beats 40 percent with some.
2. Build the labelled set from real traffic. Take 50 to 100 pairs of questions that your users actually asked, label same-answer or different-answer, and run the sweep above again whenever you change the embedding model. The golden dataset approach applies directly.
3. Scope the lookup. Put tenant, plan tier, language and document version into the cache key so a hit can only come from the same scope. This kills the "annual versus monthly" class when the plan is known.
4. Add a cheap verifier on borderline scores. For hits in the band just above the threshold, compare the two questions with a small model or a rule (do they contain different numbers, product names or negations?) before serving. This costs a call, so apply it only to the band.
5. Don't cache what depends on state. Anything involving "my", dates, balances or live data should bypass the cache.
6. Log every hit with query, matched query and score, and review the lowest-scoring 20 hits a week. Short TTLs also cap the damage from a bad entry.

Exact-match caching and provider-side prompt caching have none of these failure modes, because they only reuse work, not answers. Reach for them first, and see the cost-cutting checklist for the order of operations. For picking an embedding model in the first place, the embedding models comparison is the better starting point.

The sweep, if you want to rerun it

The core is a normalised matrix product and a count per threshold:

import numpy as np
from fastembed import TextEmbedding

model = TextEmbedding("BAAI/bge-small-en-v1.5")

def unit(vecs):
    x = np.asarray(list(vecs), dtype=np.float32)
    return x / np.linalg.norm(x, axis=1, keepdims=True)

cached = unit(model.embed(cached_questions))
queries = unit(model.embed(incoming_questions))
sims = queries @ cached.T          # cosine similarity
best = sims.argmax(axis=1)
score = sims.max(axis=1)

for t in [0.70, 0.80, 0.85, 0.90, 0.95]:
    served = score >= t
    correct = served & (best == expected_index)
    print(t, "served", served.sum(), "correct", correct.sum())

Replace the lists with your own labelled pairs, and include near-misses on purpose. A test set made only of paraphrases will make every threshold look fine.