You pasted a 90-page contract into a model with a huge context window, asked about the termination clause, and got an answer that sounded right and was wrong. The text was in the prompt. The model just didn't use it well. That gap is what people now call context rot, and the fixes are mostly boring: send less, order it deliberately, and measure.
Below is what Chroma's report says (I read it on 2026-10-08), five fixes with a decision rule for each, and a small script that builds position and size test prompts for your own model. I did not run any model for this post, so I make no claims about how any specific model scores. The script's output here is just the prompt grid.
What does the Chroma context rot report actually say?
The report is "Context Rot: How Increasing Input Tokens Impacts LLM Performance" by Kelly Hong, Anton Troynikov and Jeff Huber, a Chroma technical report dated July 14, 2025, at trychroma.com/research/context-rot. The writer and the editor both read it through a summarising fetch tool, not a browser (re-checked 2026-10-09), so check any quote you plan to reuse against the original. What it reports:
- 18 models across Anthropic, OpenAI, Google and Alibaba families, though not every model appears in every experiment.
- Needle-in-a-haystack, extended. They varied how similar the needle is to the question, added distractors, varied topic overlap with the haystack, and compared coherent text with sentence-shuffled text. Lower needle-question similarity degraded faster as length grew. Even a single distractor reduced performance versus the baseline, and four compounded it.
- A surprise on structure. In their words, structural coherence consistently hurt performance: shuffled haystacks did better across all 18 models. They don't claim to know why.
- LongMemEval. 306 prompts averaging about 113k tokens in full form, versus focused versions of about 300 tokens. Every model did better on the focused version. Thinking modes helped but did not close the gap.
- Repeated words. Models had to copy a repeated sequence with one unique word inserted. Performance fell with length across all models, with odd failures (refusals, random output) in some. Accuracy was highest when the unique word sat near the start, especially at longer lengths.
- Position. In the needle-in-a-haystack task the authors report no notable variation across the 11 needle positions they tried. So the report does not show a "lost in the middle" penalty for that task.
Their headline sentence is that models "do not use their context uniformly" and that performance degrades with input length across all experiments. They don't offer formal recommendations; they point to careful construction of the context, which is the idea behind context engineering.
Limits worth knowing: these are synthetic tasks, the models are from mid-2025 and have since been replaced, and the report doesn't tell you where your own task falls. Use it as a reason to test, not as a number to quote.
Fix 1: trim before you send
The only token that can't distract the model is the one you didn't send. Chroma's distractor result is the practical hook: near-miss passages hurt, so a bloated prompt full of almost-relevant text is worse than a short one.
Do this: list every block in your prompt, delete anything the task doesn't need, and then test again. The context window budget post has a worksheet and an ablation script for the "does this block earn its tokens" question. Rule: if removing a block doesn't change your eval score, it goes.
Fix 2: put the question and instructions where they can't be missed
Order is cheap to change. A layout I'd start from, as a hypothesis and not a finding:
[role and rules] short, stable
[documents / evidence] long, in tagged blocks
[question + output format] last, restated
Restating the question after the documents costs a few tokens. The older long-context strategies post covers anchoring at both ends and explicit referencing. Chroma's needle test showed no position effect, so don't assume "middle is worst" for your model. Run the probe below and see.
Fix 3: retrieve instead of stuffing
If only 3 of 300 pages matter, don't send 300. Retrieve the top passages, send those, and keep the source IDs so the answer can cite them. The trade is explicit: a retriever that misses the passage fails silently, while a long prompt at least contains it. Decide with data. The comparison in RAG vs long context vs compaction walks through the trade-offs, and for the build, see how RAG works.
Fix 4: summarise old context, but keep the facts that bite
For long conversations or agent runs, replace old turns with a summary. Summaries lose detail, so keep a pinned block of facts that must survive: IDs, decisions, constraints, user preferences. A template (untested, adapt it):
Summarise the conversation so far for a future assistant.
Keep verbatim: names, IDs, numbers, dates, decisions, constraints.
List open questions separately. Drop pleasantries and abandoned ideas.
Do not add anything that was not said.
Then check the summary against the source for two or three facts you know matter. Claude context compaction shows the managed version of this.
Fix 5: split the job
Chroma's focused-versus-full gap suggests a two-step pattern: first, a cheap call extracts the relevant snippets from the big input. Second, a short call answers using only those snippets. You get a prompt close to the 300-token case, and the extraction step is easy to eval on its own. The price is a second call and a place for the extractor to drop evidence, so test recall of the extraction step separately.
How do I measure position and size effects on my own task?
Here's a probe. It builds a haystack of filler sentences at a target size, inserts one fact at a chosen relative position, and asks about it. The dry run only prints the grid. The --live mode posts to Anthropic's Messages API and needs ANTHROPIC_API_KEY and a model name; I did not run live mode, so treat that part as unrun. Token counts use a crude characters-divided-by-four estimate, not a tokenizer.
import argparse, json, os, random, urllib.request
FILLER = [
"The quarterly report was filed on time and reviewed by the team.",
"Shipping volumes in the northern region rose slightly over the period.",
"A new vendor contract was drafted but has not yet been signed.",
"Support tickets were triaged each morning by the on-call engineer.",
"The migration checklist was updated after the Tuesday review meeting.",
"Office hours moved to the second floor during the renovation.",
"Inventory counts were reconciled against the warehouse system.",
"The design team shared updated mockups in the project channel.",
]
NEEDLE = "The deploy freeze for the billing service ends on 14 March."
QUESTION = "When does the deploy freeze for the billing service end? Answer with the date only."
def est_tokens(text):
return len(text) // 4
def build(target_tokens, position, seed=0):
rng = random.Random(seed)
sentences = []
while est_tokens(" ".join(sentences)) < target_tokens:
sentences.append(rng.choice(FILLER))
idx = int(position * len(sentences))
sentences.insert(idx, NEEDLE)
return " ".join(sentences) + "\n\n" + QUESTION
def ask(prompt, model):
body = json.dumps({"model": model, "max_tokens": 50,
"messages": [{"role": "user", "content": prompt}]}).encode()
req = urllib.request.Request("https://api.anthropic.com/v1/messages", body, {
"x-api-key": os.environ["ANTHROPIC_API_KEY"],
"anthropic-version": "2023-06-01",
"content-type": "application/json"})
with urllib.request.urlopen(req) as r:
return json.load(r)["content"][0]["text"]
p = argparse.ArgumentParser()
p.add_argument("--live", action="store_true")
p.add_argument("--model", default="")
p.add_argument("--trials", type=int, default=5)
a = p.parse_args()
print("size_target position est_tokens needle_char_offset")
for size in [1000, 8000, 32000, 100000]:
for pos in [0.0, 0.25, 0.5, 0.75, 1.0]:
prompt = build(size, pos)
off = prompt.index(NEEDLE)
line = f"{size:>10} {pos:>8} {est_tokens(prompt):>10} {off:>18}"
if a.live:
hits = sum(("14 March" in out or "March 14" in out)
for out in (ask(build(size, pos, seed=t), a.model) for t in range(a.trials)))
line += f" correct {hits}/{a.trials}"
print(line)
Real output of the dry run on 2026-10-08 (five selected rows of the 20):
size_target position est_tokens needle_char_offset
1000 0.0 1044 0
1000 0.25 1044 986
1000 0.5 1044 1972
32000 0.5 32042 64172
100000 1.0 100051 400062
Filler like this is easy mode: the needle is unlike everything around it, and Chroma found low similarity and distractors are where things break. For a harder, more honest test, swap the filler for real documents from your domain and add near-miss sentences ("The deploy freeze for the payments service ends on 21 March."). Then run each cell several times and compare the grid, not single answers.
What should I do on Monday?
- Pick 20 to 50 real questions with known answers.
- Score three variants: full context, retrieved context, and summarised context.
- Run the position probe on your own documents at two or three sizes.
- Keep the variant that wins at your real input size, and re-run when you change models. Tie this into your regression suite; the golden test set guide covers building one.
If the full-context variant wins, great, and you've saved yourself an engineering project. If it loses, you now have a number that justifies the extra work.
