Prompt Caching for Long-Running Agent Loops: Cache Hit Checks and Prefix Breaks

An agent loop that costs three times what you budgeted usually isn't looping more than expected. It's re-reading the same 8,000 tokens of tools and instructions on every step at full price, because something small changed in front of the cache breakpoint. You turned caching on and never saw it work.

This post is about keeping prompt caching working inside a long-running agent loop on Anthropic's API, and noticing within one turn when it stops. You'll get a list of what breaks the cached prefix, a simulator that reproduces each break with real printed output, a check you can paste into your loop, and the API's own cache diagnostics feature. If you only need the basics of what caching costs and how to switch it on, read prompt caching in the Claude API and the Claude API bill checklist first. This post starts where those stop: step 7 of a loop, when the numbers look wrong.

The vendor facts here come from Anthropic's prompt caching, cache diagnostics and tool use with prompt caching pages, fetched with curl and converted to text on 2026-10-09. The simulator is mine, built from what those pages say. It is a model of the documented rules, not the real service, and it never called any API.

How does the cache decide whether a request hits?

Four rules from the docs explain almost every surprise.

1. The prefix is cumulative and ordered. The cache follows tools, then system, then messages. A change at one level invalidates that level and everything after it.
2. Writes happen only at breakpoints. A cache_control marker writes one entry: a hash of everything up to and including that block. Nothing is written for earlier positions.
3. Reads look backwards for earlier writes. On each request the system checks the hash at your breakpoint, then walks back one block at a time, up to 20 positions, looking for an entry a previous request wrote. It does not find stable content you never marked.
4. Entries expire. The default lifetime is 5 minutes, refreshed each time the entry is read. A 1-hour option exists at a higher write price.

Pricing multipliers from the same page: a 5-minute write costs 1.25 times the base input price, a 1-hour write costs 2 times, and a read costs 10% of base (5% on Claude Opus 5.5 and Sonnet 5.5, 2.5% on Fable 5.1 and Mythos 5.1, as listed on the page). I use 10% below and have not put dollar prices in this post, because the numbers that matter for your bill are ratios.

Two practical notes before the failure list. Concurrent requests don't help each other, since an entry only becomes available after the first response begins. And a prompt shorter than the model's minimum cacheable length isn't cached and raises no error, so the only sign is both cache fields being 0. The page lists 512 tokens for the 5.5 generation models, with other values for older ones, so check the table for your model.

What breaks the prefix in a long agent loop?

This is the checklist, ordered roughly by how often a loop hits each one. The first column gives the failure, the second what the docs say happens.

Two details from the invalidation table are easy to miss. Toggling images in or out of the prompt anywhere invalidates the messages cache, so an agent that sometimes attaches a screenshot will see a miss on those turns. And on models that support it, you can add a new system instruction mid-conversation by appending a role: "system" message to messages instead of editing the top-level system field, which leaves the cached prefix intact. The page lists which models support that, so check before relying on it.

Simulating prefix breaks

Reading about breaks is one thing. Seeing the hit rate fall is another. The script below models the rules above: cumulative SHA-256 hashes per block, writes only at breakpoints, a 20-position lookback, a TTL that refreshes on a hit, and cost in base-input-token units (write 1.25 or 2.0, read 0.10).

What it leaves out: the minimum-length rule, the collapsing of tool-call runs into one position, per-model read prices and everything about the real service's internals. Treat the percentages as illustrations of the documented rules, not as predictions for your traffic. There is no random number generator in it, so output is identical on every run. I ran it with Python 3.13 on 2026-10-09 (python3 -I cache_sim.py).

"""Toy simulator of the prompt-cache behaviour described on Anthropic's prompt caching page."""
import hashlib

LOOKBACK = 20
WRITE_5M, WRITE_1H, READ = 1.25, 2.0, 0.10  # multipliers of the base input price


class Cache:
    def __init__(self):
        self.entries = {}  # prefix hash -> (expiry_seconds, ttl_seconds)

    def request(self, blocks, breakpoints, now, ttl=300):
        """blocks: list of (text, tokens). breakpoints: indices of blocks carrying cache_control."""
        hashes, running = [], hashlib.sha256()
        for text, _ in blocks:
            running.update(text.encode())
            running.update(b"\x00")
            hashes.append(running.copy().hexdigest())
        tokens_to = [sum(t for _, t in blocks[: i + 1]) for i in range(len(blocks))]
        hit_pos = -1
        for bp in breakpoints:
            for pos in range(bp, max(bp - LOOKBACK, -1), -1):
                entry = self.entries.get(hashes[pos])
                if entry and entry[0] > now:
                    hit_pos = max(hit_pos, pos)
                    self.entries[hashes[pos]] = (now + entry[1], entry[1])  # a hit refreshes the TTL
                    break
        last_bp = max(breakpoints) if breakpoints else -1
        for bp in breakpoints:
            if hashes[bp] not in self.entries or self.entries[hashes[bp]][0] <= now:
                self.entries[hashes[bp]] = (now + ttl, ttl)
        read = tokens_to[hit_pos] if hit_pos >= 0 else 0
        created = (tokens_to[last_bp] - read) if last_bp >= 0 else 0
        total = tokens_to[-1]
        return {"read": read, "created": created, "input": total - read - created, "total": total}


def cost(u, ttl=300, read_mult=READ):
    write = WRITE_1H if ttl == 3600 else WRITE_5M
    return u["input"] + u["created"] * write + u["read"] * read_mult


def build(turn, tools, system, extra_per_turn=2, big_turn=None, edit_turn=None, stamp=False, user_last=False):
    """Blocks for one turn of an append-only agent loop."""
    sys_blocks = [(f"{s} [now=T{turn}]" if stamp and i == 0 else s, n) for i, (s, n) in enumerate(system)]
    static = [(t, n) for t, n in tools] + sys_blocks
    msgs = []
    for k in range(1, turn + 1):
        count = 25 if (big_turn == k) else extra_per_turn
        for j in range(count):
            text = f"turn{k}-block{j}"
            if edit_turn is not None and turn >= edit_turn and k == 1 and j == 0:
                text += "-trimmed"
            msgs.append((text, 150 if j % 2 == 0 else 600))
    if user_last:  # a per-request block (timestamp plus the user message) sits last
        msgs.append((f"now=T{turn} user message {turn}", 100))
    return static + msgs


TOOLS = [("tool:search_orders", 1500), ("tool:decide_refund", 1500)]
SYSTEM = [("system: support agent rules", 4000)]


def run(name, turns=6, gaps=None, bp_mode="last", ttl=300, tools_swap_turn=None, **kw):
    cache, rows, total_cost, total_plain = Cache(), [], 0.0, 0.0
    now = 0
    for turn in range(1, turns + 1):
        now += (gaps or {}).get(turn, 30)
        tools = list(TOOLS)
        if tools_swap_turn and turn == tools_swap_turn:
            tools = tools[::-1]  # same tools, different order
        blocks = build(turn, tools, SYSTEM, **kw)
        if bp_mode == "last":
            bps = [len(blocks) - 1]
        elif bp_mode == "stable":
            bps = [len(blocks) - 2]  # the last block before the per-request block
        elif bp_mode == "last+tools":
            bps = sorted({len(blocks) - 1, len(tools) - 1})
        elif bp_mode == "last+prev":
            prev_len = len(build(turn - 1, tools, SYSTEM, **kw)) if turn > 1 else 0
            bps = sorted({len(blocks) - 1} | ({prev_len - 1} if prev_len else set()))
        u = cache.request(blocks, bps, now, ttl)
        total_cost += cost(u, ttl)
        total_plain += u["total"]
        pct = 100 * u["read"] / u["total"]
        rows.append(f"  turn {turn} t={now:>4}s blocks={len(blocks):>2} read={u['read']:>6} created={u['created']:>6} input={u['input']:>5} hit={pct:5.1f}%")
    print(f"{name}  (breakpoints: {bp_mode}, ttl={ttl}s)")
    print("\n".join(rows))
    print(f"  cost = {total_cost:,.0f} base-input-token units vs {total_plain:,.0f} uncached ({100 * total_cost / total_plain:.0f}%)\n")
    return total_cost / total_plain


if __name__ == "__main__":
    results = {}
    results["S1 stable, append-only"] = run("S1 stable, append-only")
    results["S2 timestamp in system"] = run("S2 timestamp in system prompt", stamp=True)
    results["S2b same, tools breakpoint"] = run("S2b same, plus a breakpoint on the last tool", stamp=True, bp_mode="last+tools")
    results["S3 breakpoint on varying block"] = run("S3 per-request block last", user_last=True)
    results["S3b fixed"] = run("S3b breakpoint moved before the per-request block", user_last=True, bp_mode="stable")
    results["S4 tools reordered at turn 4"] = run("S4 tool order changes at turn 4", tools_swap_turn=4)
    results["S5 25 blocks in turn 4"] = run("S5 one turn adds 25 blocks", big_turn=4)
    results["S5b second breakpoint"] = run("S5b plus a breakpoint on the previous turn's last block", big_turn=4, bp_mode="last+prev")
    results["S6 6 min gap, 5m TTL"] = run("S6 6 minute gap before turn 4", gaps={4: 360})
    results["S6b 1h TTL"] = run("S6b same gap, 1 hour TTL", gaps={4: 360}, ttl=3600)
    results["S7 old block edited at turn 4"] = run("S7 an old block is trimmed from turn 4", edit_turn=4)
    print("summary (cost as % of uncached)")
    for k, v in results.items():
        print(f"  {k:<34} {100 * v:5.1f}%")

The loop has two tool definitions (1,500 tokens each), one 4,000-token system block, and each turn appends an assistant block of 150 tokens and a tool-result block of 600. Six turns, 30 seconds apart unless a scenario says otherwise. The last block of each request carries the breakpoint, as with automatic caching in a growing conversation.

The healthy baseline

S1 stable, append-only  (breakpoints: last, ttl=300s)
  turn 1 t=  30s blocks= 5 read=     0 created=  7750 input=    0 hit=  0.0%
  turn 2 t=  60s blocks= 7 read=  7750 created=   750 input=    0 hit= 91.2%
  turn 3 t=  90s blocks= 9 read=  8500 created=   750 input=    0 hit= 91.9%
  turn 4 t= 120s blocks=11 read=  9250 created=   750 input=    0 hit= 92.5%
  turn 5 t= 150s blocks=13 read= 10000 created=   750 input=    0 hit= 93.0%
  turn 6 t= 180s blocks=15 read= 10750 created=   750 input=    0 hit= 93.5%
  cost = 19,000 base-input-token units vs 57,750 uncached (33%)

Turn 1 only writes. Every later turn reads the whole previous request and writes just its two new blocks. This is the shape to look for in your own logs.

A timestamp in the system prompt

S2 timestamp in system prompt  (breakpoints: last, ttl=300s)
  turn 1 t=  30s blocks= 5 read=     0 created=  7750 input=    0 hit=  0.0%
  turn 2 t=  60s blocks= 7 read=     0 created=  8500 input=    0 hit=  0.0%
  ...
  turn 6 t= 180s blocks=15 read=     0 created= 11500 input=    0 hit=  0.0%
  cost = 72,188 base-input-token units vs 57,750 uncached (125%)

Worse than not caching at all: 125% of the uncached cost, because you pay the 25% write premium and never collect a read. In the simulator, adding a second breakpoint on the last tool recovers the tools portion only:

S2b same, plus a breakpoint on the last tool  (breakpoints: last+tools, ttl=300s)
  turn 2 t=  60s blocks= 7 read=  3000 created=  5500 input=    0 hit= 35.3%
  ...
  cost = 54,938 base-input-token units vs 57,750 uncached (95%)

That matches the docs: tool definitions sit above the system prompt in the hierarchy, so a system change leaves the tools entry valid. The proper fix is moving the timestamp out of the system prompt and into the first user message, after the breakpoint.

A breakpoint on the per-request block

Here the last block is a timestamp plus the user message, and the breakpoint sits on it. This is the mistake the docs call out by name, and automatic caching falls into it when the last block is the one that changes.

S3 per-request block last  (breakpoints: last, ttl=300s)
  ...
  cost = 72,938 base-input-token units vs 58,350 uncached (125%)

S3b breakpoint moved before the per-request block  (breakpoints: stable, ttl=300s)
  turn 2 t=  60s blocks= 8 read=  7750 created=   750 input=  100 hit= 90.1%
  ...
  cost = 19,600 base-input-token units vs 58,350 uncached (34%)

Same content, different breakpoint position, a 91-point swing in cost relative to uncached. With the fix, the 100 tokens of per-request text stay in input where they belong.

Tool order that flaps

At turn 4 the two tools arrive in the opposite order, as if a dictionary were serialised differently or a plugin loaded in a different sequence.

S4 tool order changes at turn 4  (breakpoints: last, ttl=300s)
  turn 3 t=  90s blocks= 9 read=  8500 created=   750 input=    0 hit= 91.9%
  turn 4 t= 120s blocks=11 read=     0 created= 10000 input=    0 hit=  0.0%
  turn 5 t= 150s blocks=13 read=  9250 created=  1500 input=    0 hit= 86.0%
  turn 6 t= 180s blocks=15 read= 10750 created=   750 input=    0 hit= 93.5%
  cost = 30,500 base-input-token units vs 57,750 uncached (53%)

One reordering costs a full rewrite. Look at turn 5: the order is back to normal, so the request matches the entry from turn 3 (9,250 tokens read), and the model pays again to re-write the two blocks from turn 4. If your tool order depends on a set or a dict you didn't sort, this pattern can repeat at random. Sort tools by name and serialise with sorted keys.

Twenty-five blocks in one turn

Turn 4 adds 25 blocks, say a fan-out of parallel searches that the simulator doesn't collapse into one position.

S5 one turn adds 25 blocks  (breakpoints: last, ttl=300s)
  turn 4 t= 120s blocks=34 read=     0 created= 18400 input=    0 hit=  0.0%
  cost = 41,818 base-input-token units vs 82,950 uncached (50%)

S5b plus a breakpoint on the previous turn's last block  (breakpoints: last+prev, ttl=300s)
  turn 4 t= 120s blocks=34 read=  9250 created=  9150 input=    0 hit= 50.3%
  cost = 31,180 base-input-token units vs 82,950 uncached (38%)

The final block is 25 positions past turn 3's write, outside the 20-position window. A second breakpoint on the previous turn's last block starts its own lookback window and lands on the existing entry. Four breakpoints are the maximum per request, so spend them on the places where the loop can jump: the end of the static prefix, the previous turn and the latest block.

A gap longer than the TTL

A slow tool call or a human approval takes 6 minutes before turn 4.

S6 6 minute gap before turn 4  (breakpoints: last, ttl=300s)
  turn 4 t= 450s blocks=11 read=     0 created= 10000 input=    0 hit=  0.0%
  cost = 29,638 base-input-token units vs 57,750 uncached (51%)

S6b same gap, 1 hour TTL  (breakpoints: last, ttl=3600s)
  turn 4 t= 450s blocks=11 read=  9250 created=   750 input=    0 hit= 92.5%
  cost = 27,625 base-input-token units vs 57,750 uncached (48%)

With a single long gap in six turns, the 1-hour TTL wins by a small margin even though every write costs 2 instead of 1.25. If the gaps were rarer, the extra write cost would take that back. The decision depends on how often your gaps exceed 5 minutes, so log the gap between consecutive calls for a day before you pick.

An old block rewritten

Turn 4 trims the first tool result, the way a context-pruning step might.

S7 an old block is trimmed from turn 4  (breakpoints: last, ttl=300s)
  turn 4 t= 120s blocks=11 read=     0 created= 10000 input=    0 hit=  0.0%
  turn 5 t= 150s blocks=13 read= 10000 created=   750 input=    0 hit= 93.0%
  cost = 29,638 base-input-token units vs 57,750 uncached (51%)

Because entries exist only at earlier last-block positions, there's nothing before the edited block for the lookback to find, so the edit costs a full rewrite. It then recovers on turn 5. If you prune or compact context, do it rarely and in one step rather than trimming a little every turn. Anthropic's docs navigation lists context editing and compaction pages, which I did not read for this post. The context rot post covers why long contexts degrade, which is the usual reason to prune at all.

Summary of all scenarios

summary (cost as % of uncached)
  S1 stable, append-only              32.9%
  S2 timestamp in system             125.0%
  S2b same, tools breakpoint          95.1%
  S3 breakpoint on varying block     125.0%
  S3b fixed                           33.6%
  S4 tools reordered at turn 4        52.8%
  S5 25 blocks in turn 4              50.4%
  S5b second breakpoint               37.6%
  S6 6 min gap, 5m TTL                51.3%
  S6b 1h TTL                          47.8%
  S7 old block edited at turn 4       51.3%

These are toy numbers on a six-turn loop with invented block sizes. The ordering is the point: two "always miss" cases cost more than not caching, and one-off events each cost about half of a healthy run over a short loop. In a long loop a one-off costs proportionally less, and a repeating break costs proportionally more.

How do I check cache hits inside the loop?

The response's usage object has cache_creation_input_tokens, cache_read_input_tokens and input_tokens, where input_tokens counts only what comes after the last breakpoint. Total input is the sum of all three. The function below turns a list of those usage dicts into plain-language problems. It runs without any API.

def check_loop(usages, min_hit=0.5):
    """usages: list of dicts with Anthropic-style usage fields, one per request, oldest first."""
    problems, prev_read = [], 0
    for i, u in enumerate(usages, start=1):
        read = u.get("cache_read_input_tokens", 0)
        made = u.get("cache_creation_input_tokens", 0)
        plain = u.get("input_tokens", 0)
        total = read + made + plain
        if read == 0 and made == 0:
            problems.append(f"turn {i}: nothing read or written; prompt may be under the model's minimum, or has no breakpoint")
        elif i > 1 and read == 0:
            problems.append(f"turn {i}: full cache miss ({made} tokens rewritten); the prefix changed or the entry expired")
        elif i > 1 and read < prev_read:
            problems.append(f"turn {i}: read fell from {prev_read} to {read}; something early in the prompt changed")
        if i > 1 and total and read / total < min_hit and read > 0:
            problems.append(f"turn {i}: hit ratio {read / total:.0%} is under {min_hit:.0%}")
        prev_read = read
    return problems

I ran it on usage numbers copied from the simulator output above (so these are simulated, not captured from the API):

healthy: OK
flapping tool order: 1 problem(s)
   turn 4: full cache miss (10000 tokens rewritten); the prefix changed or the entry expired
under minimum: 2 problem(s)
   turn 1: nothing read or written; prompt may be under the model's minimum, or has no breakpoint
   turn 2: nothing read or written; prompt may be under the model's minimum, or has no breakpoint

Note what it missed: turn 5 of the flapping run reads 9,250 tokens after a zero, so the "read went backwards" rule doesn't fire. A check on the previous turn's total would. Wire the function to an alert on any problem in production, and tune min_hit to your own steady state. Log the three usage fields with every call in your traces too; the OpenTelemetry tracing post shows where those fit as span attributes.

Can the API tell me why the cache missed?

Yes, on the Claude API. The cache diagnostics page describes a diagnostics object you pass on every request. On turn 1 you send previous_message_id: None to opt in. On later turns you send the id of the previous response, and the API compares the two requests and reports the first point where they diverged. The page says the old beta header is no longer required.

The possible cache_miss_reason.type values are model_changed, system_changed, tools_changed, messages_changed, previous_message_not_found and unavailable. The four *_changed types also carry cache_missed_input_tokens, an estimate of how many tokens fell after the divergence, which the page says to read as a magnitude and not a billing number. The response reports only the earliest divergence, so fix that first.

This is the loop from the docs, trimmed. It is unrun, since I had no API key in this session:

messages, prev_id = [], None
for i, user_message in enumerate(turns):
    messages.append({"role": "user", "content": user_message})
    r = client.beta.messages.create(
        model=MODEL,
        max_tokens=1024,
        cache_control={"type": "ephemeral"},
        system=SYSTEM,
        messages=messages,
        diagnostics={"previous_message_id": prev_id},
    )
    miss = r.diagnostics.cache_miss_reason if r.diagnostics else None
    if miss is not None:
        print(f"turn {i + 1}: {miss.type}")
    messages.append({"role": "assistant", "content": r.content})
    prev_id = r.id

Read the diagnostics result together with cache_read_input_tokens, as the page suggests. A *_changed reason with low reads is your bug. A null diagnostic with low reads means your requests matched but the entry was gone, so shorten gaps or use the 1-hour TTL. The limitations to know: it works on the Claude API only (not Bedrock or Google Cloud), fingerprints expire after a short period so consecutive requests must be close in time, and the previous request must be in the same workspace.

A pre-flight checklist for your loop

• Tools are in a fixed order and serialised with sorted keys.
• The system prompt is byte-stable. Dates, ids and names live in the first user message.
• The breakpoint sits on the last block that is identical across requests, not on the per-request block.
• History is append-only. Pruning and summarising happen at planned points.
• If a single turn can add 20 or more blocks, a second breakpoint sits on the previous turn's end.
• tool_choice, thinking, effort, model and image presence are constant, or breakpoints sit before the change.
• You know your gap distribution and have picked 5-minute or 1-hour deliberately.
• Usage fields are logged per call and the check above runs on them.
• Cache diagnostics is on in staging, and someone reads it when the hit rate drops.
• The prompt is above the model's minimum cacheable length, or you know it isn't.

Cache hits also depend on how you manage context overall. Token budgeting for agents covers capping a loop's spend, and writing tool descriptions for LLM agents covers freezing tool definitions so they don't churn your cache.

What I did not verify

• The simulator follows the rules as I read them on 2026-10-09. I did not test them against the live API, so real behaviour may differ in details such as how a hit refreshes entries at several breakpoints.
• The diagnostics loop is unrun, and field names come from the docs page as converted to text.
• The pricing multipliers and minimum lengths are from the prompt caching page on that date. Model lists and prices change, so check the live page and your provider's price list.
• OpenAI and Gemini caching work differently and are not covered here. See prompt caching strategies on Anthropic and OpenAI.