LLM API Rate Limits and Retries in Python: Backoff, Jitter and Idempotency

Your batch job hits a 429 at 2 a.m., retries instantly, hits another 429, and by morning it has burned its retries and dropped 300 documents. The fix for LLM API rate limits in Python is a small amount of code: retry only the errors that can succeed later, wait with exponential backoff plus jitter, respect Retry-After, stop at a deadline, and make the retried work safe to run twice.

This post gives you that helper in about 50 lines, and shows it running against a local mock server that I wrote to return 429 and 500 on demand. Everything marked "output" below is copied from runs on 2026-10-09. The mock is mine, not Anthropic's, so the timings show how the code behaves, not how any real API behaves. The error behavior I cite comes from Anthropic's errors page, read in full on 2026-10-09, but pages change, so check the exact wording there before you rely on it.

If you only want the limits themselves (tiers, TPM, RPM), start with the Claude API rate limits guide. For the wider reliability picture see the agent production checklist.

Which errors are worth retrying?

Retry what can change on its own. Don't retry what you caused.

Two details from that page matter more than the table. First, a 429 caused by a tier's monthly spend cap has no retry-after header and "keeps failing until access resumes", so an unbounded retry loop on 429 can spin for hours. Second, with streaming, an error can arrive after the API has already returned a 200, and then the normal status-code handling doesn't apply. Your stream reader needs its own error branch. I didn't test mid-stream errors here.

How should backoff and jitter work?

Backoff means each wait is longer than the last. Jitter means the wait is randomized so a crowd of clients doesn't move in lockstep. Three variants, using a base of 0.5 seconds and a cap of 8 seconds, with the exact numbers from my run:

delays for attempts 0..5 (base 0.5, cap 8):
  none  [0.5, 1.0, 2.0, 4.0, 8.0, 8.0]
  equal [0.41, 0.51, 1.28, 2.45, 6.95, 6.71]
  full  [0.32, 0.03, 0.55, 0.89, 5.89, 5.41]

"None" is the plain doubling schedule. "Equal" keeps half the delay fixed and randomizes the other half. "Full" picks anywhere between zero and the ceiling. The cap matters: without it, attempt 10 would wait about 8.5 minutes.

I default to full jitter because it spreads clients out most. The cost is that an unlucky attempt can wait almost nothing (0.03 seconds above), which is fine when the server is just briefly busy and pointless when it is down. That's why you also need an attempt limit and a deadline.

The retry helper

This is the whole thing. It needs httpx. It retries the statuses above plus timeouts and connection errors, treats Retry-After as a floor, and refuses to sleep past a total deadline:

import asyncio, random, time
import httpx

RETRY_STATUS = {408, 429, 500, 502, 503, 504, 529}


class GaveUp(Exception):
    pass


def backoff_delay(attempt, base=0.5, cap=8.0, jitter="full", rng=random):
    ceiling = min(cap, base * (2 ** attempt))
    if jitter == "full":
        return rng.uniform(0, ceiling)
    if jitter == "equal":
        return ceiling / 2 + rng.uniform(0, ceiling / 2)
    return ceiling  # no jitter


def parse_retry_after(resp):
    value = resp.headers.get("retry-after")
    if value is None:
        return None
    try:
        return max(0.0, float(value))   # seconds form only; HTTP-date form is ignored here
    except ValueError:
        return None


async def call_with_retry(send, *, max_attempts=5, base=0.5, cap=8.0, jitter="full",
                          max_elapsed=30.0, log=None, rng=random):
    start = time.monotonic()
    last = None
    for attempt in range(max_attempts):
        try:
            resp = await send()
            last = f"HTTP {resp.status_code}"
            if resp.status_code not in RETRY_STATUS:
                return resp                      # 200, or a 4xx we should not retry
            retry_after = parse_retry_after(resp)
        except (httpx.TimeoutException, httpx.ConnectError) as exc:
            resp, retry_after, last = None, None, type(exc).__name__
        if attempt == max_attempts - 1:
            break
        delay = backoff_delay(attempt, base, cap, jitter, rng)
        if retry_after is not None:
            delay = max(delay, retry_after)      # the server's number is a floor
        if time.monotonic() - start + delay > max_elapsed:
            break                                # do not sleep past the caller's deadline
        if log:
            log(f"attempt {attempt + 1} failed ({last}); sleeping {delay:.2f}s")
        await asyncio.sleep(delay)
    raise GaveUp(f"gave up after {attempt + 1} attempts, last: {last}")

Use it by passing a zero-argument function that makes the call. That keeps the helper independent of any SDK:

resp = await call_with_retry(lambda: client.post(URL, json=payload), max_attempts=5, max_elapsed=30)

Two choices to know about. It returns non-retryable responses, such as a 400, instead of raising, so you decide what a 400 means. And it only reads the seconds form of Retry-After; the header can also carry an HTTP date, which I didn't handle.

Does it behave? Four scenarios against the mock

The mock server is a small FastAPI app with an admin endpoint that sets a script of status codes, so each scenario is deterministic. It sends retry-after: 1 with its 429s. Output from the run:

[429,429,500 then ok] returned HTTP 200 in 3.55s, server saw 4 requests {'429': 2, '500': 1, '200': 1}
    attempt 1 failed (HTTP 429); sleeping 1.00s
    attempt 2 failed (HTTP 429); sleeping 1.00s
    attempt 3 failed (HTTP 500); sleeping 1.53s
[400 is not retried] returned HTTP 400 in 0.00s, server saw 1 requests {'400': 1}
[always 500, 4 attempts] GaveUp: gave up after 4 attempts, last: HTTP 500 in 0.99s, server saw 4 requests {'500': 4}
[429 with Retry-After: 1, deadline 1.5s] GaveUp: gave up after 2 attempts, last: HTTP 429 in 1.01s, server saw 2 requests {'429': 2}

Read the first block closely. The two 429 waits are exactly 1.00 seconds because the server's Retry-After beat the random delay. The 500 wait (1.53 seconds) came from the jitter formula alone, since a 500 carried no header. The last scenario shows the deadline working: after the first 1-second sleep, another one would have crossed 1.5 seconds, so the helper stopped instead of sleeping. That behavior is what protects you from the spend-cap 429 described earlier.

Does jitter actually help? A herd test, with an honest result

I expected jitter to cut the number of requests hitting the server. It didn't. The test: 40 clients start at once against a mock that allows 10 requests in any rolling second and returns 429 without a Retry-After. Each client retries up to 8 times with base 0.25 and cap 4 seconds. I ran each variant three times:

[herd jitter=none ] ok=40/40 gave_up=0 server_requests=160 429s=120 wall=8.0s
[herd jitter=full ] ok=40/40 gave_up=0 server_requests=163 429s=123 wall=4.9s
[herd jitter=none ] ok=40/40 gave_up=0 server_requests=160 429s=120 wall=8.0s
[herd jitter=full ] ok=40/40 gave_up=0 server_requests=170 429s=130 wall=4.7s
[herd jitter=none ] ok=40/40 gave_up=0 server_requests=160 429s=120 wall=8.0s
[herd jitter=full ] ok=40/40 gave_up=0 server_requests=164 429s=124 wall=5.5s

Everyone succeeded either way. What changed is wall time: 8.0 seconds without jitter, 4.7 to 5.5 with it. Without jitter, the 30 losers of round one all wake together, 10 win, 20 lose, and so on in lockstep waves, and the server sits idle between waves. With jitter, retries arrive spread out and the server's capacity gets used. Request counts were about the same or slightly higher with jitter.

So the claim I'd defend is "jitter finishes the job sooner when many clients share a limit". I would not claim it reduces load. This is one mock with one limiter shape; a real API with token buckets and per-minute windows may differ.

How do you stop a retry from doing the work twice?

The dangerous case is a timeout. Your client gives up waiting, but the server may have finished the job anyway. If the job charged a card, sent an email or ran a tool, a blind retry does it again.

I reproduced this. The mock's handler runs a side effect (a counter), then delays its response beyond the client's 0.5-second timeout on the first request only. The client times out, retries, and gets a 200:

[lost response, idempotency key=False] client got HTTP 200; side effect ran 2 time(s)
[lost response, idempotency key=True ] client got HTTP 200; side effect ran 1 time(s)

With an Idempotency-Key header holding the same UUID on every attempt, the mock stored the result under that key and returned it on the retry. The fix needs the server to cooperate; the header alone does nothing. The pattern, in the handler you control:

key = request.headers.get("idempotency-key")
if key is not None and key in effects:
    return effects[key]          # already done: return the stored result
result = do_the_side_effect()
if key is not None:
    effects[key] = result        # in production: a database row with a unique constraint
return result

Generate the key once per logical operation, before the first attempt, and reuse it for every retry. A new key per attempt defeats the whole thing. For a worked example with a payment provider's webhooks, see Razorpay webhooks and idempotency in FastAPI.

Pure LLM calls are the easy case: running a completion twice costs tokens but changes nothing in your world. I did not find idempotency keys documented on Anthropic's errors page, so for model calls, assume a retry after a timeout is a fresh, billable request. The idempotency problem shows up in what you do around the call: tool executions, writes, notifications.

When retries aren't the answer: queueing and batching

Retrying makes one call survive a bad moment. It doesn't fix sustained demand above your limit. If 429s are routine, you're past the point where backoff helps, and the options are structural:

• Cap concurrency. An asyncio.Semaphore around the call keeps you under the limit before the server has to tell you. See async patterns for LLM production.
• Queue the work. Put jobs on a queue, run a fixed number of workers, and let the queue absorb bursts. Retry failed jobs by putting them back with a delay instead of sleeping inside a worker.
• Use batch processing for work nobody is waiting on. Anthropic's errors page itself points to the Message Batches API for long-running requests. Our batch API guide covers cost and setup.
• Limit your own users. If the 429s come from a free tier being hammered, fix that at the door: rate limiting a free-tier AI app.

A rule I'd apply: one layer retries. If your SDK already retries twice and your helper retries five times around it, one failed call becomes up to fifteen requests. Set max_retries=0 on the SDK client when you wrap it yourself.

What should you log?

Log every failed attempt with the status, the sleep time and the request ID. Anthropic's errors page says every response carries a request-id header (also in the error body as request_id) and asks you to include it when contacting support. That one field turns a vague "the API was flaky" into something the provider can look up. Also count retries per hour. A rising retry rate is the earliest warning you'll get that you're approaching a limit.

What I did not verify

• The mock's behavior is invented. Real providers' limiter shapes, header values and recovery times differ, and Anthropic's retry-after semantics beyond what its errors page states weren't tested.
• I did not call any real LLM API. No timing here is a measurement of a vendor.
• I didn't read the SDK's own source or README, so the SDK retry behavior is as stated on the errors page only, including "twice by default".
• Mid-stream error handling and the HTTP-date form of Retry-After are not tested.

Checklist

1. Retry only 408, 429, 5xx and network errors. Never 400, 401, 403.
2. Exponential backoff, capped, with jitter.
3. Treat Retry-After as a minimum wait.
4. Set an attempt limit and a total deadline.
5. Reuse one idempotency key per logical operation, and have your server enforce it.
6. One retry layer only.
7. Log status, sleep time and request ID for each failed attempt.

Related reading

• Gateway retries, timeouts and model fallback in Python, with a failure-injection test