LLM Gateway Retries, Timeouts and Model Fallback in Python (With a Failure-Injection Test)

Gateways make one thing easy: switching models by changing a string. That makes fallback cheap, if your client knows when to switch. Most code I see either retries everything forever or gives up on the first 429. This post builds a small wrapper that does the right thing in seven failure scenarios, with output from a test where I inject the failures.

Disclosure: the AICredits links on this site are affiliate links. I had no AICredits key, so everything here runs against a local mock gateway I wrote. The mock simulates 429s, 503s, unknown models and slow replies. It is not AICredits and says nothing about AICredits' real error rates.

What should a client retry, and what should it never retry?

The wrapper

It tries each model in order. For each model it makes up to attempts calls, sleeping with jittered exponential backoff or the server's Retry-After. It stops immediately on non-retryable errors, and it has an overall deadline so a long outage cannot hold a request for minutes.

def chat(base, key, models, messages, *, attempts=3, timeout=5.0, base_delay=0.05, log=None, headers=None):
    log = log if log is not None else []
    deadline = time.monotonic() + 30
    for model in models:
        for n in range(attempts):
            status, data, h = post(base, key, {"model": model, "messages": messages}, timeout, headers)
            log.append((model, n + 1, status))
            if status == 200:
                return data
            if status in (400, 401, 403, 404):
                if status == 404 and model != models[-1]:
                    break                       # unknown model: try the next one
                raise Fatal(f"{model}: HTTP {status} is not retryable")
            if time.monotonic() > deadline:
                raise Fatal("overall deadline exceeded")
            ra = h.get("Retry-After")
            delay = float(ra) if ra and ra.replace(".", "").isdigit() else base_delay * (2 ** n)
            time.sleep(delay + random.uniform(0, base_delay))
    raise Fatal("all models failed: " + ", ".join(models))

Timeouts and connection errors come back from post() as status 0, which falls through to the retry path.

What happens in each failure scenario?

I ran seven scenarios against the mock. Each line shows the outcome and every attempt as (model, attempt number, status):

healthy              ok -> pong   [('mock/ok', 1, 200)]
flaky then ok        ok -> pong   [('mock/flaky', 1, 429), ('mock/flaky', 2, 200)]
down, fallback ok    ok -> pong   [('mock/down', 1, 503), ('mock/down', 2, 503), ('mock/down', 3, 503), ('mock/ok', 1, 200)]
all down             FATAL all models failed: mock/down, mock/down   [('mock/down', 1, 503), ('mock/down', 2, 503), ('mock/down', 1, 503), ('mock/down', 2, 503)]
bad key (no retry)   FATAL mock/ok: HTTP 401 is not retryable        [('mock/ok', 1, 401)]
unknown model        ok -> pong   [('no/such', 1, 404), ('mock/ok', 1, 200)]
slow, tight timeout  ok -> pong   [('mock/slow', 1, 0), ('mock/ok', 1, 200)]

(The real output is one line per scenario in aligned columns; I reflowed it for width.)

Three things to read in that output:

• The bad key is tried exactly once. Retrying a 401 wastes time and, with a metered key, can lock you out.
• The 503 case burns three attempts before falling back. If the first model is clearly down, that is up to 3 backoff waits before the user sees an answer. Lower attempts for latency-sensitive paths.
• The slow case shows a hidden cost. The client timed out at 0.1 s (status 0) and moved on, but the mock server was still working on that request. My run printed a BrokenPipeError from the mock server when it tried to reply to a client that had already left. On a real API, a request abandoned like that may still complete and be billed.

Does a gateway's failover make this unnecessary?

No, because they cover different failures. AICredits' homepage lists "Automatic Failover" and describes it as retrying on the next available route if a provider returns an error or rate-limits. That is a vendor claim I could not test without a key, so treat it as a feature to verify, for example by watching which attempts succeed during a known provider outage.

Even if it works as described, your client still faces:

• the gateway itself being unreachable or returning 5xx,
• your own network or DNS failing,
• one model being unavailable on every route,
• a quality failure, where the answer is wrong rather than an error.

Layering also has a cost. If the gateway retries three routes and your client retries three times, one user request can fan out to nine upstream calls during an incident. Keep client attempts low, put an overall deadline on the request, and log every attempt so you can see the fan-out.

How should you choose fallback models?

• Same capability tier first. Falling back from a large model to a tiny one can change answers silently. Decide whether you accept that.
• Check the model ID exists. Use the catalog search from custom base URL settings and model IDs so you do not fall back to a retired string.
• Check feature parity. If your primary uses tool calls or JSON mode, the fallback must too. The conformance test checks this per model.
• Record which model answered. Log it with the response so cost and quality reports are honest.

What this does not cover

• Streaming retries: once tokens have started flowing, a retry restarts the answer. Decide whether to buffer or surface the error.
• Idempotency for side effects. Chat completions are usually safe to repeat; tool executions are not.
• Circuit breakers that stop calling a model for a while after repeated failures. They are worth adding once you have real error data.

For a deeper treatment of backoff math, read rate limits, retries and backoff. For routing across providers with a proxy, see model routing with LiteLLM and OpenRouter. To try a gateway with Indian payment options, the AICredits signup is an affiliate link.

What I could not verify

• AICredits' real error rates, rate limits, or whether its failover works as described.
• Retry-After behaviour on the live service. My mock sends Retry-After: 0.
• Whether abandoned requests on the live service are billed.