Benchmark LLM Latency: Direct API vs Gateway, a Method and Python Harness You Can Rerun

"Is the gateway slower?" is a question you can answer in an afternoon, but most people answer it badly: ten requests, one route after the other, averages only. This post gives you a method that holds up and a Python harness to run it. I tested the harness itself, on a mock, to make sure it can detect a difference it should detect. I did not benchmark AICredits or any provider, and you will not find a latency number for them here.

Disclosure: AICredits links on this site are affiliate links. I had no API key, so every number below comes from a local mock server and describes the tool, not a real service.

What is worth measuring?

• Time to first token (TTFT): from sending the request to the first content chunk. This is how responsive a chat UI feels.
• Total time: from sending until the stream ends. This drives agent loops and batch jobs.
• Error count: a route that is fast but fails 3 percent of the time is not fast.
• Percentiles: p50 shows typical behaviour, p95 shows the bad days. Never report only the mean.

How do you set up a fair comparison?

Most bad benchmarks fail here:

1. Same model, same prompt, same output length. Different models invalidate everything. Set max_tokens to cap the output so that generation length does not dominate.
2. Same machine and network. Run both routes from the same place. If your app runs in a Mumbai data centre, benchmark from there, not from your laptop.
3. Warm up first. Connection setup and cold caches distort the first calls. Discard a few warm-up requests.
4. Interleave routes. Alternate A, B, A, B so both see the same provider load. My harness below runs one route at a time to keep the code short; wrap two bench loops in an alternating loop for real use.
5. Avoid prompt caching effects. If the provider caches repeated prompts, the second call is faster. Use varied prompts of equal length, or make sure both routes get the same caching conditions.
6. Run at different times of day. One 10-minute window tells you about that window.
7. Report the sample count. n=30 and n=500 are different claims.

The harness

It sends streaming chat requests, timestamps the first content chunk and the end of the stream, discards warm-up calls, counts errors, and reports nearest-rank percentiles. Standard library only.

def one_call(base, key, model, prompt, timeout=60):
    body = json.dumps({"model": model, "stream": True,
                       "messages": [{"role": "user", "content": prompt}]}).encode()
    req = urllib.request.Request(base + "/chat/completions", data=body, method="POST",
                                 headers={"Content-Type": "application/json", "Authorization": f"Bearer {key}"})
    t0 = time.perf_counter()
    ttft = None
    with urllib.request.urlopen(req, timeout=timeout) as r:
        for raw in r:
            line = raw.decode().strip()
            if not line.startswith("data:") or line == "data: [DONE]":
                continue
            ch = json.loads(line[5:])["choices"]
            if ch and ch[0].get("delta", {}).get("content") and ttft is None:
                ttft = time.perf_counter() - t0
    return ttft, time.perf_counter() - t0

bench(label, base, key, model, prompt, n, warmup) loops over one_call, collects the numbers and returns a row with ttft_p50_ms, ttft_p95_ms, total_p50_ms, total_p95_ms, n_ok and errors.

How do you know the harness works?

A benchmark tool you have not validated can be wrong in either direction. I tested it on a mock gateway, with route A answering immediately and route B (the mock's mock/slow model) sleeping 0.3 seconds before replying. If the harness is right, it should report B about 300 ms slower:

         route          n_ok        errors   ttft_p50_ms   ttft_p95_ms  total_p50_ms  total_p95_ms
     A mock/ok            20             0           1.2           1.5           1.4           1.7
   B mock/slow            20             0         302.2         305.0         302.4         305.1
difference in total p50 (B - A): 301.0 ms

It found the injected 300 ms (301.0). That says the timing logic is sound on a local server. It says nothing about the internet: real routes have jitter, TLS setup, queueing and variable generation length, which is why you need many samples.

How do you run it against a real gateway and the direct API?

1. Pick one model that both routes serve, and look up its exact ID on each side. For AICredits, the catalog search is in the model ID lookup post; the migration mapper translates direct IDs to catalog IDs.
2. Run the conformance test on both so that you know streaming works the same way.
3. Run bench for route A and B alternately, at least a few hundred calls each, in three windows across the day.
4. Save the raw per-call times, not only the summary, so you can plot or re-slice them.
5. Compare p50 and p95 TTFT and total. Decide whether the difference matters for your product. A 100 ms difference in time to first token is invisible in a batch job and noticeable in a voice agent.

How should you read the result?

• If p50 is close but p95 is much worse on one route, look at errors and retries before you blame the network.
• If TTFT is higher but total time is similar, the extra cost is up front (connection, routing), not in generation.
• If results change between time windows, report the range, not one number.
• If the difference is smaller than the spread between your own runs, you have not measured a difference.

Latency is one factor next to payment convenience, price and reliability. For cost, see the cost-per-feature script; for failure behaviour, retries and fallback. If you want to benchmark AICredits, the signup is an affiliate link. Publish your numbers with the sample size and date, whichever way they come out.

What I could not verify

• Real latency of AICredits, OpenAI, Anthropic or any other service.
• That the harness behaves identically over TLS and the public internet. The logic is the same, but I tested locally.
• How providers' own queueing changes results at your volume.