Is Your OpenAI-Compatible API Gateway Really Compatible? A Python Conformance Test

"OpenAI-compatible" is a marketing word until you test it. One gateway streams but drops the usage block. Another returns tool calls as text. A third accepts response_format and ignores it. You find out in production unless you check first. This post gives you a short Python script that runs eight checks against any gateway, shows it catching four deliberately broken behaviours, and applies the key-less 401 check to AICredits.

Disclosure: the AICredits links on this site are affiliate links. I did not have an AICredits API key when I wrote this, so I could run only the key-less 401 check (with a wrong key and with no key) against the real service. The rest of the suite is for you to run with your own key, and I say below what I did and did not observe.

What should a compatible gateway get right?

Eight things that break real apps:

The script

It uses only the standard library, so it runs anywhere Python does. call() sends a request and returns status, body and headers, treating HTTP errors as data. run_checks() returns a list of (name, ok, detail). The finish_reason check came from a real failure: my first mock sent streamed text and usage but no finish chunk, the OpenAI Python SDK accepted it, and the Vercel AI SDK threw Response stream ended without a finish reason. I added the check after that.

def run_checks(base, key, model):
    out = []
    def add(name, ok, detail=""):
        out.append((name, bool(ok), detail))

    s, b, _ = call(base, "sk-wrong", "/chat/completions",
                   {"model": model, "messages": [{"role": "user", "content": "hi"}]})
    err = json.loads(b).get("error", {})
    add("bad key returns 401 + error.message", s == 401 and "message" in err, f"status={s}")
    ...
    s, b, h = call(base, key, "/chat/completions",
                   {"model": model, "stream": True, "messages": [{"role": "user", "content": "ping"}]})
    lines = [l for l in b.splitlines() if l.startswith("data:")]
    add("stream=true yields SSE data lines ending in [DONE]",
        s == 200 and len(lines) >= 2 and lines[-1].strip() == "data: [DONE]")
    ...
    return out

Usage is one call: run_checks("https://your-gateway/v1", API_KEY, "your/model"), then print failures.

Does the test catch broken gateways?

A test you have not seen fail is not a test. I wrote a small local mock gateway whose behaviour I can break per model name: mock/no-stream ignores stream=true, mock/no-finish streams without a finish chunk, mock/no-tools never returns tool calls, mock/bad-json ignores JSON mode. The mock is mine and is not AICredits. Output on 2026-10-16:

mock/ok: 8/8 passed
mock/no-stream: 6/8 passed
   FAIL stream=true yields SSE data lines ending in [DONE]  [0 data lines]
   FAIL stream includes a chunk with finish_reason  [0 finish chunks]
mock/no-finish: 7/8 passed
   FAIL stream includes a chunk with finish_reason  [0 finish chunks]
mock/no-tools: 7/8 passed
   FAIL tool call returns parseable arguments  [status=200]
mock/bad-json: 7/8 passed
   FAIL json_object mode returns valid JSON  [status=200]

Notice the tool failure says status=200. That is the dangerous kind of failure: nothing errored, the gateway just did not do the thing. Status-code checks alone would miss it.

What does the real AICredits endpoint return without a key?

I sent two unauthenticated requests to https://api.aicredits.in/v1/chat/completions on 2026-10-16. A wrong key:

{"error":{"code":401,"message":"Invalid API Key","type":"authentication_error"}}

And no Authorization header at all:

{"error":{"code":401,"message":"Missing or invalid Authorization header","type":"authentication_error"}}

Both return HTTP 401 with an error object containing message and type, which satisfies the first check and matches the shape OpenAI-style clients expect. That is all I verified live. I did not test streaming, tool calls, JSON mode or usage on AICredits, because those need a funded key. The AICredits homepage claims "same request format, same streaming, same tool calling". Treat that as a claim until your own run passes.

You can try it: create an AICredits account (affiliate link), top up the minimum, and run the suite with a cheap model. Run it again with the exact model you plan to ship.

How do you read failures?

• Auth check fails (non-401): your client may retry auth errors forever. Fix the retry classifier before anything else.
• Usage missing: you cannot track cost from responses. Fall back to counting tokens yourself, and read cost tracking per feature.
• Streaming fails: chat UIs will hang or show the whole answer at once.
• No finish chunk: some SDKs tolerate it and some raise an error. See the TypeScript guide.
• Tools fail with status 200: the model or gateway is not returning structured tool calls. Agents will loop or stall. See function calling basics.
• JSON mode fails: add a validation and repair step, as in structured outputs and JSON mode.

Make it a CI check

1. Put the key in a secret, never in the repo.
2. Use a cheap model for the broad suite, and the production model for a nightly run.
3. Fail the build on any check failure, and print the status code and detail.
4. Add a check for every bug you hit in production. The suite should grow.

For retries and backoff once a check starts failing intermittently, see rate limits, retries and backoff. To compare gateways on more than compatibility, read AICredits vs OpenRouter and the AICredits review.

What I could not verify

• Any AICredits behaviour that needs an authenticated call: streaming, tools, JSON mode, usage fields, failover.
• Compatibility across all models. It differs per model behind a gateway.
• Anything about performance. This suite checks behaviour, not speed.