Python Async Concurrency Interview Questions for AI Engineers, With Runnable Output

The question is usually phrased as "you have 500 documents to send to an LLM API, how do you do it?" and the weak answer is a for loop, or gather over all 500 at once. The strong answer caps concurrency, retries with jitter, times out per call, handles one failure without losing 499 results, and knows why a stray time.sleep freezes everything. This post turns those into practice questions with an answer outline, a follow-up and a common wrong answer for each, backed by scripts I ran on 2026-10-09 with Python 3.13.

These are practice questions I wrote for this site. They aren't leaked and they aren't from any company. The API in every script is a fake (asyncio.sleep plus random failures), so no model was called and the timings show concurrency behaviour, not any vendor's latency. For a longer pattern catalogue see async Python for LLM apps; this post is the interview version, and for other rounds see AI engineer interview questions on RAG, agents and evals.

How do you call an LLM API 50 times concurrently?

Strong answer outline

• Each call spends nearly all its time waiting on the network, so one thread running an event loop can keep many in flight. await is the point where a coroutine hands control back to the loop.
• asyncio.gather(*coros) starts them all. That is fine for 50 and dangerous for 5,000, because providers enforce rate limits and your process holds every connection and response at once.
• Cap in-flight work with an asyncio.Semaphore(n). The semaphore doesn't make calls faster, it makes the peak predictable.

Fifty fake calls of 0.1 seconds each:

sequential: 5.04s
gather all 50: 0.10s
semaphore(5): 1.01s, peak in flight = 5
semaphore(10): 0.50s, peak in flight = 10

The arithmetic is worth saying aloud: 50 calls times 0.1 s divided by the concurrency limit gives the lower bound. A limit of 5 gives 1 second and a limit of 10 gives 0.5.

Follow-up: How do you pick the limit? Start from the provider's documented rate limits for your account tier (requests and tokens per minute), divide by the typical call, and leave headroom. Then watch for 429 responses and lower it. I'm not quoting a provider's numbers here because they change; read your own dashboard.

Common wrong answer: "Async makes it parallel, so use as many tasks as possible." Concurrency isn't parallelism, and unbounded concurrency turns into rate-limit errors and memory use.

What's the difference between asyncio, threads and processes?

Strong answer outline

• asyncio: one thread, cooperative switching at await. Best for many I/O waits, needs async-aware libraries.
• Threads: the OS switches them, so blocking libraries work. In CPython the GIL stops two threads running Python bytecode at once, so threads help I/O-bound work, not pure-Python computation.
• Processes: separate interpreters, real parallelism for CPU-bound work, with serialisation cost to move data.
• For an LLM app: network calls go in asyncio, a blocking SDK call goes in a thread (asyncio.to_thread), CPU-heavy steps such as large PDF parsing or local embedding go in a process pool or a service.

Follow-up: Where does local model inference fit? Mostly into a separate process or server, because it's compute-bound and often releases the GIL inside native code anyway. Say you'd put it behind its own endpoint and await that.

Common wrong answer: "The GIL means Python can't do concurrency." It can do I/O concurrency well. The GIL limits parallel Python bytecode, not waiting.

What goes wrong if you call a blocking function inside async code?

This one catches people who "added async" to a function and kept a synchronous SDK inside it.

Strong answer outline

• The loop is one thread. A blocking call holds it, so every other coroutine, including timeouts, heartbeats and other requests, stalls until it returns.
• Fixes: use the library's async client, or push the call to a worker thread with asyncio.to_thread. The Python docs say to_thread is primarily for I/O-bound functions, and that because of the GIL it generally can't make CPU-bound code non-blocking.

I measured it with a 50 ms heartbeat task running while a 0.5 s blocking call happened:

blocking time.sleep(0.5)                      heartbeat every 50 ms, worst gap = 549 ms
await asyncio.to_thread(time.sleep, 0.5)      heartbeat every 50 ms, worst gap = 59 ms

In a web server, that 549 ms gap is what every other user's request experiences while one request does something blocking.

Follow-up: How would you find this in production? Enable asyncio debug mode, which logs callbacks that take too long, and watch event-loop lag as a metric. I haven't benchmarked the debug-mode overhead here.

Common wrong answer: "Marking the function async def makes it non-blocking." Only await on something that actually yields does that.

gather or TaskGroup: what happens when one task fails?

Strong answer outline

• With default settings, gather raises the first exception to the caller, but the docs state the other awaitables won't be cancelled and keep running. With return_exceptions=True, errors come back as values in the result list.
• asyncio.TaskGroup (Python 3.11 and later) cancels the remaining tasks when one fails with something other than CancelledError, then raises an ExceptionGroup after all tasks have finished. You handle it with except*.
• Choose by intent: partial results are useful (batch of documents) means catch per item or use return_exceptions=True; all-or-nothing means a TaskGroup.

One task fails after 0.1 s while two siblings need 0.3 s:

gather default      : ['gather raised: bad failed', 'a finished', 'b finished']
TaskGroup           : ['a cancelled', 'b cancelled', 'TaskGroup raised ExceptionGroup of 1']
return_exceptions   : ["ValueError('bad failed')", 'a']

Notice what gather did: the caller already saw the exception, yet "a" and "b" kept running and burning API budget in the background.

Follow-up: What does gather do if you cancel it? Per the docs, cancelling gather cancels all the awaitables that have not completed yet. A single child being cancelled doesn't cancel the gather.

Common wrong answer: "gather cancels the others when one raises." It doesn't, by default.

How do you retry rate-limited calls without making it worse?

Strong answer outline

• Retry only on errors that are transient (429, 5xx, timeouts), never on a 400 or auth failure.
• Exponential backoff: the wait cap doubles per attempt, up to a maximum. Add jitter, meaning randomness within that wait, so clients that failed together don't retry together.
• Limit the number of retries. The AWS Builders' Library article on timeouts, retries and backoff with jitter (read 2026-10-09 through a summarising fetch; check the page for exact wording) makes the same points and adds that retrying at every layer of a call stack multiplies load: its example is a five-layer stack with three retries per layer, which could raise database load 243 times, so retry at one point.
• Honour Retry-After headers when the provider sends them.

Toy test: 30 clients start at once against a fake server that accepts 5 calls per 100 ms and returns a rate-limit error otherwise. Each client's wait doubles from 0.1 s. Five seeds each:

none succeeded [30, 30, 30, 30, 30] requests [105, 105, 105, 105, 105] wall s [3.12, 3.13, 3.12, 3.12, 3.12]
full succeeded [30, 30, 30, 30, 30] requests [95, 91, 91, 96, 92] wall s [0.89, 1.08, 0.91, 0.83, 0.99]

Without jitter every client waits the same time, so they return in lockstep and the server's window refills with the same crowd: about 3.1 seconds. With full jitter (a uniform random wait between 0 and the current delay), the same work finished in roughly a fifth to a half of the time and with about 10 percent fewer requests. The jittered runs aren't seeded: my first five-seed run gave 0.83 to 1.08 s and a later re-run gave 0.62 to 1.51 s, while the no-jitter rows were identical every time. This is one synthetic server with a hard window, so don't expect those exact ratios from a real API.

Follow-up: Why not retry forever? Retries add load to a system that's already struggling and keep user requests hanging. Set a retry cap and a total deadline, then fail visibly.

Common wrong answer: "Exponential backoff solves thundering herds." Without jitter, clients that started together stay synchronised.

How do timeouts and cancellation actually work?

Strong answer outline

• asyncio.timeout(seconds) (3.11 and later) cancels the current task when time runs out by raising CancelledError inside it, then converts that into TimeoutError at the edge of the async with block, so you catch TimeoutError outside the block.
• Cancellation is an exception delivered at an await. Use try/finally for cleanup. If you catch CancelledError, re-raise it after cleaning up. The docs warn that TaskGroup and timeout() are built on cancellation and can misbehave if a coroutine swallows it.

Both a well-behaved and a swallowing coroutine under a 0.2 s timeout:

['slow_call saw CancelledError, cleaning up', 'slow_call finally ran', 'TimeoutError after 0.20s']
['swallowed CancelledError (bug)', 'returned normally'] 0.20s

The second case is the nasty one. The timeout fired, the coroutine ate the cancellation, the caller got a "normal" return and no error. In production that looks like a call that quietly produced nothing.

Follow-up: Where do you put the timeout: per call or per batch? Both. A per-call timeout stops one hung request from holding a semaphore slot forever. A batch deadline bounds the whole job.

Common wrong answer: "Catch the exception and continue." Catching CancelledError and continuing is the bug above.

What happens to a task you create and never await?

Strong answer outline

• The event loop keeps only weak references to tasks. The Python docs say a task with no other reference may be garbage collected at any time, even mid-execution, so keep a reference.
• The documented pattern is a module-level set: add the task, and add_done_callback(background_tasks.discard).
• An unawaited task's exception is never retrieved, and asyncio only logs "Task exception was never retrieved" when it is garbage collected.
• Better default: use a TaskGroup so the structure guarantees tasks finish before the block ends.

I didn't try to reproduce the garbage-collection case, because it's non-deterministic. I'm relying on the docs (read 2026-10-09 via a summarising fetch).

Follow-up: When is fire-and-forget acceptable? Logging or metrics where loss is fine, with the reference-set pattern and an exception handler. Not for billing events or anything you must not drop.

Common wrong answer: "create_task schedules it, so it'll run to completion." Only if something holds a reference.

Live-coding version: write bounded_map

The typical prompt: "process N items with at most 8 in flight, retry transient failures, don't let one bad item sink the batch, and keep results in input order." One answer, which I ran:

import asyncio, random

class Transient(Exception): pass

async def call_with_retry(fn, *, retries=4, base=0.05, cap=1.0):
    for attempt in range(retries + 1):
        try:
            return await fn()
        except Transient:
            if attempt == retries:
                raise
            await asyncio.sleep(random.uniform(0, min(cap, base * 2 ** attempt)))

async def bounded_map(fn, items, *, limit=8, retries=4, timeout=2.0):
    sem = asyncio.Semaphore(limit)
    async def one(item):
        async with sem:
            try:
                async with asyncio.timeout(timeout):
                    return await call_with_retry(lambda: fn(item), retries=retries)
            except (Transient, TimeoutError) as e:
                return e
    async with asyncio.TaskGroup() as tg:
        tasks = [tg.create_task(one(i)) for i in items]
    return [t.result() for t in tasks]

Against a fake API with 30 percent random transient failures and item 13 hanging forever, 40 items:

38/40 ok, 61 calls made, 2.11s
item 13 -> TimeoutError
first five -> [0, 1, 4, 9, 16]
failed items -> [13, 21] ['TimeoutError', 'Transient']

The failures are random, so your run will differ: a re-run of mine made 63 calls and failed items 13 and 39 instead. Talking points the interviewer will probe: failures come back as values, so the batch finishes; the TaskGroup only sees unexpected exceptions, which still cancel everything; the timeout covers the retries for one item but not the time spent waiting for a semaphore slot (is that what you want?); results stay in input order because the list is built from the task list, not completion order. Next steps you'd mention: honour Retry-After, a shared token-bucket for tokens per minute, and logging attempt counts per item.

How to prepare for this round

• Run the scripts above, then break them: remove the semaphore, remove jitter, add a blocking call. Seeing the failure beats reading about it.
• Memorise the four sentences: cap concurrency, jitter retries, time out every call, re-raise cancellation.
• Know your Python version. TaskGroup, asyncio.timeout and except* need 3.11 or later; on older versions you'd use wait_for and gather.
• Be ready to connect this to cost and rate limits; the LLM API cost per 1,000 requests post shows how retries inflate spend.
• Pair it with AI agent design interview walkthrough, where tool-call retries and timeouts come up again.

What I could not verify

I never called a real LLM API, so provider rate-limit behaviour, Retry-After handling and real latencies are untested. The retry experiment uses one synthetic server with a fixed window and five seeds. I haven't covered async queues and backpressure, async generators for streaming, or multiprocessing. The Python documentation and the AWS article were read through a summarising fetch tool on 2026-10-09, so check exact wording on the source pages. The scripts ran on Python 3.13.16 on a shared virtual machine; timings under 10 percent apart are noise.