Voice Agent Latency Budget: Turn-Taking and Interruptions

A voice agent feels slow when the gap between the caller finishing a sentence and hearing a reply is longer than a person would leave. The gap is the sum of five delays: the silence you wait before deciding the caller is done, STT finalisation, the LLM's time to first sentence, TTS time to first audio, and the network hops on both sides. The first delay is the easiest to cut, and cutting it too far makes the agent talk over people who pause to think.

This post gives you a latency budget you can fill in with your own measurements, a mock pipeline I ran to show which levers matter, and tuning rules for turn-taking and interruptions. The mock uses assumed stage delays, not vendor benchmarks. It shows structure and proportions, not what Deepgram or ElevenLabs will do on your line.

Where does the time go in a voice agent turn?

Here is the budget I used. Every number is an assumption you should replace with timings from your own logs.

Added up with streaming, that is about 1,540 ms, which is the first lesson. The commonly quoted "under a second" target needs real cuts to the first and fourth rows, and each cut has a cost.

What does the mock measure?

I wrote a small asyncio script that sleeps for random stage delays drawn from the table above (a normal distribution with a small jitter) and measures four things over 60 simulated turns: a sequential pipeline that waits for the whole LLM reply and whole TTS before playing, a streaming pipeline that starts TTS after the first sentence, the same with the endpoint wait cut to 300 ms, and the time from barge-in to the agent going silent. The core of it:

async def streaming_turn():
    t0 = time.perf_counter()
    await sleep(d(ENDPOINT) + d(STT_FINAL) + d(NET))      # caller done -> final transcript
    await sleep(d(LLM_TTFT) + d(LLM_TO_FIRST_SENTENCE))   # LLM writes the first sentence
    await sleep(d(TTS_TTFB) + d(NET))                     # TTS first byte -> caller's ear
    return time.perf_counter() - t0

I ran it on 2026-10-09 with Python 3 and got:

60 simulated turns each (assumed stage delays, see top of file)
sequential (wait for full reply)   p50   2521 ms   p95   2940 ms
streaming (TTS on first sentence)  p50   1575 ms   p95   1831 ms
streaming + 300 ms endpoint        p50   1323 ms   p95   1577 ms
barge-in to agent audio silent     p50    321 ms   p95    395 ms
budget sums (means): endpoint 500 + stt 120 + llm 600 + tts 200 + net 60 ms

Read it as arithmetic, not a benchmark. Three things fall out.

1. Streaming is the big structural win. Starting TTS on the first sentence cut the median by about 950 ms. It costs you nothing in accuracy, only some engineering in splitting the stream at sentence boundaries.
2. Shortening the endpoint wait buys about 250 ms and costs interruptions. In the mock, nothing goes wrong when the wait drops to 300 ms because the mock caller never pauses. A real caller reading out a 10-digit number does.
3. Barge-in is bounded by detection time plus buffered audio. I assumed 250 ms for the voice activity detector to be confident and 100 ms of audio already buffered downstream. That gives roughly 320 ms before the caller hears silence. Note that this row only adds up my two assumptions (it is not a measurement of anything real), so use it as a reminder of which terms exist. If your telephony leg buffers more, the number grows and no model can fix it.

How do you tune turn-taking without cutting people off?

Silence alone is a poor signal. LiveKit's turn documentation lists the options: a turn-detector model layered on voice activity detection (the default), VAD only for the lowest latency, STT-based end-of-utterance detection, and manual control for push-to-talk. It also describes a configurable minimum and maximum endpointing delay, and an adaptive interruption mode that tries to distinguish a real interruption from a backchannel like "hmm" or "okay". The page did not give default values for those delays, so check the reference for your version.

Practical rules, none of them magic:

• Use a semantic end-of-turn signal (a turn-detection model or STT utterance-end event) before shortening silence thresholds.
• Vary the wait by what you asked. After "what's your phone number?" wait longer, because people pause between digit groups. After a yes/no question, wait less.
• Let the agent say a short filler while a slow tool runs: "Let me check that." It resets the caller's sense of time.
• Set a minimum number of words before an interruption counts, so a cough or "uh-huh" does not make the agent stop mid-sentence. LiveKit exposes min_duration and min_words for this purpose.
• Handle false interruptions. If the agent stopped and the caller said nothing meaningful, resume or restate. LiveKit documents a timeout (their example uses 2 seconds) before it treats a pause as a false interruption.

How do you handle interruptions correctly?

When the caller barges in, four things must happen, in this order:

1. Stop TTS generation.
2. Flush audio already queued to the phone leg.
3. Truncate the assistant message in your history to what was actually spoken. If the model believes it said the full reply, it will skip information the caller never heard.
4. Pass the caller's new words to the LLM.

Step 3 is the one people forget. Track how much text the TTS has emitted audio for, and store only that plus a marker such as [interrupted].

What should you measure on a real line?

Log timestamps at each boundary: caller speech end (from VAD), final transcript received, LLM first token, LLM first sentence complete, TTS first byte, first byte sent to the carrier. Then compute per-stage p50 and p95 from at least 50 real calls. A tiny table in a spreadsheet beats any benchmark in a blog post, including this one.

Two cheap experiments:

• Prompt for short first sentences. Add one line to the system prompt: "Start every reply with a sentence of eight words or fewer." This shortens the time to the first sentence boundary, so TTS can start sooner. Untested template: try it and compare your LLM-to-first-sentence timings.
• Keep the prompt prefix stable so provider-side prompt caching can apply. See prompt caching for the Claude API. Whether it lowers latency for your model and region is something to measure, not assume.

Faster is not free: a bigger model or a premium voice may cost more per minute, which the voice agent cost calculator lets you check before you trade cost for latency.

When to stop optimising

If p50 is under about 1.5 seconds, interruptions stop within half a second and callers rarely get cut off, spend your time on what the agent says. Latency work has diminishing returns, and a fast wrong answer is still wrong. The prompting side of voice agents and the architecture overview are the next reads, and streaming patterns for agents cover the text side of the same idea.

What I did not verify

The mock uses invented stage delays. I did not measure any vendor's STT, LLM or TTS latency, and I did not place a phone call. LiveKit's documentation was read on 2026-10-09 through a summarising fetch, so check exact option names and defaults on the page before you rely on them.