LLM Latency Budget: A Time-to-First-Token Playbook

A chat feature feels slow when the screen stays blank, even if the whole answer arrives in two seconds. The fix starts with splitting LLM latency into two numbers: time to first token (TTFT), which decides when the user sees anything, and generation time, which decides when the answer is done. They have different causes and different fixes.

This playbook shows how to measure both, with a script I ran on 2026-10-09 against a mock server, and which lever moves which number. The mock's timings are numbers I chose (a 120 ms queue, 90 ms of prefill per 1,000 input tokens, 25 ms per output token), so they illustrate the method and the shape of the trade-offs. They are not measurements of any real model. Run the script against your own endpoint to get yours.

Related reading: streaming agents with Claude and LangGraph covers the streaming plumbing, prompt caching covers cost, and for voice products see the voice agent latency budget.

Where does LLM latency come from?

A request spends its time in four places:

Anthropic's latency page lists the same levers at a high level: pick the right model, shorten prompt and output, and stream. It also says to get a prompt working first and optimize second, because trimming too early can hide what top quality looks like. I'd add one rule of my own: don't optimize anything until you have p95 numbers, since averages hide the slow tail that users complain about.

In my mock, TTFT is 120 ms + 90 ms per 1,000 input tokens, and decode is 25 ms per token. Treat that as a cartoon of the real thing. The useful part is the structure: input size moves TTFT, output size moves total time.

How do you measure TTFT and total time?

Time the request from the moment you send it to the first streamed event, then to the last. This is the core of the script (the full version is about 80 lines and loops over scenarios):

async def one_stream(client, payload):
    t0 = time.perf_counter()
    first = None
    n = 0
    async with client.stream("POST", URL, json=payload) as resp:
        async for line in resp.aiter_lines():
            if line.startswith("data: ") and line != "data: [DONE]":
                n += 1
                if first is None:
                    first = time.perf_counter() - t0
    total = time.perf_counter() - t0
    return first, total, n

Run it 30 times per scenario, sort the results, and report p50 and p95. With 30 samples, p95 is roughly the second-slowest run, so don't read it as precise. Use more samples against a real API.

Two measurement traps. First, measure from the client side as the user experiences it, not from inside your handler, or you'll miss proxy and network time. Second, if you have a proxy in front of the app, a buffered stream makes TTFT look equal to total time. My nginx, Caddy and Traefik streaming test measured exactly that: nginx's default held ten tokens for about two seconds until proxy_buffering was off. A full server block with that setting is in the FastAPI, Docker and nginx deployment post.

What do the numbers look like for each lever?

Here is the output of the full run against the mock (30 requests per row, 2026-10-09). Rows 1 and 2 use the same request: 2,000 input tokens, 60 output tokens.

streaming, 2000 in / 60 out        TTFT p50=   305ms p95=   310ms | total p50=  1880ms p95=  2026ms
non-streaming (user sees nothing)  TTFT p50=  1806ms p95=  1808ms | total p50=  1806ms p95=  1808ms
streaming + cache hit              TTFT p50=   133ms p95=   137ms | total p50=  1701ms p95=  1738ms
streaming, 8000 in / 60 out        TTFT p50=   844ms p95=   848ms | total p50=  2397ms p95=  2420ms
streaming, 2000 in / 300 out       TTFT p50=   304ms p95=   308ms | total p50=  8009ms p95=  8144ms
3 independent calls: sequential=5414ms parallel=1809ms

What each row tells you:

• Streaming moved the first visible text from 1,806 ms to 305 ms. Total time went up by about 75 ms because of per-event overhead. If your users read as the text arrives, that's a good trade. If another program consumes the output, streaming gains you nothing.
• A cache hit (my mock charges prefill for only 5% of the prompt on a hit) cut TTFT from 305 ms to 133 ms. The real figure depends on the model and prompt; the direction is what Anthropic's prompt caching page describes, with a stated improved time-to-first-token for long documents.
• Quadrupling the input (2,000 to 8,000 tokens) moved TTFT from 305 ms to 844 ms. Long stuffed prompts hurt before a single word appears.
• A longer answer (60 to 300 tokens) left TTFT untouched at 304 ms and pushed total time from about 1.9 seconds to 8.0 seconds. Asking for shorter output is the biggest lever on total time.
• Parallel calls: three independent calls took 5,414 ms in sequence and 1,809 ms with asyncio.gather, about the slowest single call. This only works when the calls don't depend on each other.

Which fix should you try first?

Pick by the symptom.

How do you cut time to first token?

Trim the input, then cache what repeats, then consider a smaller model.

1. Delete what the model doesn't need. Old chat turns, unused tool definitions and pasted documents all add prefill. Summarize or drop history past a threshold. Our token counting guide shows how to count what you're sending.
2. Cache the stable prefix. Per Anthropic's prompt caching page (re-read on 2026-10-09; confirm the exact wording, because the model list changes): the default cache lifetime is 5 minutes, refreshed each time the cached content is used; prompts shorter than the model's minimum cacheable length are processed uncached with no error, and the minimum varies by model, from 512 to 4,096 tokens in the list I saw (512 for the newest models listed, 4,096 for some older ones such as Claude Opus 4.5 and Haiku 4.5). A short system prompt may therefore cache nothing. The page also describes pre-warming the cache before a user's first request, which targets exactly the first-token delay on a cold cache. Check the response's usage: the page says if both cache_creation_input_tokens and cache_read_input_tokens are 0, the prompt was not cached.
3. Put stable content first. Caching works on a prefix, so a timestamp or user name at the top of the prompt can make every request a miss.
4. Pick a faster model for the step that needs speed. Anthropic's latency page names Claude Haiku 5.5 as its fastest option and says effort is its main control for speed and cost. Test that your quality holds at the smaller size before switching.
5. Cache whole answers when questions repeat. An exact or semantic cache skips the model entirely, so the TTFT is a lookup.

How do you cut total generation time?

Generate fewer tokens. Decode time is roughly tokens times per-token time, so halving the output roughly halves this part.

• Ask for the format you need. "Three bullets, under 15 words each" beats "summarize". Anthropic's latency page notes that asking for paragraph or sentence limits works better than word counts, because models count tokens, not words.
• Set max_tokens as a hard ceiling, but know the same page warns that hitting it cuts the response off, possibly mid-sentence. Use it as a safety net, not as your length control.
• Split a long job. Produce an outline first, show it, then expand sections on demand.
• Parallelize independent pieces. Summarizing five documents is five calls, not one call with five documents. Cap concurrency to stay under your rate limits; the async patterns post shows a semaphore, and the retries and backoff post covers what to do when you hit 429s.

How do you stream without breaking the UI?

Streaming adds failure modes that a normal request doesn't have:

• Proxy buffering (covered above).
• Mid-stream errors. Anthropic's errors page says an error can occur after the API has returned a 200, and the usual status-code handling doesn't apply. Your reader needs a branch for an error event and a way to show a partial answer plus a retry button.
• Idle connection drops. The same page warns that some networks drop idle connections, and recommends streaming or batches for long requests.
• Structured output. You can't parse half a JSON object. Stream prose to the user; for JSON, wait for the whole thing, or show a progress state.

What should a latency budget look like?

Write it down per feature, with a number for each stage, then check against measured p95. A template (the numbers are placeholders, fill in your own):

The budget makes arguments concrete. When a new feature adds 1,500 tokens of context, you can say it costs roughly its share of TTFT, check that against the table, and decide.

What I did not verify

• The mock's latency model is mine. Real prefill and decode costs vary by model, load and prompt.
• I didn't call a real LLM API. No number here is a vendor measurement.
• The caching and latency claims come from Anthropic's pages as read on 2026-10-09 (latency, caching and errors pages, re-read in full by the editor); model names and minimums on those pages change often.
• Mid-stream error handling is described from the docs, not tested.