Stream LLM Responses with SSE: FastAPI Endpoint to a Next.js Client

The model takes eight seconds to finish an answer, and your user stares at a spinner for all eight. Streaming fixes that: the first words show up in a few hundred milliseconds. The standard way to do it over plain HTTP is server-sent events (SSE). This post builds an SSE endpoint in FastAPI, reads it with a fetch client that works in a browser or Next.js, checks what happens when the user closes the tab, and covers the buffering traps that make streams look broken behind nginx or Caddy.

There is no model in the code. A fake generator emits words 200 ms apart, so you can see the plumbing without an API key. For the model side, the FastAPI + Claude production patterns post streams from Claude, and the TypeScript gateway post streams with the OpenAI and Vercel AI SDKs. This one stays on the transport: the wire format, the client, disconnects and proxies.

What I ran

All of this ran on 2026-10-23 with Python 3.13, FastAPI 0.161.1, Starlette 1.9.0, Uvicorn 0.54.0 and Node 22. The token source is a stand-in. The nginx and Caddy parts are from their documentation, which I read today, and I did not run either proxy.

How do you write an SSE endpoint in FastAPI?

An SSE response is a normal HTTP response with Content-Type: text/event-stream that never ends until the server says so. Each event is one or more field: value lines followed by a blank line. The HTML specification requires UTF-8 and describes data: lines being joined with newlines, which is why I put JSON in data: and avoid worrying about spaces and line breaks in tokens.

import asyncio, json
from fastapi import FastAPI
from fastapi.responses import StreamingResponse

app = FastAPI()
WORDS = "Streaming lets the first words appear while the rest is still being generated".split()

async def fake_llm(prompt: str):
    for w in WORDS:
        await asyncio.sleep(0.2)
        yield w + " "

def sse(data: str, event: str | None = None) -> str:
    head = f"event: {event}\n" if event else ""
    body = "".join(f"data: {line}\n" for line in data.split("\n"))
    return head + body + "\n"

@app.get("/stream")
async def stream(prompt: str = "hi"):
    async def gen():
        async for tok in fake_llm(prompt):
            yield sse(json.dumps({"token": tok}))
        yield sse("[DONE]", event="done")

    return StreamingResponse(
        gen(),
        media_type="text/event-stream",
        headers={"Cache-Control": "no-cache", "X-Accel-Buffering": "no"},
    )

Run it with uvicorn app:app and read it with curl -N. The -N flag turns off curl's own output buffering, which otherwise makes a working stream look dead. I added -i to see the headers:

$ curl -sN -i "http://127.0.0.1:8103/stream?prompt=hi"
HTTP/1.1 200 OK
date: Fri, 23 Oct 2026 13:11:06 GMT
server: uvicorn
cache-control: no-cache
x-accel-buffering: no
content-type: text/event-stream; charset=utf-8
Transfer-Encoding: chunked

data: {"token": "Streaming "}

data: {"token": "lets "}

data: {"token": "the "}

...

data: {"token": "generated "}

event: done
data: [DONE]

I trimmed the middle words. The chunked transfer encoding and the missing Content-Length are what you want. Send your own errors as an event too, for example event: error, so the client can tell a failed generation from a finished one. HTTP status codes cannot change after the first byte.

How do you read the stream with fetch?

The browser's EventSource only issues GET requests and cannot set an Authorization header, which rules it out for many chat APIs. fetch plus a stream reader has no such limits. This client parses frames split on blank lines, ignores comment lines, and stops on the done event:

export async function streamTokens(url, onToken, signal) {
  const res = await fetch(url, { signal, headers: { Accept: "text/event-stream" } });
  if (!res.ok || !res.body) throw new Error(`HTTP ${res.status}`);

  const reader = res.body.pipeThrough(new TextDecoderStream()).getReader();
  let buffer = "";
  for (;;) {
    const { value, done } = await reader.read();
    if (done) return;
    buffer += value;
    let end;
    while ((end = buffer.indexOf("\n\n")) !== -1) {
      const frame = buffer.slice(0, end);
      buffer = buffer.slice(end + 2);
      let event = "message";
      const data = [];
      for (const line of frame.split("\n")) {
        if (line.startsWith(":")) continue; // comment / heartbeat
        if (line.startsWith("event:")) event = line.slice(6).trim();
        if (line.startsWith("data:")) data.push(line.slice(5).replace(/^ /, ""));
      }
      if (event === "done") return;
      if (data.length) onToken(JSON.parse(data.join("\n")).token);
    }
  }
}

The buffer matters. A network chunk can end in the middle of a frame, so you accumulate text and only act on complete frames. Skipping that is the classic "works on localhost, breaks in production" bug. The parser handles \n\n only. The spec also allows \r\n, so if your server or a proxy rewrites line endings you would need to handle that, and I did not test it.

I syntax-checked the file with node --check and ran it under Node 22 against the server above, printing the elapsed time for each token:

250 ms  "Streaming "
450 ms  "lets "
651 ms  "the "
853 ms  "first "
1054 ms  "words "
...
2663 ms  "generated "
stream finished

First token at 250 ms, last at 2.66 seconds. A non-streaming endpoint would have shown nothing for the full 2.66. In React you would call streamTokens from an event handler, append tokens to state, and keep an AbortController in a ref so a Stop button or an unmount can cancel it.

What happens when the client disconnects?

If you do nothing, an abandoned stream can keep calling the model and keep billing you. Test it. I started a request with curl and killed it after 0.7 seconds (--max-time 0.7), then asked the server what happened. The server code wraps the generator like this:

async def gen():
    stats["started"] += 1
    try:
        async for tok in fake_llm(prompt):
            yield sse(json.dumps({"token": tok}))
        yield sse("[DONE]", event="done")
        stats["finished"] += 1
    except asyncio.CancelledError:
        stats["cancelled"] += 1
        raise                      # always re-raise
    finally:
        pass                       # close upstream streams / release slots here

After one complete run and one curl timeout, the stats endpoint returned:

{"started":2,"finished":1,"cancelled":1,"log":["generator closed","cancelled after 0.70s","generator closed"]}

The cancelled stream stopped after 0.70 seconds, the same moment curl gave up, instead of running to the end. In this stack, Starlette cancels the task when the client goes away, and the generator gets a CancelledError. Two rules follow. Put the real model call inside the generator, so cancelling the generator cancels the call. And re-raise the exception, because swallowing it leaves a task that should be dead.

I also aborted from the Node client with an AbortController after three tokens. It printed stopped: AbortError, and the server's cancelled counter went up again.

Not every server setup behaves this way. Behind a proxy, the disconnect has to travel through it, which brings us to the next section.

Can you proxy the stream through Next.js?

Often you want the browser to talk to your Next.js app, not to FastAPI directly. A route handler can pass the upstream body straight through. The one line that matters is signal: req.signal, which cancels the upstream fetch when the browser disconnects:

// app/api/chat/stream/route.ts, shown here as plain JS so Node could syntax-check it
const FASTAPI = process.env.FASTAPI_URL ?? "http://127.0.0.1:8000";

export async function GET(req) {
  const prompt = new URL(req.url).searchParams.get("prompt") ?? "hi";
  const upstream = await fetch(`${FASTAPI}/stream?prompt=${encodeURIComponent(prompt)}`, {
    signal: req.signal,
    cache: "no-store",
  });
  return new Response(upstream.body, {
    status: upstream.status,
    headers: {
      "Content-Type": "text/event-stream; charset=utf-8",
      "Cache-Control": "no-cache, no-transform",
      "X-Accel-Buffering": "no",
    },
  });
}

I did not run this inside a Next.js server. I imported the handler in plain Node, called it with a Request that carried an AbortSignal, read two chunks, then aborted. It printed the status and content type, both chunks, then read after abort: AbortError, and the FastAPI server logged another cancelled stream after 0.41 seconds. That shows the abort chain works in principle. Check it again in your deployed setup, since Next.js version, hosting platform and any middleware can change it. If you deploy on a platform with function time limits, a long stream can be cut off by the platform. That is an issue I have not tested.

Why does the stream arrive in one piece behind nginx or Caddy?

Because proxies buffer. Your tokens reach the proxy on time and sit there until the buffer fills or the response ends. Locally everything looks fine, and production looks like a spinner followed by a wall of text.

nginx. From the nginx proxy module documentation, read on 2026-10-23: proxy_buffering is on by default. With it off, the response is passed to the client synchronously as it is received. The same page says buffering can also be switched on or off by passing yes or no in the X-Accel-Buffering response header, and that this capability can be disabled with proxy_ignore_headers. The page does not say which wins if both are set, so do not rely on a guess. Set the header in the app (as above) and, for a path that only serves streams, also set it explicitly:

location /api/chat/stream {
    proxy_pass http://127.0.0.1:8000;
    proxy_http_version 1.1;
    proxy_buffering off;
}

If it still buffers, check whether something else sits in front (a CDN, another nginx, gzip middleware) and whether proxy_ignore_headers lists X-Accel-Buffering. I have not run nginx for this post, so treat this block as a documented-but-unrun configuration.

Caddy. From the reverse_proxy documentation, read on 2026-10-23: flush_interval defaults to no periodic flushing, and -1 disables response buffering and flushes after each write. The page also says that for responses with Content-Type: text/event-stream, the option is ignored and responses are flushed immediately. So a correctly typed SSE response should work without extra config, and the setting is for streams that do not use that content type. The same page says that in the -1 mode the backend request is not cancelled if the client disconnects early. If you set flush_interval -1 and your cancellation test stops working, that is the documented reason.

A keep-alive helps against idle timeouts. The HTML spec suggests a comment line about every 15 seconds. An untested sketch of the idea: wrap your token generator so that if nothing arrives within 15 seconds, it yields : keep-alive\n\n. The client above already skips lines that begin with a colon.

Checklist

• Content-Type: text/event-stream, Cache-Control: no-cache, and X-Accel-Buffering: no on the response.
• Tokens inside JSON data: lines, frames ended with a blank line, a clear done event, and an error event for failures.
• Client accumulates a text buffer and only parses complete frames.
• AbortController on the client, signal: req.signal in any proxying route handler.
• Model call inside the generator, finally closes upstream streams, CancelledError re-raised.
• You tested disconnect with curl --max-time and watched a counter or log, behind the real proxy.
• nginx proxy_buffering and proxy_ignore_headers checked; Caddy flush_interval -1 understood.

Where to host it

A streaming endpoint wants a long-lived process, not a short-timeout function, so a small VPS with Caddy or nginx in front is a natural fit. Deploying a FastAPI LLM app with Docker and nginx walks through the setup. Hostinger's KVM 2 is one option (affiliate link); Hetzner and DigitalOcean also work, and this comparison lays out the trade-offs.

Disclosure: the Hostinger link is an affiliate link. It costs you nothing, and the code above runs the same on any host.

What I did not verify

I did not run nginx or Caddy, so the proxy sections rest on their documentation. I did not run the Next.js route inside Next.js, and I did not test a real model, authentication, or many concurrent streams. The keep-alive sketch is unrun. The timings come from a fake generator with a fixed 200 ms delay.