Nginx vs Caddy vs Traefik for AI Apps: Streaming Test and Configs

If your AI app streams tokens and the answer appears all at once, the reverse proxy is the first suspect. For AI apps, nginx, Caddy and Traefik differ less in features than in defaults: nginx buffers streamed responses unless you tell it not to, while Caddy and Traefik passed a token stream through untouched in my test.

This post is a comparison built on a measurement. I put a small FastAPI app that streams ten tokens, one every 200 ms, behind each proxy and recorded when each token arrived. Then I validated production-style configs for all three, and ran Traefik's Docker label routing for real. The versions matter, so here they are: nginx 1.24.0 and Caddy 2.6.2 from Ubuntu 24.04's apt repos, Traefik 3.6.25 from the official Docker image, FastAPI 0.143.0 on uvicorn 0.53.0. The Ubuntu packages are older than current upstream releases (Docker Hub rate-limited me when I tried to pull newer nginx and Caddy images), so treat version-specific behavior as something to re-check on your own server.

If you haven't set up the server itself, do the VPS hardening checklist first. The proxy is the only thing that should be listening publicly.

Which reverse proxy should you pick for an AI app?

Pick by who will maintain it, not by benchmark.

If you can't decide, use Caddy. You can move to nginx or Traefik later without touching the app.

Does the proxy break token streaming?

Nginx does by default. The test: the app serves /stream as text/event-stream, yielding data: tokenN every 200 ms. A Python socket client records the arrival time of each token in milliseconds after sending the request. Direct to the app, the first token came at 101 ms and the last at 1912 ms.

import socket, sys, time

host, port = sys.argv[1], int(sys.argv[2])
path = sys.argv[3] if len(sys.argv) > 3 else "/stream"
s = socket.create_connection((host, port))
t0 = time.perf_counter()
s.sendall(f"GET {path} HTTP/1.1\r\nHost: {host}\r\nConnection: close\r\n\r\n".encode())
buf, seen, arrivals = b"", 0, []
while True:
    chunk = s.recv(65536)
    if not chunk:
        break
    buf += chunk
    n = buf.count(b"data: token")
    while seen < n:
        seen += 1
        arrivals.append(round((time.perf_counter() - t0) * 1000))
print(f"{path}: tokens={seen} first={arrivals[0]}ms last={arrivals[-1]}ms all={arrivals}")

Results, run on 2026-10-09 on a 4-vCPU sandbox (loopback, so absolute numbers are small; the shape is what matters):

Each row is a single run, not an average, so differences of a few milliseconds are noise. A second nginx run on the default config gave 2012 ms for every token, the same shape.

The nginx result matches its docs. proxy_buffering defaults to on, and the same page says buffering can be switched per response by an X-Accel-Buffering header with yes or no. Caddy's reverse_proxy docs say responses with text/event-stream and responses of unknown length are flushed immediately, which fits what I saw for both content types. I didn't find a matching statement in Traefik's docs for this behavior, so for Traefik I only have my measurement.

One oddity to be honest about: a first version of my test used a tiny hand-written Python HTTP server instead of uvicorn, and nginx passed that stream through without buffering. I didn't chase down why. The lesson is that the buffering you see depends on the upstream too, so test with the server you really run.

What does the nginx config look like for an AI app?

This is the shape I'd use for a FastAPI or Node app on port 8000. I ran nginx -t on it (with a throwaway self-signed certificate substituted for the Let's Encrypt paths) and it passed on nginx 1.24.0:

map $http_upgrade $connection_upgrade {
    default upgrade;
    ''      close;
}

server {
    listen 80;
    server_name ai.example.com;
    return 301 https://$host$request_uri;
}

server {
    listen 443 ssl;
    server_name ai.example.com;

    ssl_certificate     /etc/letsencrypt/live/ai.example.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/ai.example.com/privkey.pem;

    client_max_body_size 25m;

    location / {
        proxy_pass http://127.0.0.1:8000;
        proxy_http_version 1.1;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection $connection_upgrade;

        proxy_buffering off;
        proxy_read_timeout 300s;
    }
}

Why each line is there:

• proxy_buffering off is the streaming fix. It applies to the whole location. If the same location serves large downloads you'd rather buffer, leave it on and have only the streaming route send X-Accel-Buffering: no.
• proxy_read_timeout 300s matters for slow model calls. The default is 60 seconds between two successive reads, per nginx's docs, so a model that goes quiet for a minute while thinking gets cut off with a 504.
• proxy_http_version 1.1 plus the Upgrade and Connection lines let WebSockets through. Nginx's own docs list proxy_http_version default as 1.1 only from 1.29.7, and 1.0 before that, so setting it explicitly keeps your config portable.
• client_max_body_size 25m lifts the 1 MB default for file uploads to a document chatbot. Pick a number that matches your real limit.
• HTTP/2: on nginx 1.25.1 and later you write http2 on;. On my 1.24.0, that exact line failed nginx -t with unknown directive "http2". I left it out. If you target an older nginx, the parameter form listen 443 ssl http2; is what older guides use.

Certificates are on you. Certbot is the common route. Nginx has an ACME module, but its docs describe it as a separate package or source build, so it is not something you get in a default install.

What does the Caddy config look like?

Shorter. This validated with caddy validate on 2.6.2:

ai.example.com {
	request_body {
		max_size 25MB
	}

	reverse_proxy 127.0.0.1:8000 {
		health_uri /health
		health_interval 10s
		transport http {
			response_header_timeout 300s
		}
	}
}

Caddy gets the certificate on its own once DNS points at the server and ports 80 and 443 are reachable, per its automatic HTTPS docs. Let's Encrypt is the default CA with ZeroSSL as a fallback. Keep the data directory on a persistent volume, or you will re-issue certificates on every container rebuild and may hit rate limits.

The health_uri lines turn on active health checks. The docs say health_uri is what enables them and the default interval is 30 seconds. With one backend this only changes what Caddy reports, since there is nothing to fail over to. It becomes useful when you list two backends, which the zero-downtime deploys post uses.

Streaming needs nothing, but if you ever proxy a non-SSE stream and see buffering, flush_interval -1 disables response buffering and flushes after each write (documented in the same reverse_proxy page).

Honest downsides: Caddy has fewer ready-made recipes on forums than nginx, and its default behavior of managing certificates means you need to understand where it stores state before you wipe a volume.

What does Traefik look like on Docker?

Traefik is configured by labels on your containers. Here is the production-shaped Compose file. I ran docker compose config --quiet on it, which passed with the variables set and printed the two guard errors without them:

services:
  traefik:
    image: traefik:v3.6
    restart: unless-stopped
    command:
      - --providers.docker=true
      - --providers.docker.exposedbydefault=false
      - --entrypoints.web.address=:80
      - --entrypoints.web.http.redirections.entrypoint.to=websecure
      - --entrypoints.web.http.redirections.entrypoint.scheme=https
      - --entrypoints.websecure.address=:443
      - --certificatesresolvers.le.acme.email=${ACME_EMAIL:?set ACME_EMAIL}
      - --certificatesresolvers.le.acme.storage=/letsencrypt/acme.json
      - --certificatesresolvers.le.acme.tlschallenge=true
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - letsencrypt:/letsencrypt
      - /var/run/docker.sock:/var/run/docker.sock:ro

  api:
    image: myorg/ai-api:1.4.2
    restart: unless-stopped
    labels:
      - traefik.enable=true
      - traefik.http.routers.api.rule=Host(`${API_DOMAIN:?set API_DOMAIN}`)
      - traefik.http.routers.api.entrypoints=websecure
      - traefik.http.routers.api.tls.certresolver=le
      - traefik.http.services.api.loadbalancer.server.port=8000

volumes:
  letsencrypt:

I could not issue a real certificate in my sandbox, so the ACME parts are validated for syntax only, not run. The routing part I did run, using a test file with an HTTP entrypoint and two busybox containers. With exposedbydefault=false, the labelled container answered and the unlabelled one returned 404:

$ curl -H "Host: ai.localhost" http://127.0.0.1:8090/
hello-from-app
$ curl -o /dev/null -w "%{http_code}" -H "Host: hidden.localhost" http://127.0.0.1:8090/
404

Then I removed that flag and recreated Traefik. The container that never asked to be exposed answered:

$ curl -H "Host: hidden.localhost" http://127.0.0.1:8090/
should-not-be-routed

That is the main Traefik trap. Traefik's Docker provider docs list exposedByDefault as true by default, so any container on the same Docker network with a router label is reachable unless you turn it off. On a box where you also run an admin dashboard or a database UI with labels, that is how things get published by accident. Always set exposedbydefault=false and add traefik.enable=true per service.

The second trap is the socket. Traefik reads container events from /var/run/docker.sock. Its docs say unrestricted Docker API access is a security concern and that a compromised Traefik could give an attacker the host; they suggest a socket proxy or restricting access. The :ro flag in the example is the common habit, and as the hardening checklist notes, I haven't verified that it limits what the API allows, so don't treat it as a boundary. If that bothers you, Caddy or nginx with a static config has no such exposure.

What do AI apps need that a normal website doesn't?

Four things, the same in any proxy:

1. Streaming that isn't buffered. Test it with the probe script above after every proxy change. The fix list: nginx proxy_buffering off or the X-Accel-Buffering header; Caddy and Traefik worked as shipped here.
2. Long timeouts. Reasoning models and tool-using agents can be silent for a long time. Set read timeouts to match your slowest call, and make sure the client and any CDN in front also allow it. See streaming agents in real time for the app side.
3. Upload limits. Document chat apps hit nginx's small default body limit first. Set one number, on purpose.
4. Headers your app trusts. Rate limiting by client IP is useless if every request appears to come from the proxy. Pass X-Forwarded-For and make your app trust it only from the proxy's address. Caddy sets the forwarded headers itself according to its docs. In nginx you set them as shown above.

Webhook endpoints deserve the same care at the proxy: webhook-driven agents should sit behind authentication, and the proxy is a good place to add an IP allowlist for callers you know.

What should you pick, and what does it cost?

For most readers of this site, a single VPS running one API and maybe a vector database or n8n: Caddy. It is the least config, certificates are automatic, and streaming worked without a setting. For existing nginx shops, keep nginx and add the one buffering line. For container-heavy setups where services are added weekly, Traefik is the one whose design fits, provided you set exposedbydefault=false and think about the socket.

All three are free software, so the real cost is the server. If you want somewhere to try this, a small VPS is enough; I've used Hostinger's KVM plans in other posts, for instance the KVM 2 plan. Some links on this page are affiliate links; they cost you nothing and I only link tools I'd mention anyway. Hetzner, DigitalOcean and Linode run the same Ubuntu image and every command here works the same on them. A full FastAPI-behind-nginx walkthrough is in deploying a FastAPI LLM app with Docker and nginx. A worked example of an app behind Caddy is in n8n on a VPS with Docker, and the Next.js on a VPS guide covers the nginx route.

What I did not verify

• Current nginx and Caddy releases. I tested the Ubuntu 24.04 packages, nginx 1.24.0 and Caddy 2.6.2, because Docker Hub rate-limited the newer images. Behavior, especially around HTTP/2 and defaults, may differ on current versions.
• Real certificate issuance for any of the three. Everything involving Let's Encrypt is syntax-checked only.
• HTTP/2 or HTTP/3 streaming. My probe used HTTP/1.1 on loopback.
• Behavior behind a CDN or a cloud load balancer, which can add its own buffering and timeouts.
• Memory use under load. I did not run a fair benchmark, so I'm not comparing resource use.