Zero-Downtime Deploys with Docker Compose for a Small AI App

Zero-downtime deploys with Docker Compose come down to one idea: start the new container beside the old one, send traffic only after the new one passes a health check, then remove the old one. Plain docker compose up -d does the opposite, and for an AI app that loads a model or warms a cache on startup, the gap is long enough that users see errors.

I measured both on 2026-10-09. A client sent about 20 requests per second while I deployed. With plain up -d, 88 of 443 requests failed. With the script below, 0 of 551 failed. The rest of this post is the setup, the script, the traps I hit, and what I did not test.

Everything ran on Docker Engine 29.8.2 with Compose v5.6.0 and Traefik 3.6.25 as the proxy. The "app" is a busybox web server with an artificial 4-second startup, standing in for a model load. That is a stand-in, not a real model server, so your timings will differ. If you're choosing a proxy first, read nginx vs Caddy vs Traefik for AI apps.

Why does docker compose up cause downtime?

Because with one replica, there is a moment with zero. Compose notices the service config changed (a new image tag or environment value), stops the old container, removes it and creates the new one. The proxy has nothing healthy to route to until the new container is up and passing its check.

Here is the measurement. The client (below) requests the proxy every 50 ms or so and counts what comes back. I started it, then ran APP_VERSION=v2 docker compose up -d --wait:

total requests: 443
  200 v1: 191
  200 v2: 164
  HTTP 404: 86
  HTTP 502: 2
first error at t = 10.4

That is 88 failures, about 20 percent of the run. The 404s are Traefik saying no route matches, because the old container was gone and the new one wasn't registered yet. The 502s are requests that were routed to a container that was going away.

The client, if you want to reproduce it:

import collections, sys, time, urllib.request

url, seconds = sys.argv[1], float(sys.argv[2])
counts, first_error, t0 = collections.Counter(), None, time.time()
while time.time() - t0 < seconds:
    t = time.time() - t0
    try:
        with urllib.request.urlopen(url, timeout=2) as r:
            counts[f"200 {r.read().decode().strip()}"] += 1
    except urllib.error.HTTPError as e:
        counts[f"HTTP {e.code}"] += 1
        first_error = first_error or round(t, 1)
    except Exception as e:
        counts[f"ERR {type(e).__name__}"] += 1
        first_error = first_error or round(t, 1)
    time.sleep(0.05)
print("total requests:", sum(counts.values()))
for k, v in sorted(counts.items()):
    print(f"  {k}: {v}")
print("first error at t =", first_error)

What does the Compose file need?

Three things: a health check that means "ready for traffic", a proxy that only routes to healthy containers, and no fixed container_name or published port on the app, because two copies must coexist. This is the file I tested (with the APP_VERSION guard so a missing value fails loudly):

services:
  proxy:
    image: traefik:v3.6
    restart: unless-stopped
    command:
      - --providers.docker=true
      - --providers.docker.exposedbydefault=false
      - --entrypoints.web.address=:8091
      - --log.level=ERROR
    ports:
      - "8091:8091"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro

  api:
    image: busybox:1.37
    restart: unless-stopped
    environment:
      APP_VERSION: ${APP_VERSION:?set APP_VERSION}
      STARTUP_DELAY: ${STARTUP_DELAY:-4}
    command:
      - sh
      - -c
      - |
        sleep $$STARTUP_DELAY
        mkdir -p /www
        echo "$$APP_VERSION" > /www/index.html
        echo ok > /www/health
        exec httpd -f -p 8000 -h /www
    healthcheck:
      test: ["CMD", "wget", "-q", "-O", "/dev/null", "http://127.0.0.1:8000/health"]
      interval: 1s
      timeout: 2s
      retries: 3
      start_period: 8s
    labels:
      - traefik.enable=true
      - traefik.http.routers.api.rule=PathPrefix(`/`)
      - traefik.http.routers.api.entrypoints=web
      - traefik.http.services.api.loadbalancer.server.port=8000

docker compose config --quiet parsed it. To adapt it to your app, replace the image, drop the fake command, and change the health check to something your image has. Python slim images usually have no curl, so a check using the standard library works (this line is an untested template):

    healthcheck:
      test: ["CMD", "python", "-c", "import urllib.request; urllib.request.urlopen('http://127.0.0.1:8000/health', timeout=2)"]
      interval: 5s
      timeout: 3s
      retries: 3
      start_period: 60s

Make /health mean something. For a model server, return 200 only after the model is loaded, not as soon as the web framework is up. A health check that passes too early sends traffic to a container that can't answer.

Traefik is the proxy here because its Docker provider watches container state. In my run it only routed to the new container once it was healthy, and the zero-error result below is evidence of that. If you use Caddy or nginx instead, you need a different mechanism: Caddy's active health checks (health_uri) against two named backends, or a config reload that points at the new container after you've checked it yourself. I didn't test those here. Also remember what the hardening checklist says about the Docker socket: Traefik reading it is a trade-off, not free.

What is the rolling deploy script?

Start another copy with the new config, wait for healthy, remove the old one. I ran this exact file (bash -n clean, then used for the deploy test):

#!/usr/bin/env bash
# Usage: APP_VERSION=v3 ./rollout.sh api
set -euo pipefail

SERVICE="${1:?service name}"
COMPOSE="docker compose -f $(dirname "$0")/compose.yaml"
TIMEOUT="${ROLLOUT_TIMEOUT:-120}"

old_ids=$($COMPOSE ps -q "$SERVICE")
[ -n "$old_ids" ] || { echo "no running $SERVICE container to replace"; exit 1; }
count=$(echo "$old_ids" | wc -l)

echo "starting new $SERVICE next to $count old one(s)"
$COMPOSE up -d --no-deps --no-recreate --scale "$SERVICE=$((count + 1))" "$SERVICE"

new_id=$(comm -13 <(echo "$old_ids" | sort) <($COMPOSE ps -q "$SERVICE" | sort) | head -n1)
[ -n "$new_id" ] || { echo "could not find the new container"; exit 1; }

echo "waiting for $new_id to become healthy (max ${TIMEOUT}s)"
for _ in $(seq 1 "$TIMEOUT"); do
  status=$(docker inspect -f '{{if .State.Health}}{{.State.Health.Status}}{{else}}none{{end}}' "$new_id")
  [ "$status" = "healthy" ] && break
  sleep 1
done
if [ "$status" != "healthy" ]; then
  echo "new container never became healthy ($status); removing it, old version keeps serving"
  docker stop "$new_id" >/dev/null && docker rm "$new_id" >/dev/null
  exit 1
fi

echo "healthy; retiring old container(s)"
for id in $old_ids; do
  docker stop -t 30 "$id" >/dev/null
  docker rm "$id" >/dev/null
done
echo "done"

How it works: --scale api=2 --no-recreate leaves the running container untouched and adds one more created from the new config (the new APP_VERSION is read from your environment at that moment). It then polls Docker's health status for that new container, and only then stops the old one. The -t 30 gives the old app up to 30 seconds after SIGTERM to finish what it's doing before Docker kills it.

I ran it with the client going. Output of the script:

starting new api next to 1 old one(s)
waiting for c4150ec8a013... to become healthy (max 120s)
healthy; retiring old container(s)
done

And the client:

total requests: 551
  200 v2: 331
  200 v3: 220
first error at t = None

Zero failures, with the version flipping from v2 to v3 mid-run. The new container needed about four seconds to start, and users never saw it.

What happens when the new version is broken?

The old one keeps serving. I tested it by deploying a version with a 100-second startup delay and a 15-second timeout:

waiting for d7cffa34... to become healthy (max 15s)
new container never became healthy (unhealthy); removing it, old version keeps serving

A request to the proxy afterwards still returned v3. That is the main reason to prefer this over up -d: with plain Compose, a release that crashes on boot takes the site down, and you roll back after the fact.

Be clear about the limit. This only catches releases that fail the health check. A release that starts fine and then gives wrong answers, or burns tokens in a loop, passes. For that you need evals and a rollback habit; see production readiness for AI agents.

What traps did I hit?

• start_period too short. My first attempt used start_period: 1s with a 4-second startup, interval: 1s and retries: 3. The container was marked unhealthy before the app began listening, and docker compose up --wait exited with container rollout-api-1 is unhealthy. Docker's Dockerfile reference says probe failures during the start period don't count toward the maximum retries, but that once a check succeeds during the start period the container counts as started and later failures do count. So set it above your real warmup (model download, weight loading, index build), and make /health fail until the model is really loaded.
• Compose variables on every command. After the deploy, running docker compose ps in a fresh shell failed with required variable APP_VERSION is missing a value. The guard is good for deploys, but put the current value in a .env file next to the compose file so other commands still work. The script then has to write the new tag to that file before it scales up; I passed it as an environment variable in the test instead.
• Name clashes. A fixed container_name or ports: "8000:8000" on the app blocks the second copy. Publish only the proxy's ports.
• Memory. Two copies of a model server need twice the RAM during the overlap. On a small VPS running a local model, this alone can make the new container get OOM-killed. See sizing a private ChatGPT on a VPS. For apps that call hosted APIs, the overlap is cheap.
• State. Two versions running at once must share a database schema that both can read. Migrate in two steps: add columns first, deploy, then remove old ones later.

Should you automate this?

Once it works by hand, yes, but keep it dull. A GitHub Actions job can SSH in, pull the image, set the new tag and run the script. Keep the previous image tag in the output so rollback is the same script with the old tag. Don't add auto-deploy on every push until the health check is something you trust. For the app image itself, the FastAPI LLM app with Docker and nginx post covers the Dockerfile side.

If you need a box to try this on, any small VPS works; I've used Hostinger for other guides, for example the KVM 2 plan. Some links on this page are affiliate links; they cost you nothing and I only link tools I'd mention anyway. Hetzner, DigitalOcean and Linode run the same commands.

What I did not verify

• A real model server. The app was busybox with a sleep, so memory pressure, model load time and SIGTERM handling of a real FastAPI or vLLM process are untested.
• Streaming responses surviving the cutover. In-flight long requests on the old container are cut at the stop timeout if your app doesn't finish them first.
• Caddy or nginx as the proxy for this pattern.
• Traefik's exact rule for hiding unhealthy containers. I observed the behavior. Traefik's Docker provider reference does describe an allowEmptyServices option (default false) that creates a service "regardless of the healthiness" of the containers, which is consistent with what I saw, but it is not a full statement of the rule, so confirm in your version.
• Multiple hosts. This is one machine; it is not a replacement for an orchestrator.