Self-Host Open WebUI with vLLM or Ollama Behind One API Gateway

One endpoint, separate keys per person, and a web UI on top: that is the whole goal of this setup. A LiteLLM gateway gives you that on a VPS. Open WebUI, your scripts and your teammates all talk to one OpenAI-style address, and the gateway forwards each request to Ollama on the same box or to a vLLM server somewhere else.

Why bother? Because the model servers themselves can't tell your users apart. Ollama has no authentication and vLLM has a single shared key. The moment a second person or a second app needs access, you want revocable keys, a model allowlist and a spending cap.

I checked the settings below against the LiteLLM, vLLM, Ollama and Open WebUI docs on 2026-10-09 (some pages were read through a summarising fetch, so verify exact wording when something matters). I validated the Compose file with docker compose config and ran a small config checker. I did not deploy this stack or send a request through LiteLLM, so every "unrun" label below means exactly that. If you haven't built the base stack yet, start with the Ollama and Open WebUI sizing guide, which this post builds on.

What does the gateway layout look like?

Five containers, one public door:

The big model, if you have one, lives outside the stack. vLLM is a high-throughput serving engine, and its docs say it runs on CPU as well as GPUs (x86 and ARM, with prebuilt wheels and vllm/vllm-openai-cpu Docker images). On a small VPS I'd still treat CPU vLLM as a curiosity: the memory-bandwidth ceiling I worked out in the sizing post applies to any engine. So the practical layout is a cheap always-on VPS for the gateway and UI, plus vLLM on a GPU machine you start when needed. The gateway doesn't care which side a model lives on.

What does the Docker Compose file look like?

Create a folder and three files. First the .env (generate every secret with openssl rand -hex 32, and hex matters here because the Postgres password is embedded in a URL):

CHAT_DOMAIN=chat.yourdomain.com
API_DOMAIN=api.yourdomain.com
OLLAMA_VERSION=<pin from Docker Hub>
LITELLM_VERSION=<pin a stable tag from the LiteLLM releases>
OPEN_WEBUI_VERSION=<pin a vX.Y.Z tag>
POSTGRES_PASSWORD=<openssl rand -hex 32>
LITELLM_MASTER_KEY=sk-<openssl rand -hex 32>
OPEN_WEBUI_LITELLM_KEY=<created later, see below>
WEBUI_SECRET_KEY=<openssl rand -hex 32>

The master key must start with sk-, per LiteLLM's virtual keys page. The Caddyfile is two blocks, one per hostname, both pointing at DNS A records on your server:

{$CHAT_DOMAIN} {
	reverse_proxy open-webui:8080
}

{$API_DOMAIN} {
	reverse_proxy litellm:4000
}

And the compose.yaml:

services:
  postgres:
    image: postgres:16
    restart: unless-stopped
    environment:
      POSTGRES_USER: litellm
      POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:?set POSTGRES_PASSWORD in .env}
      POSTGRES_DB: litellm
    volumes:
      - pg_data:/var/lib/postgresql/data
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U litellm -d litellm"]
      interval: 5s
      timeout: 5s
      retries: 10

  ollama:
    image: ollama/ollama:${OLLAMA_VERSION:?set OLLAMA_VERSION in .env}
    restart: unless-stopped
    environment:
      OLLAMA_KEEP_ALIVE: 30m
      OLLAMA_NUM_PARALLEL: 1
    volumes:
      - ollama:/root/.ollama

  litellm:
    image: docker.litellm.ai/berriai/litellm:${LITELLM_VERSION:?set LITELLM_VERSION in .env}
    restart: unless-stopped
    command: ["--config", "/app/config.yaml"]
    depends_on:
      postgres:
        condition: service_healthy
    environment:
      LITELLM_MASTER_KEY: ${LITELLM_MASTER_KEY:?set LITELLM_MASTER_KEY in .env}
      DATABASE_URL: postgresql://litellm:${POSTGRES_PASSWORD}@postgres:5432/litellm
      VLLM_BASE_URL: ${VLLM_BASE_URL:-http://ollama:11434}
      VLLM_UPSTREAM_KEY: ${VLLM_UPSTREAM_KEY:-none}
    volumes:
      - ./litellm_config.yaml:/app/config.yaml:ro

  open-webui:
    image: ghcr.io/open-webui/open-webui:${OPEN_WEBUI_VERSION:?set OPEN_WEBUI_VERSION in .env}
    restart: unless-stopped
    depends_on:
      - litellm
    environment:
      ENABLE_OLLAMA_API: "False"
      OPENAI_API_BASE_URL: http://litellm:4000/v1
      OPENAI_API_KEY: ${OPEN_WEBUI_LITELLM_KEY:?set OPEN_WEBUI_LITELLM_KEY in .env}
      WEBUI_SECRET_KEY: ${WEBUI_SECRET_KEY:?set WEBUI_SECRET_KEY in .env}
      ENABLE_SIGNUP: "False"
    volumes:
      - open-webui:/app/backend/data

  caddy:
    image: caddy:2
    restart: unless-stopped
    depends_on:
      - open-webui
      - litellm
    ports:
      - "80:80"
      - "443:443"
    environment:
      CHAT_DOMAIN: ${CHAT_DOMAIN:?set CHAT_DOMAIN in .env}
      API_DOMAIN: ${API_DOMAIN:?set API_DOMAIN in .env}
    volumes:
      - ./Caddyfile:/etc/caddy/Caddyfile:ro
      - caddy_data:/data
      - caddy_config:/config

volumes:
  pg_data:
  ollama:
  open-webui:
  caddy_data:
  caddy_config:

Here is what I ran. With a dummy .env, docker compose config --quiet printed nothing and exited 0. With an empty env file it failed with nine messages, each naming a missing variable, for example:

error while interpolating services.litellm.environment.LITELLM_MASTER_KEY: required variable LITELLM_MASTER_KEY is missing a value: set LITELLM_MASTER_KEY in .env
error while interpolating services.open-webui.image: required variable OPEN_WEBUI_VERSION is missing a value: set OPEN_WEBUI_VERSION in .env

That's the guard working: the stack refuses to start with blank secrets. I did not start any containers.

A few choices worth knowing about:

• Only Caddy has a ports key. Docker's docs say a published port without a host address binds to all addresses and bypasses ufw rules, so the model servers, Postgres and the gateway are reachable only on the Compose network. The hardening checklist explains that trap.
• ENABLE_OLLAMA_API: "False" makes Open WebUI skip its direct Ollama connection, so every model it shows comes through the gateway. Open WebUI's reference lists this as a boolean defaulting to true. It also lists ENABLE_SIGNUP and ENABLE_OLLAMA_API as persistent settings: after the first launch, the value saved in its database can win over the variable. So change them in the admin settings afterwards, or set ENABLE_PERSISTENT_CONFIG to false if you want the environment to always win (at the price of admin-UI changes not being saved).
• The LiteLLM image path (docker.litellm.ai/berriai/litellm) is the one in LiteLLM's Docker quick start on the day I read it. Pin a tag.

How do you configure LiteLLM for Ollama and vLLM?

One litellm_config.yaml. The model prefixes come from LiteLLM's provider pages: ollama_chat/ for Ollama chat (the page says it recommends it over ollama/ for better responses) and hosted_vllm/ for a vLLM OpenAI-compatible server.

model_list:
  - model_name: chat-small
    litellm_params:
      model: ollama_chat/llama3.1:8b
      api_base: http://ollama:11434

  - model_name: chat-big
    litellm_params:
      model: hosted_vllm/Qwen/Qwen2.5-1.5B-Instruct
      api_base: os.environ/VLLM_BASE_URL
      api_key: os.environ/VLLM_UPSTREAM_KEY

general_settings:
  master_key: os.environ/LITELLM_MASTER_KEY
  database_url: os.environ/DATABASE_URL

The model_name is what your users see, so chat-small and chat-big stay stable when you swap the model behind them. That is half the value of a gateway. The vLLM model here is the small Qwen model from vLLM's own quick start, as a placeholder; use whichever model your vLLM server actually loaded, and the same name it reports.

On the vLLM side, the quick start says vllm serve <model> listens on port 8000 and that --api-key (or the VLLM_API_KEY variable) makes the server check a key, with several keys allowed for rotation. That shared key is exactly what you put in VLLM_UPSTREAM_KEY. Only the gateway ever holds it. If the GPU box is reachable over the public internet, put TLS in front of it and use an https:// URL in VLLM_BASE_URL, otherwise that key crosses the network in plain text.

I ran a checker over these two files before writing this section. It confirms every model has an api_base, flags plain http to anything that isn't a private or Compose address, requires a master key and database, and complains if any service other than Caddy publishes a port:

model chat-small   -> ollama_chat/llama3.1:8b                  base=http://ollama:11434
model chat-big     -> hosted_vllm/Qwen/Qwen2.5-1.5B-Instruct   base=os.environ/VLLM_BASE_URL
service caddy publishes ['80:80', '443:443']
OK

It is a lint, not proof the gateway works. The config shape follows the docs, but I haven't started LiteLLM.

How do you create per-person API keys?

Bring the stack up, then use the master key once to mint limited keys. Unrun:

docker compose up -d
docker compose logs --tail 50 litellm

Create the key Open WebUI will use. LiteLLM's virtual keys page shows /key/generate taking models, and references max_budget, rpm_limit and duration as key fields (the summary I read told me to confirm exact semantics in the API reference, so do that before relying on a budget):

curl -X POST https://api.yourdomain.com/key/generate \
  -H "Authorization: Bearer <your master key>" \
  -H "Content-Type: application/json" \
  -d '{"models": ["chat-small", "chat-big"], "rpm_limit": 60, "duration": "365d", "metadata": {"owner": "open-webui"}}'

The response contains a new key starting with sk-. Put it in .env as OPEN_WEBUI_LITELLM_KEY, then recreate only that container so the new value is picked up:

docker compose up -d open-webui

That ordering is the chicken-and-egg part of this design. Open WebUI can't start meaningfully until a key exists, and the key can't exist until LiteLLM is up. Do it in two passes the first time.

Now a key per person or per app, each limited to what they need. A contractor's script might get chat-small only, a 20 requests-per-minute limit and a 30-day duration. When they leave, delete their key and nothing else changes. Compare that with sharing one Ollama address with everyone.

Test the boundaries yourself (unrun): call with no key (expect a 401-style rejection), call with a key for a model it isn't allowed (expect a refusal), and call with the right one. Those three requests tell you more than any config review.

What do you point your own code at?

The same gateway, using any OpenAI-compatible client. Unrun, with a placeholder key:

from openai import OpenAI

client = OpenAI(base_url="https://api.yourdomain.com/v1", api_key="sk-your-virtual-key")
reply = client.chat.completions.create(
    model="chat-small",
    messages=[{"role": "user", "content": "One sentence: what does a reverse proxy do?"}],
)
print(reply.choices[0].message.content)

If you later add a hosted API model to the same model_list, your app code doesn't change. My LiteLLM and OpenRouter routing guide covers fallbacks and cost-based routing, which this gateway can do too.

What can go wrong?

Is a gateway worth the extra moving parts?

For one person on one laptop, no. Run Ollama and Open WebUI and stop there. The gateway earns its place when a second user, a second app or a second backend appears, because that's when "who used what" and "turn off just that one" start to matter. It adds a database to back up (virtual keys live in Postgres, so a dump of the litellm database is your key store) and one more thing to patch.

Where to run it: the stack needs very little CPU, since the gateway only forwards requests. If you want a single always-on server for the gateway and UI, a small plan is enough; Hostinger's KVM 2 (2 vCPU, 8 GB RAM, 100 GB NVMe) was listed at $8.99 a month promotional and $14.99 renewal on their VPS page on 2026-10-09, and you can get KVM 2 here. Some links on this page are affiliate links; they cost you nothing and I only link tools I'd mention anyway. Hetzner, DigitalOcean or a home server work the same. If you're deciding whether to self-host at all, the local vs cloud LLM privacy guide covers that choice, and Nginx vs Caddy vs Traefik compares the proxy used above. If you also want the 8B model on the same machine, size up using the sizing guide instead, and read how quantization changes model quality before picking a file.