Background Jobs for LLM Apps: Celery vs RQ vs arq (Same Job, Three Ways)

Your summarise endpoint worked fine with a 2 KB test document. Then a user uploads a 90-page PDF, the model call takes 40 seconds, and your proxy gives up at 30. The fix is to stop doing the slow work inside the request. This post compares Celery, RQ and arq for LLM background jobs by writing the same stubbed task in each, and being clear about which ones I actually ran.

Which of these did I actually run?

Honest scorecard, dated 2026-10-31, macOS, Python 3.14, packages installed from PyPI into a scratch virtualenv:

So the Celery code and output below are real, and the RQ and arq code is unrun against a live Redis. I read their docs the same day. Treat those two snippets as templates and run them against a real Redis before you copy them. The "model" in every case is a stub that sleeps 0.2 seconds; no LLM or API key was involved.

What does an LLM job need that a normal job doesn't?

Four things make model calls awkward for a queue:

• Long and variable runtime. Seconds to minutes, so default timeouts matter.
• Flaky upstreams. 429 and 5xx errors are normal. Retries are not optional, but they must only fire on retryable errors. See retry and backoff for LLM APIs.
• Real money per run. A job that quietly runs five times costs five times. See token budgeting.
• Side effects. An agent job might send an email. If the queue can run it twice, the tool must tolerate that; see idempotent tool calls.

Hold those four in mind and most of the comparison below is about defaults.

What does the job look like in Celery?

Celery is the heavyweight: many brokers, result backends, routing, schedules. Here the same task, with retries on a connection error and late acknowledgement:

from celery import Celery
import time

app = Celery("q2", broker="filesystem://", backend="file:///tmp/results",
             broker_transport_options={...})   # folders omitted for brevity
app.conf.task_acks_late = True

@app.task(bind=True, autoretry_for=(ConnectionError,), retry_backoff=0.1, max_retries=3)
def summarise(self, doc_id):
    time.sleep(0.2)                       # stub for the model call
    if doc_id == "flaky" and self.request.retries < 2:
        raise ConnectionError("stub: upstream 503")
    return f"summary of {doc_id} (retries={self.request.retries})"

In production you would use redis:// or RabbitMQ as the broker; the filesystem transport is what I used to get a real worker running without a Redis server. I started the worker with celery -A celery_app worker --pool=solo and enqueued three jobs, one of which fails twice on purpose. The client printed:

summary of a (retries=0)
summary of b (retries=0)
summary of flaky (retries=2)
all done in 3.53s

And the worker log showed the retries happening:

Task celery_app.summarise[844bdfec-...] retry: Retry in 1s: ConnectionError('stub: upstream 503')
Task celery_app.summarise[844bdfec-...] retry: Retry in 0s: ConnectionError('stub: upstream 503')
Task celery_app.summarise[844bdfec-...] succeeded in 0.206s: 'summary of flaky (retries=2)'

The logged delays (1s, then 0s) are not a clean doubling from 0.1 seconds, and I did not investigate why; set retry_backoff, retry_backoff_max and retry_jitter deliberately and watch your own logs.

Things the Celery docs say that matter for LLM work, from the task user guide as read on 2026-10-31:

• By default no exceptions are auto-retried; you opt in with autoretry_for.
• By default the message is acknowledged just before the task runs, so a started task will not be re-delivered after a crash. Setting acks_late acknowledges after execution instead, which means a worker crash mid-task can run it more than once. The docs tie this to idempotency: the worker cannot know whether a task is idempotent, which is why early acknowledgement is the default.
• Time limits can kill the process by force. The docs recommend manual timeouts on I/O and using limits to catch the cases where those are missing. Set a client timeout on the model call yourself.
• Tasks do not ignore results by default when a result backend is configured, so decide whether you need one at all.

Late acknowledgement is the choice to make deliberately. Default (early ack) means a crash loses the job. acks_late means a crash repeats it. For a job that costs money or sends mail, pick the one your job can survive, and make the job safe for the other.

What does the job look like in RQ?

RQ is the small one. A job is any importable function; a queue is a Redis list. This is a template and I did not run it against Redis:

# tasks.py
import time

def summarise(doc_id):
    time.sleep(0.2)                       # stub for the model call
    return f"summary of {doc_id}"

# enqueue.py  (unrun against a live Redis)
from redis import Redis
from rq import Queue
from tasks import summarise

q = Queue("llm", connection=Redis())
job = q.enqueue(summarise, "doc1", job_timeout=120, result_ttl=3600)
print(job.get_status())

Start a worker with rq worker llm. What the RQ docs state (read 2026-10-31): it needs Redis (5 or newer) or Valkey (7.2 or newer); job_timeout sets how long a job may run before it is interrupted and marked failed; and result_ttl controls how long results are kept, defaulting to 500 seconds. That default surprises people: fetch a result 10 minutes later and it is gone. If users poll for results, store the output in your database from inside the job and treat the Redis result as a convenience.

The retry options appear on the docs page as CLI flags (--retry-max, --retry-interval) and a separate section I did not read, so check how retries are configured per job in your version before you rely on them.

RQ jobs are plain synchronous functions. If your model client is async, you call asyncio.run inside the job. That is fine for one call per job and awkward if you want hundreds of concurrent calls in one worker.

What does the job look like in arq?

arq is built on asyncio, which suits I/O-bound model calls: one worker process can hold many jobs in flight at once. Template, also unrun against a live Redis:

# worker.py
import asyncio
from arq.worker import Retry

async def summarise(ctx, doc_id):
    await asyncio.sleep(0.2)                # stub for the model call
    if ctx["job_try"] < 3 and doc_id == "flaky":
        raise Retry(defer=2)                # run again in 2 seconds
    return f"summary of {doc_id} (try {ctx['job_try']})"

class WorkerSettings:
    functions = [summarise]
    job_timeout = 120
    max_tries = 3
    max_jobs = 20

Enqueue with pool = await create_pool(RedisSettings()) then await pool.enqueue_job("summarise", "doc1"), and start the worker with arq worker.WorkerSettings. From the arq docs (read 2026-10-31): the library is "job queues and RPC in python with asyncio and redis"; the default job_timeout is 300 seconds; max_tries defaults to 5; raising Retry retries the job if max_tries has not been reached. The docs also say "Jobs may be called more than once!" and that arq "favours multiple times over zero times." That is the clearest statement of delivery semantics of the three. The docs page shows version 0.28.0 and a release dated 2026-04-16; it does not describe the project's maintenance status, so check the repository's recent activity yourself before betting a product on it.

The default of five tries on a job that calls a paid API deserves a second look. If a bug makes the job raise after the model call returns, you pay for five calls. Set max_tries low, and make the model call the last thing that can fail, or cache its result.

How do they compare for LLM work?

The Celery async point is from general knowledge of how Celery tasks run, not from this session's reading, so verify it for your version.

Which one should you pick?

A short, opinionated rule:

• Already on FastAPI with async everything, mostly I/O to model APIs: arq. One worker can keep dozens of calls in flight, and the code reads like the rest of your app.
• Small team, a few job types, no need for routing or schedules: RQ. Fewer concepts, less to misconfigure. Accept one job at a time per worker process, and run more processes.
• You need scheduled jobs, multiple queues with priorities, workflows, or RabbitMQ: Celery. Pay the configuration cost once, and read the acknowledgement and timeout sections before launch.
• Jobs run for hours, wait on humans, or must survive restarts mid-run: none of these. Use a workflow engine; see durable agents with Temporal, or checkpoint inside the job as in agent checkpointing.
• Bulk work where nobody waits for the answer: consider the provider's batch API before building a queue; see Anthropic's Batch API for cost.

What settings should every LLM job set?

1. A client timeout on the model call, shorter than the queue's job timeout. The queue timeout is a backstop.
2. A retry list limited to retryable errors (timeouts, 429, 5xx). Never retry a 400 or a content-policy refusal.
3. Small arguments. Pass a document id, not the document.
4. Results in your database. Queue result stores expire or get cleaned; your database is the source of truth. Update a jobs row from inside the job: queued, running, done, failed.
5. Idempotent side effects. Key them on the job id, so a duplicate run changes nothing.
6. One place to look. Log the job id and trace id with every model call; see tracing LLM calls with OpenTelemetry.
7. A process manager for workers. They need restarts and logs; see systemd versus Docker for Python AI workers.

What the web endpoint looks like

All three share the same shape on the web side: accept, enqueue, return an id, let the client poll or subscribe.

@app.post("/summaries", status_code=202)
async def create_summary(doc_id: str):
    job_id = enqueue_summary(doc_id)          # Celery .delay / RQ enqueue / arq enqueue_job
    return {"job_id": job_id, "status_url": f"/summaries/{job_id}"}

Return 202, not 200, and give the client somewhere to ask. For streaming progress instead of polling, see streaming agents with LangGraph and the general FastAPI production patterns.

What I did not verify

• RQ and arq behaviour under a real Redis, including retry timing, worker restarts and what happens to in-flight jobs when a worker is killed.
• Celery with Redis or RabbitMQ; I used the filesystem broker, so throughput and failure behaviour will differ.
• Any benchmark between the three. I measured nothing about speed, and the 3.53 second total above mostly reflects retry delays, not throughput.
• Real model latencies. Every job slept for 0.2 seconds.