Prompt A/B Testing and Canary Rollout for LLM Features

You rewrote the system prompt, your eval set says it's better, and now you have to ship it to real users without finding out the hard way that it isn't. The mechanics are plain: send a small, sticky slice of users to the new version, watch a few numbers, widen in stages, and have a rule that rolls back without a debate. The part people get wrong is the rule.

This post gives you the code for sticky bucketing, a sample-size calculator for both a quality metric and a guardrail metric, and a simulation comparing two rollback rules. All of it ran on 2026-10-30 with Python's standard library. The traffic volume is an assumption I picked for illustration; the arithmetic is real.

If you need the statistics background for comparing two prompts, A/B testing prompts in production covers metrics and significance. This post is about the rollout around the test: who sees what, in what order, and when you stop. Prompts should be versioned first, see prompt versioning in production.

Canary or A/B test: which one do you need?

They answer different questions, and mixing them up wastes weeks.

Most prompt changes need a canary. Fewer need an A/B test, because the lifts are small and the sample sizes below are large. If your change is a safety or cost fix, "no worse on quality, better on cost" is a canary question.

How do you assign users to a prompt version without flip-flopping?

Hash. Never use a random number per request, or the same user sees version A on one message and B on the next, and your multi-turn conversations become a mess.

import hashlib

def bucket(experiment: str, user_id: str) -> float:
    """Stable position in [0, 100) for this user in this experiment."""
    h = hashlib.sha256(f"{experiment}:{user_id}".encode()).hexdigest()
    return int(h[:8], 16) % 10000 / 100

def in_canary(experiment: str, user_id: str, percent: float) -> bool:
    return bucket(experiment, user_id) < percent

I ran this on 100,000 synthetic user IDs with the experiment name prompt-v7 and widened the ramp:

ramp to  1%:   1001 users (1.00%); kept from previous stage: 0/0
ramp to  5%:   5075 users (5.08%); kept from previous stage: 1001/1001
ramp to 25%:  25227 users (25.23%); kept from previous stage: 5075/5075
ramp to 50%:  50259 users (50.26%); kept from previous stage: 25227/25227
overlap of two different 5% experiments: 252 users (independent expectation: 250)
same 5% with a new salt: 246 users carried over (a new salt reshuffles everyone)

What this proves:

• Ramping up is safe. Everyone in the 1 percent slice is still in the 5 percent slice, and so on. Nobody gets switched back to the old prompt mid-conversation.
• Experiments are independent. Two different experiment names overlap about as much as chance predicts, so a prompt test and a model test don't contaminate each other.
• The experiment name is a salt. Change it and everyone is reshuffled (246 carried over, close to the 250 chance level). Treat the name as part of the experiment's identity and never edit it mid-flight.

Pick the ID with care. Use a user or conversation ID that is stable across sessions. If you bucket by session, a returning user switches versions. If you bucket by request, you lose multi-turn consistency. For logged-out traffic a long-lived cookie is the usual compromise. And bucket on the server so users can't opt themselves in.

What stages should the ramp have?

A reasonable shape: shadow, 1 percent, 5 percent, 25 percent, 50 percent, 100 percent. Not a rule, just a shape.

• Shadow (optional). Run the new prompt on a copy of live inputs and log the output, but show users the old one. It costs extra tokens and catches crashes, format failures and cost blow-ups before any user sees them. It can't tell you how users react.
• Each stage has an entry criterion and an exit criterion. Write both down before you start. Entry: the previous stage passed its guardrails with at least N requests. Exit: any guardrail trips its rollback rule.
• Keep the old version deployable. Rollback should be a config flag, not a revert and redeploy. If the prompt lives in code, you've made rollback slower than the incident.

How many requests does each stage need?

Here are two calculators. The first is for a quality lift (two-sided, 80 percent power, alpha 0.05). The second is for a guardrail regression (one-sided, same power and alpha). Both use the normal approximation for two proportions.

import math
from statistics import NormalDist

Z = NormalDist()

def n_two_prop(p1, p2, alpha=0.05, power=0.80):
    za, zb = Z.inv_cdf(1 - alpha / 2), Z.inv_cdf(power)
    pbar = (p1 + p2) / 2
    num = (za * math.sqrt(2 * pbar * (1 - pbar)) + zb * math.sqrt(p1 * (1 - p1) + p2 * (1 - p2))) ** 2
    return math.ceil(num / (p1 - p2) ** 2)

def n_one_sided(p1, p2, alpha=0.05, power=0.80):
    za, zb = Z.inv_cdf(1 - alpha), Z.inv_cdf(power)
    pbar = (p1 + p2) / 2
    num = (za * math.sqrt(2 * pbar * (1 - pbar)) + zb * math.sqrt(p1 * (1 - p1) + p2 * (1 - p2))) ** 2
    return math.ceil(num / (p1 - p2) ** 2)

Output (requests per arm):

Quality metric, baseline 80% task success, two-sided
  detect  10 point lift (80% -> 90%):     199
  detect   5 point lift (80% -> 85%):     906
  detect   2 point lift (80% -> 82%):   6,039
  detect   1 point lift (80% -> 81%):  24,641

Guardrail, baseline 2% error rate, one-sided regression test
  catch a rise 2% -> 10%:    108
  catch a rise 2% -> 6%:    296
  catch a rise 2% -> 4%:    899
  catch a rise 2% -> 3%:  3,013

The lesson sits in the gap. Confirming a 2 point quality lift needs about 6,000 requests per arm, and a 1 point lift needs about 25,000. Catching a doubling of your error rate needs under 900. That's why a canary is cheap and an honest A/B test for small improvements is expensive.

Convert to time with your own volume. I assumed 20,000 requests a day, an invented figure, and that the canary arm is the bottleneck. The equal-arm formula slightly overestimates the canary size when the control is bigger, so these are conservative:

canary at  1%: 2%->4% guardrail    needs    899 in canary arm ->    4.5 days
canary at  1%: 80%->82% quality    needs  6,039 in canary arm ->   30.2 days
canary at  5%: 2%->4% guardrail    needs    899 in canary arm ->    0.9 days
canary at  5%: 80%->82% quality    needs  6,039 in canary arm ->    6.0 days
canary at 25%: 2%->4% guardrail    needs    899 in canary arm ->    0.2 days
canary at 25%: 80%->82% quality    needs  6,039 in canary arm ->    1.2 days

So at low volume, a 1 percent canary is for catching disasters. A claim like "the new prompt improves success by 2 points" needs a big slice or a long wait. If you can't afford that, say so, and decide the change on offline evals plus guardrails alone.

What should the rollback rule be?

The tempting rule is "if the canary's error rate is above control's, roll back". I simulated it. The setup: control arm 10 times larger than the canary, 4,000 simulated rollouts per row, fixed seed. Rule 1 is the tempting one. Rule 2 rolls back only if a one-sided z-test says the canary is worse at alpha 0.05.

 canary n |   true rates (ctl->canary) | rule 1 rolls back | rule 2 rolls back
      200 | 2% -> 2%   no real change |             45.8% |              6.2%
      200 | 2% -> 4%  real regression |             92.1% |             53.1%
    1,000 | 2% -> 2%   no real change |             46.9% |              5.6%
    1,000 | 2% -> 4%  real regression |            100.0% |             97.6%
    5,000 | 2% -> 2%   no real change |             48.2% |              5.1%
    5,000 | 2% -> 4%  real regression |            100.0% |            100.0%

Read it row by row.

• Rule 1 is a coin flip when nothing changed. It rolled back 46 to 48 percent of the time, no matter how big the canary was. More data doesn't fix it, because the rule has no notion of noise. You'd abandon good prompts half the time and learn to ignore the alarm.
• Rule 2 holds its false alarm rate near the 5 percent you set (6.2, 5.6, 5.1), though it runs slightly high with only 200 requests.
• Rule 2 is blind when the canary is small. With 200 requests it caught the doubled error rate 53 percent of the time. With 1,000 requests, 97.6 percent. A passing check at 200 requests means "inconclusive", not "safe".

Practical rule set:

1. Define each guardrail as a rate with a baseline: error or format-failure rate, refusal rate, p95 latency, cost per request.
2. Set a minimum sample before the rule can fire or pass. Use the one-sided table above.
3. Roll back on a significant regression, or immediately on a hard failure (for example a spike of malformed outputs) that you've defined in advance.
4. If the minimum sample isn't reached, hold the stage. Don't widen on an inconclusive result.
5. Check once at the minimum sample, not every hour. Repeated looks inflate false alarms and false passes; A/B testing prompts in production explains why peeking hurts.

One caveat I didn't simulate: this treats each request as independent. Requests from one user are correlated, so effective sample sizes are smaller than the raw counts. If a few heavy users generate most of your traffic, compute your rates per user, or treat these numbers as optimistic.

Which metrics belong on the dashboard?

Keep it short, and write it before the rollout.

• Guardrails (canary): error and timeout rate, malformed-output rate (JSON parse failures), refusal rate, p95 latency, tokens and cost per request. Latency and cost often move even when quality doesn't, see LLM latency budgets.
• Quality (A/B): one primary metric chosen in advance, such as task success judged by a rubric, thumbs-up rate, or human-escalation rate. Judged scores need the checks in LLM-as-judge pitfalls.
• Segments: language, plan tier, new versus returning users. A flat average can hide a bad result in a small segment.

Log the prompt version and experiment arm on every request. Without that field you can't analyse anything afterwards, and a postmortem becomes guesswork, see the incident postmortem template.

Rollout checklist

Copy this into the pull request that changes the prompt.

[ ] Offline eval run: new vs old on the golden set, per-bucket results attached
[ ] Experiment name fixed and recorded: ______
[ ] Bucketing key: user id / conversation id (circle one), server-side
[ ] Prompt version + arm logged on every request
[ ] Guardrails and baselines written down: ______
[ ] Minimum sample per stage (from one-sided table): ______
[ ] Hard-fail rule (immediate rollback): ______
[ ] Rollback is a flag flip; tested once in staging
[ ] Stages and who decides to widen: 1% -> 5% -> 25% -> 50% -> 100%
[ ] If claiming a quality lift: primary metric, MDE and required n per arm fixed in advance

What this post didn't verify

The traffic volume (20,000 requests a day) is an assumption. The simulation uses independent binomial draws at fixed true rates, which real LLM traffic is not: quality varies by time of day, input mix and user. The normal approximation gets rough for rates near 0 or 1 with small samples. Feature-flag products and managed experimentation tools vary; I didn't evaluate any of them, so check the docs of whatever you use.