LLM Feature Incident Postmortem Template (With a Fictional Worked Example)

The bot said customers had 30 days to return an item. The policy said 14. By the time someone noticed, hundreds of people had been told the wrong thing, and the first question in the incident channel was "who changed the prompt?"

That question is the problem. A postmortem that opens with who produces defensive people and thin write-ups. This one gives you a blameless template built for LLM features, a list of metrics worth capturing, and a worked example. The example is entirely fictional: the company, the people, the numbers and the incident were invented by me for this post to show how the template reads when filled in.

It pairs with the AI agent production checklist, which is about preventing incidents, and agent observability, which is about seeing them. If the incident was a jailbreak or an injection, the jailbreaking explainer and prompt injection defence cover the prevention side.

Why do AI features need their own postmortem fields?

The core format is the one software teams already use. Google's SRE book defines a postmortem as a written record of an incident, its impact, the actions taken to mitigate or resolve it, the root causes and follow-up actions to prevent recurrence. It also says a blameless postmortem focuses on contributing causes without blaming any individual or team, on the assumption that everyone involved had good intentions. I read that chapter on 2026-10-08; it's the basis for the structure below.

What changes with LLM features:

• The failing component is not deterministic. The same input may succeed ten times and fail on the eleventh. "Can't reproduce" is the default, so you need to capture settings and, ideally, the exact request.
• There are more things that can change silently. The model version (including provider-side updates), the system prompt, retrieved documents, tool definitions and sampling settings can each shift behaviour. A normal deploy log shows only one of them.
• Wrong answers look like right answers. No error code fires. Detection often depends on a human noticing, so how it was detected becomes a first-class field.
• Harm can be informational. The damage may be a false statement to a customer, not downtime. Impact needs a different yardstick.

What triggers an AI postmortem?

Decide this in advance, in writing. The SRE book's examples of triggers include user-visible degradation beyond a threshold, data loss, an on-call engineer having to intervene (such as a rollback), a long resolution time, and a monitoring failure that meant the incident was found by hand. All of those apply to LLM features. I'd add these, which are my suggestions and not a standard:

• A false, harmful or off-policy output reached a real user
• Private or restricted data appeared in an output
• An agent took a wrong action with a real-world effect (sent, deleted, charged)
• A prompt, model or index rollback was needed
• A user or customer complained publicly

Anyone should be able to request one. A near miss caught by luck is worth a short version.

The template

Copy this into your wiki. Sections marked (AI) are the additions for LLM features. This is a template I wrote; it is not a vendor or regulatory form.

# Postmortem: [short factual title]

Status: draft | in review | final
Authors: [roles, not blame]
Incident severity: [your scale]
Date of incident: [YYYY-MM-DD]   Date of review: [YYYY-MM-DD]

## Summary
Two or three sentences: what happened, who was affected, how it ended.

## Impact
- Who: [users / customers / internal teams]
- What they saw: [exact wrong or harmful behaviour]
- How many: [count and how measured: logs, replay of sample, estimate]
- Business or safety consequence: [refunds, complaints, data exposed, none]

## Versions in play (AI)
- Model and version / provider alias:
- System prompt version:
- Retrieval index or knowledge base version:
- Tool definitions version:
- Sampling settings (temperature etc.):
- Feature flags / traffic share:

## Timeline (all times in one time zone)
| Time | Event | Source |
|---|---|---|
| | Change introduced | deploy log |
| | First bad output | traces |
| | First user report | ticket |
| | Detected by humans or monitor | |
| | Mitigated | |
| | Resolved / follow-up complete | |

## Detection
How did we find out? [user / support / monitor / internal review]
What should have alerted us earlier?

## Contributing factors
List several. For each, name the control that could have caught it.
- Factor:  Missing or weak control:
(Avoid "human error" as a factor. Ask why the system allowed it.)

## Reproduction (AI)
Could we reproduce it? With which inputs and how often (for example 7 of 20 runs)?
Where are the saved requests and outputs?

## What went well

## What went poorly

## Where we got lucky

## Action items
| Action | Type (prevent / detect / mitigate) | Owner | Due | Ticket |
|---|---|---|---|---|

## Eval and monitoring gap (AI)
Which test case would have failed before release? Add it. Which live metric would have moved? Add an alert.

## Lessons shared with
[teams / channels]

Two rules for filling it in. Write facts in the timeline and keep opinions for the contributing-factors section. And require every action item to have an owner, a date and a type, because "improve testing" without an owner is how the same incident returns.

Which metrics should you capture?

Most of these are plain incident metrics. The AI-specific ones are about measurement honesty.

Keep the units straight. Time to detect is not the same as exposure window, and teams commonly report whichever one looks better. Report both.

Here's the small script I used to compute the timing metrics for the fictional example below. This is an excerpt (the timestamps are invented); I ran the full version on 2026-10-08.

from datetime import datetime
FMT = "%Y-%m-%d %H:%M"
t = {k: datetime.strptime(v, FMT) for k, v in {
    "introduced": "2026-03-02 14:05",
    "first_bad_answer": "2026-03-02 14:21",
    "detected": "2026-03-03 10:15",
    "mitigated": "2026-03-03 11:02",
    "resolved": "2026-03-05 17:30",
}.items()}

def hours(a, b):
    return (t[b] - t[a]).total_seconds() / 3600

Output of the full script on the fictional data:

time to detect   (first bad answer -> detected): 19.9 h
time to mitigate (detected -> mitigated):        47 min
exposure window  (introduced -> mitigated):      20.9 h
time to resolve  (mitigated -> resolved):        54.5 h

replayed sample on day 1: 103 of 302 answers wrong (34%)
note: replay covers day-1 hours only; day 2 overnight exposure is unmeasured

Look at what the numbers say: mitigation took 47 minutes, which looks fine. Detection took almost 20 hours, which is the real problem. A team that only tracks "time to fix" would miss that.

Worked example (fictional)

Everything in this section is fictional. "Lumen Home" is an invented online shop. The people, numbers, dates and the incident did not happen. It exists to show the template filled in.

Postmortem: Returns bot stated a 30-day window instead of 14 days

Status: final. Authors: support engineering, ML platform. Severity: 2 of 4. Date: 2026-03-02 to 2026-03-05 (fictional).

Summary. After a prompt update at 14:05 on 2 March, the returns assistant told some customers they had 30 days to return items instead of 14. A support lead noticed the next morning, and we rolled the prompt back 47 minutes later. We contacted affected customers over the following two days.

Impact.

• Who: customers asking about returns via the web chat.
• What they saw: a refund window of 30 days in place of the policy's 14.
• How many: not known exactly. A replay of logged day-1 conversations from 14:00 to 21:00 found 103 wrong answers out of 302 (34%). Overnight and next-morning conversations were not replayed, so the true count is higher. This is an estimate, not a log count.
• Consequence: customers acting on the wrong date, extra refund requests outside the policy, 38 goodwill exceptions approved (fictional).

Versions in play. Model alias: a provider "latest" alias, unchanged during the incident. System prompt: v14 (bad), v13 (rollback). Knowledge base: policy document unchanged, still stating 14 days. Temperature 0.3.

Timeline.

Detection. A customer. No monitor looked at answer content, and our dashboards tracked latency and error rate only.

Contributing factors.

1. The new "holiday returns" paragraph in v14 mentioned an extended window for one seasonal campaign, and the model generalised it. Missing control: no review step that checks policy statements in prompts against the policy source.
2. The regression suite had 12 cases, none about return windows. Missing control: eval coverage of money-affecting facts.
3. Prompts deployed without a gradual rollout. Missing control: traffic ramp with sampled review.
4. No alert on answer content. Missing control: a check comparing numeric policy claims in outputs to the policy table.
5. Support did not have a one-step way to flag "bot said something wrong". Missing control: a flag button feeding a review queue.

Reproduction. With the saved v14 prompt and a returns question, the model stated 30 days in 7 of 20 runs (fictional). That variability is why the first engineer to try it said "can't reproduce" and lost an hour.

What went well. Rollback was fast once the right people were looking. The prompt was versioned, so reverting was one command.

What went poorly. About 21 hours of exposure (20 hours 57 minutes). The estimate of impact depended on replay because logs didn't record whether answers were correct.

Where we got lucky. The error was in the customer's favour, so complaints were polite and no regulator-facing statement was involved. A wrong answer in the other direction would have been worse.

Action items.

The format matters more than the specifics. Notice that no item says "be more careful", no factor names a person, and each factor maps to a missing control.

How do you run the review meeting?

Keep it short and structured. Start with the timeline and correct it, since people's memories differ from the logs. Walk through the contributing factors and ask for each: what would have caught this? Spend most of the time on action items and push back on vague ones. Do not ask "why did you do X"; ask "what made X look reasonable at the time?" The second question surfaces the missing guardrail, and the first produces silence.

Publish the final write-up where the whole engineering and support team can read it. The SRE book's review criteria include whether key incident data was collected, whether impact was fully assessed, whether the analysis went deep enough, whether the action plan is appropriate and whether the outcome was shared with stakeholders. Those make a decent checklist for the review itself.

Make the next one cheaper

Postmortems become routine when the data is already there. Log the model, prompt and index versions on every request. Store inputs and outputs under whatever privacy rules apply to you. Keep a flag mechanism and a rollback path that anyone on call can use. If you're deciding what to record, the observability post has concrete fields.

What I did not verify

The worked example is fictional, so none of it is evidence about real systems. The template is my own synthesis, built on the structure and review criteria in Google's SRE book chapter on postmortem culture, which I read on 2026-10-08. I did not check any legal or regulatory incident-reporting duties; if your feature touches regulated data or decisions, ask a qualified professional what you must report and when.