You shipped a support bot. Three weeks later someone pastes a screenshot into your Slack: the bot, calmly and in your brand voice, doing something it was never meant to do. Nobody on the team knows which category of trick was used, or whether five other tricks would have worked too.
This post is about that gap. It explains how LLM jailbreaks work at the level a defender needs, which categories to test, and how to build a plan that tells you what you haven't covered. It deliberately contains no working attack strings. If you want those, they exist in public research datasets, and you don't need them to understand the defence.
I read the primary sources named below on 2026-10-08. The one script in the post is a coverage checker I ran; it contains no attack text. For the existing lesson on this topic, see the jailbreaking lesson, and for the sibling problem of hostile content inside documents and tool output, see prompt injection explained.
What is a jailbreak, and how is it different from prompt injection?
A jailbreak is input that gets a model to produce something its safety training or your instructions were meant to prevent. A prompt injection is when instructions hidden in untrusted content get followed as if they came from you. OWASP's LLM01 page says jailbreaking is a form of prompt injection and that people often use the terms interchangeably.
For defence, I find one distinction more useful than the definition: where does the text arrive?
| Arrives via | Who controls it | Typical name | Your main lever |
|---|---|---|---|
| The chat box | The user | Jailbreak | Model choice, filters, scope limits |
| A file, web page, email | A third party | Indirect injection | Segregating untrusted content, least privilege |
| A tool or API result | Whoever feeds that system | Indirect injection | Validating results, limiting what tools can do |
The harm is also different. A jailbroken chat reply is mostly a content-policy and reputation problem. The same trick against an agent with tools is an action problem. If your system can send email or write to a database, jump to the agent red-teaming post after this one.
Why do jailbreaks work at all?
Because safety behaviour is learned, not enforced. The clearest framing I know is the 2023 paper "Jailbroken: How Does LLM Safety Training Fail?" by Alexander Wei, Nika Haghtalab and Jacob Steinhardt (arXiv 2307.02483). It proposes two failure modes:
- Competing objectives. The model is trained to be capable and to follow instructions, and also to refuse certain things. When a request is framed so that following the instruction and refusing both look "correct", capability can win.
- Mismatched generalization. Safety training covers some part of the input space, but the model's capabilities extend further. Inputs that fall outside what safety training saw, while still being understood by the model, can slip through.
The authors tested attacks designed from those two ideas on GPT-4 and Claude v1.3, models that had been heavily red-teamed, and reported that they still found many working attacks. They argued that scaling by itself would not fix this and that safety mechanisms need to be as sophisticated as the model they protect. That is a 2023 result on 2023 models. Current models have been trained against many of the specific patterns since, but the two failure modes are structural, so they remain a useful map of where to look.
Two practical consequences follow:
- A model that refuses the direct version of a request tells you little about whether it refuses a reframed version.
- A defence that is only "the model knows better" fails in exactly the places where the model is most capable.
What are the main jailbreak categories to test for?
Here are the families, described by mechanism. This is a testing taxonomy, not a how-to. I'm deliberately not giving phrasing for any of them.
| Category | What it does | Which failure mode it leans on |
|---|---|---|
| Persona framing | Asks the model to act as a character or system with different rules | Competing objectives |
| Fiction or hypothetical framing | Wraps a request in a story, a game or a "for research" setup | Competing objectives |
| Multi-turn escalation | Moves toward a prohibited outcome across many small steps, each of which looks fine alone | Competing objectives; also a gap in per-message checks |
| Encoding or language shift | Presents the request in another language, a transformed encoding or odd formatting | Mismatched generalization |
| Many-shot context | Fills a long context with fabricated examples of the assistant complying, so the pattern is continued | Competing objectives, enabled by long context windows |
| Indirect delivery | Puts the instructions in content the system ingests rather than in the user's message | Not a model flaw alone; a trust-boundary flaw |
On many-shot: Anthropic described this technique in April 2024. Its write-up says effectiveness grows with the number of fabricated dialogues, that it worked on models from several developers, and that long context windows are what made it practical. Their mitigations are instructive. Shortening the context window works but removes legitimate value. Fine-tuning the model to refuse these queries only delayed the harmful output. What helped most was classifying and modifying the prompt before it reached the model; in one case that reduced attack success from 61% to 2%. That is one vendor's measurement on its own setup, so treat it as evidence that input-side checks can matter, not as a number your system will match.
Notice what the table says about multi-turn escalation: a filter that judges each message in isolation will miss it. If you only test single messages, you have not tested this category.
How do you defend against jailbreaks in layers?
No single layer holds. OWASP's mitigation list for this class of problem reads like a stack, and it's a good checklist to compare against your own: constrain behaviour in the system prompt, define and validate output formats with deterministic code, filter inputs and outputs, enforce least privilege, require human approval for high-risk actions, segregate untrusted content, and run adversarial tests while treating the model as an untrusted user.
Here is how I'd order them for a typical product, from cheapest to most decisive.
1. State scope in the system prompt. Say what the assistant is for and what it must decline. This is cheap and worth doing. It is also the weakest layer, since it's text competing with other text.
2. Judge the whole conversation, not just the last message. If you run a moderation or classifier step, give it the recent history. This is the direct answer to multi-turn escalation.
3. Check outputs, not only inputs. Whatever sneaks past the input side, you can still stop at the output. Define what a valid answer looks like for your product (format, topic, length, no secrets) and reject anything else with code, not with a second prompt. The guardrails comparison covers tools for this.
4. Remove what the model should never be able to do. This is the layer that changes the risk, not just its odds. If the bot can't issue refunds, nobody can jailbreak it into issuing one. Give the application its own narrowly scoped credentials, keep sensitive logic in code, and put a human approval step in front of anything irreversible.
5. Keep untrusted content apart from instructions. Mark documents and tool results as data in your prompt structure and don't let them change the system's permissions. This reduces indirect delivery, though it doesn't eliminate it.
6. Watch and rate-limit. Probing leaves a pattern: many near-duplicate requests, rapid topic switching, long conversations ending in a policy boundary. Log refusals, flag clusters and cap volume per user. The observability post covers what to capture.
If you take only one thing from this section: decide what the worst successful jailbreak could cause in your product, then spend most of your effort making that outcome impossible rather than unlikely.
How do you test your own app for jailbreaks?
You need a test plan with coverage, not a folder of clever prompts. Two ideas make the difference: a category-by-surface matrix, and payloads kept out of your public repo and your blog posts.
The surfaces are where text can reach the model: the chat input, uploaded documents, tool results, and attempts to extract your system prompt. Multiply categories by surfaces and you get a grid. Most teams I'd expect to have a handful of chat-input cases and nothing else.
I wrote a small script that takes a test plan as metadata only (id, category, surface, and a path to a payload file you keep privately) and prints the grid. It contains no attack text. I ran it on a deliberately thin seven-case plan on 2026-10-08:
7 cases, 7 of 24 category/surface pairs covered
category chat_input uploaded_doc tool_result system_promp
persona_framing x x . .
fiction_framing x . . .
multi_turn_escalation x . . .
encoding_or_language_shift x . . .
many_shot_context . . . .
indirect_via_content . x x .
categories with no case at all: many_shot_context
That output is the useful part. Seven cases sounds like effort, but they cover 7 of 24 cells and miss one category entirely. The check is trivial, and the core of it looks like this:
covered = {(c["category"], c["surface"]) for c in PLAN}
missing_cats = [c for c in CATEGORIES if not any(k[0] == c for k in covered)]
Where do payloads come from? Not from you inventing them under deadline. Use a maintained open-source red-teaming tool or a published dataset, read its licence, and store the payloads privately. The Promptfoo guide and the CI regression post show how to run a suite on every change. I haven't tested any particular payload set against any particular model for this post, so I can't tell you what passes.
What to record per run:
- The model name and version, your system prompt version, and the tool list. Results are only comparable if these are fixed.
- Pass or fail per case, with the actual reply stored, because a "refusal" that leaks half the content is a fail.
- The outcome class: refused, partially complied, complied, or took a tool action.
- Repeat each case several times. Model replies vary, so a single pass is weak evidence.
Decide the pass criterion before you run. "Looks fine to me" drifts toward whatever the current build does.
What should you do when a jailbreak works in production?
Treat it as an incident, not an embarrassment. Contain first: tighten the specific layer that failed (a filter rule, a tool permission, a rate limit) rather than rewriting the whole prompt in a panic. Then add the case to your test plan so it cannot silently regress, and check neighbouring cells of the grid, because attackers who find one working category usually try the adjacent ones.
Avoid the temptation to patch with a longer system prompt full of "never do X" lines. It tends to fix the one phrasing you saw and nothing else. Prefer removing the capability or adding a check in code.
If the incident reached users, write it up. A blameless format keeps the team focused on the system. The AI feature postmortem template is built for this case.
A short pre-launch checklist
- Written list of the worst outcomes a successful jailbreak could cause
- Capabilities and credentials reduced so the worst outcomes are impossible, not just discouraged
- Output validation in code for format, topic and secrets
- Moderation or classification that sees conversation history, not only the latest message
- Untrusted documents and tool results handled as data
- Category-by-surface test plan with payloads stored privately and a gap report
- Tests re-run on every model, prompt and tool change
- Logging and alerting for refusal spikes and probing patterns
- An owner and a written response path for when something gets through
What I couldn't verify
I did not run any jailbreak against a live model for this post, so I make no claim about how any current model behaves. The Wei et al. results are from 2023 models. The 61% to 2% figure is Anthropic's, for one setup. Vendor safety training changes quickly, so check current vendor documentation and run your own tests before relying on any statement here about what a given model will or won't do.
