A content moderation layer for an LLM app is a short pipeline, not a single API call. Cheap rules go first, then a moderation classifier, optionally a small-model screen, then the main model, then the same checks on its output. The pipeline is easy. The decisions are hard: what happens at each score, what happens when a classifier fails, and how do you test any of it without paying for a thousand live calls.
This post gives you a working layer built on stubbed classifiers, so the policy code and its tests run offline, plus what the OpenAI and Anthropic docs say about the real services, checked on 2026-10-30. The stubs are keyword rules I wrote. They test your plumbing, not any vendor's accuracy. That distinction runs through everything below.
For the attack side of this problem, see prompt injection defence in production. For agent-specific controls, see production guardrails with NeMo Guardrails and Guardrails AI. This post is the moderation piece: harmful content in, harmful content out.
What does a moderation layer need to check?
Six checks, in this order, cheapest first.
- Rules on the raw input. Length cap, empty input, obvious injection phrases. Costs nothing, catches junk.
- Moderation classifier on the input. A dedicated classifier returns per-category scores.
- Optional LLM screen. A small model answers one yes/no question about the input. Slower, more flexible, catches policy that is specific to you ("no financial advice").
- The main model call.
- Rules on the output. A planted canary string to detect system prompt leaks, patterns for personal data, links you don't allow.
- Moderation classifier on the output. Models can produce harmful text from harmless prompts.
Each layer can return allow, review or block, and the pipeline takes the strictest. Keep the layers independent so you can turn one off, tune it, or replace it.
What do the vendor docs actually say?
I read these on 2026-10-30. Where the fetch tool returned a summary rather than the raw page I say so.
OpenAI moderation (guide at developers.openai.com/api/docs/guides/moderation, redirected from the older platform.openai.com URL; read as a fetch-tool summary):
- The moderation endpoint is described as free to use. The model is
omni-moderation-latest, which takes text and images; audio isn't classified. - Output fields are
flagged,categories(booleans),category_scores(0 to 1) andcategory_applied_input_types. - Categories listed:
harassment,harassment/threatening,hate,hate/threatening,illicit,illicit/violent,self-harm,self-harm/intent,self-harm/instructions,sexual,sexual/minors,violence,violence/graphic. Some are text-only; images are covered for the self-harm, sexual and violence categories. For an image-only input, text-only categories return a score of 0. - Caveats from the guide that shape your design: treat flags and scores as signals for your own policy; the underlying model is updated over time so score-based policies may need recalibration; a refusal can be flagged if it discusses harmful content; a moderation step can fail and return an error in place of scores, so check the result type first; the guide says not to send known or suspected CSAM to the API and that it is not a substitute for dedicated child-safety safeguards.
- For tool calling it says moderation covers tool-call arguments and tool outputs in conversation content but not tool names, descriptions or schemas.
Anthropic (raw pages fetched; platform.claude.com/docs):
- The jailbreak mitigation page recommends a harmlessness screen: a lightweight model (the page names Claude Haiku 5.5) pre-screens user input, with structured outputs constraining the reply to a boolean such as
is_harmful. Because that model runs safety classifiers that can decline the screening request itself, the page says to treatstop_reason: "refusal"as a harmful verdict. - The same page recommends screening tool outputs the same way before the main model acts on them, putting untrusted content only in
tool_resultblocks, and responding to repeat offenders (throttle or ban). - The refusals pages say a refusal is a normal response with
stop_reason: "refusal", not an error, so error-rate monitoring won't see it.stop_details.categoryandexplanationcan be null, so write your own user message. After a refusal, reset the conversation context before continuing. In a Message Batch, a refused request comes back as a succeeded result with that stop reason.
What I did not verify: rate limits, pricing for the screening model, and whether Anthropic offers a standalone moderation endpoint (I found no mention on the pages above, which isn't proof it doesn't exist).
How do I turn scores into actions?
Write a policy table. One row per category: what action, at what score. The numbers below are illustrative placeholders, not recommendations. Tune them on your own labelled data (see building a golden dataset from production logs).
POLICY = {
"sexual/minors": ("block", 0.05),
"self-harm/intent": ("review", 0.30),
"self-harm/instructions": ("block", 0.30),
"violence/graphic": ("block", 0.80),
"harassment/threatening": ("block", 0.60),
"illicit/violent": ("block", 0.50),
"harassment": ("review", 0.70),
}
Three design choices in that table matter more than the numbers.
- Not every category is a block. Someone typing that they don't want to be here anymore needs a different response from someone making a threat.
reviewhere means: don't send it to the model as normal, route to a supportive canned response or a human, and log. Write that response with someone qualified; I'm not offering advice on what it should say. - Thresholds differ by category. A false positive on harassment costs you a rude user's annoyance. A false negative on a child-safety category is a different kind of problem, so that threshold is set very low.
- You can ignore
flagged. Make the decision from scores and your own table so a vendor change to its own flag line doesn't silently change your behaviour.
The pipeline code
This is the part that runs. Stubs stand in for the moderation API and the small-model screen, with the same result shapes the docs describe.
from dataclasses import dataclass, field
SEVERITY = {"allow": 0, "review": 1, "block": 2}
@dataclass
class Verdict:
action: str = "allow"
layer: str = ""
reasons: list = field(default_factory=list)
def escalate(self, action, layer, reason):
if SEVERITY[action] > SEVERITY[self.action]:
self.action, self.layer = action, layer
self.reasons.append(f"{layer}:{reason}")
def apply_policy(result, verdict, layer):
for cat, (action, threshold) in POLICY.items():
if result["category_scores"].get(cat, 0) >= threshold:
verdict.escalate(action, layer, f"{cat}={result['category_scores'][cat]:.2f}")
def check_input(text, fail_closed=True):
v = Verdict()
if len(text) > MAX_INPUT_CHARS:
v.escalate("block", "rules", "too_long")
return v
for p in INJECTION_PATTERNS:
if p.search(text):
v.escalate("review", "rules", "injection_pattern")
try:
apply_policy(moderation(text), v, "moderation")
except Exception as e: # a classifier outage is a decision, not an accident
v.escalate("block" if fail_closed else "allow", "moderation", f"error:{type(e).__name__}")
if not fail_closed:
v.reasons.append("moderation:skipped_fail_open")
screen = llm_screen(text)
if screen["stop_reason"] == "refusal" or screen.get("is_harmful"):
v.escalate("block", "llm_screen", "refusal_or_harmful")
return v
moderation() is the OpenAI call in production. This is unrun, since I had no API key, but the SDK signature is moderations.create(*, input, model=...), which I confirmed against the installed openai 2.41.1 package:
from openai import OpenAI
client = OpenAI()
def moderation(text: str) -> dict:
r = client.moderations.create(model="omni-moderation-latest", input=text).results[0]
return {"flagged": r.flagged, "category_scores": r.category_scores.model_dump(by_alias=True)}
One wrinkle I did check in the installed SDK: the Python field names use underscores (harassment_threatening) with the slash form (harassment/threatening) set as an alias. A plain model_dump() returns the underscore names, so a policy table keyed by slashes would silently never match. Use model_dump(by_alias=True) or key your table to match, and print one real result to confirm. That bug looks exactly like a working filter.
The output side runs a planted canary check, an email pattern, and the same classifier:
def check_output(text):
v = Verdict()
if SYSTEM_PROMPT_CANARY in text:
v.escalate("block", "output_rules", "system_prompt_canary")
if EMAIL.search(text):
v.escalate("review", "output_rules", "email_in_output")
try:
apply_policy(moderation(text), v, "output_moderation")
except Exception as e:
v.escalate("block", "output_moderation", f"error:{type(e).__name__}")
return v
Put a unique string (mine was CANARY-7f3a) in your system prompt. If it ever appears in a response, the prompt leaked. It's a cheap check, though a determined attacker can ask for it paraphrased.
How do I test it without calling a live API?
Table-driven tests with the stubs. Each case is an input and the action you expect. I ran this on 2026-10-30:
benign -> allow layer=-
benign_kill_process -> allow layer=-
threat -> block layer=moderation ['moderation:harassment/threatening=0.91', 'moderation:harassment=0.88']
self_harm_intent -> review layer=moderation ['moderation:self-harm/intent=0.74']
insult_only -> review layer=moderation ['moderation:harassment=0.72']
weapon -> block layer=moderation ['moderation:illicit/violent=0.93']
injection_phrase -> review layer=rules ['rules:injection_pattern']
too_long -> block layer=rules ['rules:too_long']
llm_screen_hit -> block layer=llm_screen ['llm_screen:refusal_or_harmful']
llm_screen_refusal -> block layer=llm_screen ['llm_screen:refusal_or_harmful']
output_leak -> block layer=output_rules ['output_rules:system_prompt_canary', 'output_rules:email_in_output']
The five unit tests passed (Ran 5 tests ... OK), including two that matter most:
def test_fail_closed_on_classifier_error(self):
v = check_input("hello TIMEOUT", fail_closed=True)
self.assertEqual(v.action, "block")
def test_fail_open_is_explicit_and_logged(self):
v = check_input("hello TIMEOUT", fail_closed=False)
self.assertEqual(v.action, "allow")
self.assertIn("moderation:skipped_fail_open", v.reasons)
What this tests: that your policy table maps scores to actions as you intended, that layers escalate rather than overwrite each other, that outages behave the way you decided, and that a benign sentence containing "kill" (kill the process) isn't blocked by your own rules. What it does not test: whether the real classifier scores those sentences the way my stub does. For that you need a labelled set of real examples run against the live API, and a record of the date and model version, since OpenAI's guide says the model changes over time.
What goes wrong in practice?
- Streaming. OpenAI's guide says moderation results arrive after the full output is complete when you use its generation-time moderation option. If you stream tokens to the user as they're produced, they may see text before your output check runs. Either buffer, or accept and handle a retraction. Anthropic's streaming refusals can arrive mid-stream, and the documented recovery is to reset context.
- Unchecked paths. Moderating the chat box but not the file upload, the retrieved document or the tool result. Anthropic's page treats tool output as untrusted, so screen those too.
- Errors read as pass. A result object with an error where scores should be, indexed with
.get(cat, 0), reads as score zero and allows everything. Check the result type first, as OpenAI's guide says. - Refusals counted as success. An HTTP 200 with
stop_reason: "refusal"looks healthy in a dashboard that only counts errors. Track it separately. - One-way logging. Log the layer, category, score and action, not the full user text, if the text may contain personal data. See PII masking before LLM calls.
- Abuse. A user who trips the filter ten times is a signal. Anthropic's page recommends throttling or banning repeat offenders; rate limiting and abuse prevention covers mechanics.
Do I need all six layers?
No. A minimal useful version is: length cap, moderation on input, moderation on output, fail-closed. Add the LLM screen when you have a policy that no general classifier knows about (for example, an education bot that must stay on syllabus, see guardrail prompts for an edtech doubt-solving bot). Add the canary when your system prompt matters. Each extra layer adds latency and a failure mode, and every layer needs its own tests.
Moderation also isn't legal or compliance advice. What counts as unacceptable content, what you must report and what you must retain depends on your jurisdiction and product, so check with a professional.
What I'd cut and what's unverified
The stubs are keyword rules; real classifier behaviour on your traffic is unmeasured here. The threshold numbers are placeholders. The OpenAI guide was read as a fetch-tool summary, so confirm category names and field shapes against the live page and an actual response before shipping. Pricing, rate limits and data-retention terms for both vendors are unchecked; see the data retention table for the retention question and re-verify it.
