Snapshot and Property Tests for LLM Outputs That Don't Flake

Your test asked the model to triage a support email and compared the result to a saved string. It passed ten times, then failed because the JSON keys came back in a different order. Nothing was wrong. The test was.

Snapshot and property tests for LLM outputs fix this by checking what must stay true instead of what happened to be said. This post builds a small triage function, tests it three ways, and shows what each style catches, with real numbers from a run on 2026-10-30 (Python 3.14, pytest 9.2.0, hypothesis 6.170.14).

One limit up front. The "model" here is a stub, fake_llm, that I wrote to vary its surface the way real models do: random key order, optional code fences, three phrasings of the summary, and a 15 percent chance of sloppy casing like "High". I chose those failure modes, so I know the truth. I did not run this against a live model, and nothing here says how often real models misbehave.

Why do exact-match tests on LLM output fail?

Because correct output isn't unique. The stub returns the same meaning for the same email, and the test below compares to the one string I'd expect:

def test_exact_match_is_flaky():
    expected = '{"category": "billing", "priority": "normal", "summary": "Billing issue reported."}'
    failures = sum(fake_llm(EMAILS["refund"], s) != expected for s in SEEDS)
    assert failures == 0, f"{failures}/200 runs differ from the exact string"

Real result:

AssertionError: 195/200 runs differ from the exact string

195 of 200 runs differ, and in every one of them the underlying answer was fine. Temperature zero helps with a real model but doesn't give you a stable string across versions, providers or prompt edits, so don't build a suite on the hope.

The function under test

The stub model plus the production code: a tolerant parser that strips fences, lowercases enums, validates, and caps the summary.

# triage.py
import json, random, re

CATEGORIES = {"billing", "bug", "other"}
PRIORITIES = {"low", "normal", "high"}
FENCE = re.compile(r"^```(?:json)?\s*|\s*```$", re.S)


def fake_llm(email: str, seed: int) -> str:
    rng = random.Random(seed)
    low = email.lower()
    category = "billing" if ("refund" in low or "invoice" in low) else "bug" if "error" in low else "other"
    priority = "high" if "urgent" in low else "normal"
    summary = rng.choice([
        f"Customer writes about {category}.",
        f"A {category} request from a customer.",
        f"{category.title()} issue reported.",
    ])
    obj = {"category": category, "priority": priority, "summary": summary}
    if rng.random() < 0.15:
        obj["priority"] = priority.title()          # sloppy casing: "High"
    items = list(obj.items())
    rng.shuffle(items)                              # key order varies
    text = json.dumps(dict(items), indent=rng.choice([None, 2]))
    if rng.random() < 0.4:
        text = f"```json\n{text}\n```"              # sometimes fenced
    return text


def parse_reply(raw: str) -> dict:
    data = json.loads(FENCE.sub("", raw.strip()))
    data["priority"] = str(data.get("priority", "")).strip().lower()
    data["category"] = str(data.get("category", "")).strip().lower()
    if data["category"] not in CATEGORIES:
        raise ValueError(f"bad category: {data['category']!r}")
    if data["priority"] not in PRIORITIES:
        raise ValueError(f"bad priority: {data['priority']!r}")
    data["summary"] = str(data.get("summary", ""))[:200]
    return data


def triage(email: str, seed: int = 0) -> dict:
    return parse_reply(fake_llm(email, seed))

If your real model is forced into a schema with structured outputs, some of this variation disappears. The tests below still earn their keep, because they also cover everything your code does after parsing.

Level 1: invariant assertions across many outputs

Run the function over many outputs (here 200 seeds times 3 emails) and assert the rules that must hold every time. Keep exactly one semantic check, the part you'd be embarrassed to get wrong.

@pytest.mark.parametrize("name,email", EMAILS.items())
@pytest.mark.parametrize("seed", range(200))
def test_invariants_hold_for_every_phrasing(name, email, seed):
    out = triage(email, seed)
    assert out["category"] in CATEGORIES
    assert out["priority"] in PRIORITIES
    assert 0 < len(out["summary"]) <= 200
    assert "@" not in out["summary"]                       # no address echoed back
    expected = {"refund": "billing", "bug": "bug", "other": "other"}[name]
    assert out["category"] == expected

Pick invariants from this list, then delete the ones you can't justify:

All free, none depend on wording. The "must contain" check is the one I'd reach for first in a RAG app: every cited id must be one you retrieved.

Level 2: snapshot the shape, not the words

A snapshot test is worth having for the contract. Reduce the output to keys and types, store that, and fail on any change. A tiny helper with no dependencies:

# snap.py
import json, os
from pathlib import Path

SNAP_DIR = Path(__file__).parent / "snapshots"


def shape(value):
    if isinstance(value, dict):
        return {k: shape(v) for k, v in sorted(value.items())}
    if isinstance(value, list):
        return [shape(value[0])] if value else []
    return type(value).__name__


def assert_snapshot(name, value):
    path = SNAP_DIR / f"{name}.json"
    text = json.dumps(value, indent=2, sort_keys=True) + "\n"
    if os.environ.get("UPDATE_SNAPSHOTS") or not path.exists():
        SNAP_DIR.mkdir(exist_ok=True)
        path.write_text(text)
        return
    assert text == path.read_text(), (
        f"snapshot {name} changed. If intended, rerun with UPDATE_SNAPSHOTS=1 "
        f"and review the diff in git."
    )

The stored snapshot for this function is:

{
  "category": "str",
  "priority": "str",
  "summary": "str"
}

I simulated a refactor that renamed priority to priority_level. The snapshot test failed with its message pointing at the fix. A rename is exactly the change a reviewer should see, and a text snapshot would have been red for the wrong reason a dozen times first.

If you'd rather not maintain the helper, the same idea exists in libraries such as syrupy or pytest-snapshot. I haven't used either in this session, so check their docs before adopting one.

Two rules keep snapshots honest:

• Never auto-update in CI. Updating is a local, deliberate command whose diff gets reviewed.
• If a snapshot contains free text, it will change every run and people will stop reading it. Snapshot structure, or normalise the text first (lowercase, strip, replace ids and dates with placeholders).

Level 3: property tests with hypothesis

Invariants over 200 seeds only cover the variations you thought of. hypothesis generates inputs for you and shrinks any failure to a small example. Two properties are worth writing for LLM code.

First, the parser must ignore formatting. Generate every combination of fence, indentation, padding and casing, and assert the parsed result is the same:

@given(
    category=st.sampled_from(sorted(CATEGORIES)),
    priority=st.sampled_from(sorted(PRIORITIES)),
    summary=st.text(max_size=300),
    fenced=st.booleans(),
    indent=st.sampled_from([None, 2, 4]),
    pad=st.text(alphabet=" \n\t", max_size=5),
    upper=st.booleans(),
)
@settings(max_examples=300, deadline=None)
def test_parser_ignores_formatting(category, priority, summary, fenced, indent, pad, upper):
    obj = {"category": category, "priority": priority.upper() if upper else priority, "summary": summary}
    raw = json.dumps(obj, indent=indent)
    if fenced:
        raw = f"```json\n{raw}\n```"
    out = parse_reply(pad + raw + pad)
    assert out["category"] == category
    assert out["priority"] == priority
    assert out["summary"] == summary[:200]

Second, no user input may break the output contract. Hypothesis feeds arbitrary text as the "email":

@given(st.text(max_size=500))
@settings(max_examples=200, deadline=None)
def test_pipeline_never_returns_invalid_shape(email):
    out = triage(email, seed=1)
    assert set(out) == {"category", "priority", "summary"}
    assert out["category"] in CATEGORIES

Both passed here. With a stub whose logic is a few if lines, that's unsurprising; against a real model you'd replace fake_llm with a recorded reply (see mocking LLM responses with VCR) and the second property becomes more interesting.

Do these tests catch real bugs?

Passing tests prove little until you've seen them fail. I copied the project and removed one line from the parser, the .lower() on priority, a plausible bug. Then I ran the suite (excluding the deliberately flaky test):

98 failed, 506 passed, 1 deselected in 0.56s

The failures were ValueError: bad priority: 'High' and 'Normal', hit by the 15 percent of stub outputs with sloppy casing and by the hypothesis property with its uppercase variants. The shape snapshot stayed green, correctly, because the shape didn't change. Each layer caught a different class of mistake.

Unmutated, the same suite gave:

1 failed, 604 passed in 0.92s

The one failure is the exact-match test, kept in the file on purpose as a demonstration. Delete it in your own project.

What should I do on a real model?

A suggested split, none of which I have run against a live model:

1. Every commit, free: the invariants and hypothesis tests over recorded or faked replies.
2. Every commit, free: the shape snapshot.
3. Nightly or per prompt change, costs tokens: a small golden set with a few trials per case, graded by rules first and a model judge only for what rules can't see. See agent evals from scratch for the repeated-trial mechanics, and prompt regression tests in CI for wiring it up.

For a real model, run each case several times and assert the invariant holds in all of them, not most of them. A rule like "never echoes an email address" doesn't get to pass 95 percent of the time.

Checklist

• No test compares full model text to a saved string.
• Each output type has 3 to 6 invariants from the table above, and one semantic check.
• One snapshot of the output shape, updated only by a deliberate command.
• One hypothesis property for "format variation doesn't change the parsed result".
• You've broken the parser on purpose once and watched the tests fail.
• Quality is judged elsewhere, by evals on a golden set.