Write Unit Tests With AI: A pytest Mutation Testing Check

You asked an assistant for unit tests, it returned nine tidy pytest functions, and the coverage report says 100%. That feels finished. It isn't, because coverage tells you the lines ran, not that any test would fail if a line were wrong.

The fix is a mutation check: change the code in small ways and see whether your tests object. This post gives you a runnable mutation script, a before-and-after on a toy module, and a prompt for getting an AI to close the gaps it left. If you want to write unit tests with AI and trust them, this is the verification step.

All numbers below come from runs on 2026-10-09 with Python 3.13.16 and pytest 9.1.1. The module is invented for this post. The weak test file was written by hand to mimic what a typical happy-path generation looks like; it was not captured from a model, so treat it as illustrative.

What does 100% coverage actually prove?

Here is the module under test. It is a shipping fee and a bulk discount for an invented shop.

def shipping_fee(weight_kg, express=False):
    if weight_kg <= 0:
        raise ValueError("weight must be positive")
    if weight_kg <= 1:
        fee = 40
    elif weight_kg <= 5:
        fee = 40 + (weight_kg - 1) * 20
    else:
        fee = 120 + (weight_kg - 5) * 15
    if express:
        fee = fee * 1.5
    return round(fee, 2)


def bulk_discount(quantity, unit_price):
    if quantity < 1:
        raise ValueError("quantity must be at least 1")
    subtotal = quantity * unit_price
    if quantity >= 10:
        return round(subtotal * 0.90, 2)
    if quantity >= 5:
        return round(subtotal * 0.95, 2)
    return round(subtotal, 2)

The weak suite has nine tests: one value per band (0.5 kg, 3 kg, 7 kg), one express case, one invalid weight, one quantity per discount tier (2, 6, 12) and one invalid quantity. Every test passes. Running coverage with branch tracking:

Name                     Stmts   Miss Branch BrPart  Cover   Missing
/tmp/C5b/p1/pricing.py      20      0     14      0   100%

Twenty statements, fourteen branches, nothing missed. A dashboard would show green. Now the mutation check.

How do I run a mutation check on a Python module?

The script below walks the module's syntax tree and produces one mutant per eligible site: it flips <= with < and >= with >, swaps + with - and * with /, and adds one to every numeric constant. For each mutant it writes the changed module into a temp folder beside a copy of your test file and runs pytest. If pytest still passes, the mutant survived.

"""Tiny mutation checker: python mini_mutate.py <module.py> <test_file.py>"""
import ast
import os
import shutil
import subprocess
import sys
import tempfile

CMP_SWAP = {ast.LtE: ast.Lt, ast.Lt: ast.LtE, ast.GtE: ast.Gt, ast.Gt: ast.GtE}
BIN_SWAP = {ast.Add: ast.Sub, ast.Sub: ast.Add, ast.Mult: ast.Div, ast.Div: ast.Mult}


class Mutator(ast.NodeTransformer):
    """Mutates the Nth eligible site (target). target=None only counts sites."""

    def __init__(self, target=None):
        self.target = target
        self.count = 0
        self.note = ""

    def _hit(self, note):
        hit = self.count == self.target
        if hit:
            self.note = note
        self.count += 1
        return hit

    def visit_Compare(self, node):
        self.generic_visit(node)
        op = type(node.ops[0])
        if op in CMP_SWAP:
            new = CMP_SWAP[op]
            if self._hit(f"line {node.lineno}: {op.__name__} -> {new.__name__}"):
                node.ops[0] = new()
        return node

    def visit_BinOp(self, node):
        self.generic_visit(node)
        op = type(node.op)
        if op in BIN_SWAP:
            new = BIN_SWAP[op]
            if self._hit(f"line {node.lineno}: {op.__name__} -> {new.__name__}"):
                node.op = new()
        return node

    def visit_Constant(self, node):
        if isinstance(node.value, (int, float)) and not isinstance(node.value, bool):
            if self._hit(f"line {node.lineno}: constant {node.value} -> {node.value + 1}"):
                return ast.copy_location(ast.Constant(node.value + 1), node)
        return node


def run_tests(workdir, test_file):
    proc = subprocess.run(
        [sys.executable, "-m", "pytest", "-x", "-q", "-p", "no:cacheprovider", test_file],
        cwd=workdir, capture_output=True, text=True,
    )
    return proc.returncode == 0


def main():
    module_path, test_path = sys.argv[1], sys.argv[2]
    source = open(module_path).read()
    sites = Mutator()
    sites.visit(ast.parse(source))
    total = sites.count
    survivors = []
    for i in range(total):
        m = Mutator(i)
        tree = m.visit(ast.parse(source))
        ast.fix_missing_locations(tree)
        work = tempfile.mkdtemp(prefix="mut_")
        shutil.copy(test_path, work)
        with open(os.path.join(work, os.path.basename(module_path)), "w") as fh:
            fh.write(ast.unparse(tree))
        if run_tests(work, os.path.basename(test_path)):
            survivors.append(m.note)
        shutil.rmtree(work)
    killed = total - len(survivors)
    print(f"{os.path.basename(test_path)}: {total} mutants, {killed} killed, "
          f"{len(survivors)} survived, score {killed / total:.0%}")
    for note in survivors:
        print("  SURVIVED", note)


if __name__ == "__main__":
    main()

Run it against the weak suite:

test_weak.py: 36 mutants, 22 killed, 14 survived, score 61%
  SURVIVED line 7: constant 1 -> 2
  SURVIVED line 7: LtE -> Lt
  SURVIVED line 9: constant 5 -> 6
  SURVIVED line 9: LtE -> Lt
  SURVIVED line 15: constant 2 -> 3
  SURVIVED line 19: constant 1 -> 2
  SURVIVED line 19: Lt -> LtE
  SURVIVED line 22: constant 10 -> 11
  SURVIVED line 22: GtE -> Gt
  SURVIVED line 23: constant 2 -> 3
  SURVIVED line 24: constant 5 -> 6
  SURVIVED line 24: GtE -> Gt
  SURVIVED line 25: constant 2 -> 3
  SURVIVED line 26: constant 2 -> 3

Fourteen of 36 changes went unnoticed by a suite with 100% coverage. Read the list as a to-do list:

• Lines 7, 9, 19, 22 and 24 are boundaries. Nothing tests exactly 1 kg, exactly 5 kg, quantity 5 or quantity 10, so >= could become > without a single failure. Real bugs live in exactly these spots.
• Lines 15, 23, 25 and 26 are the round(..., 2) calls. The weak tests only use prices that produce clean numbers, so rounding to three places looks identical.

Caveats on the script: it is a teaching tool, not a replacement for a maintained mutation tester. It mutates only those three kinds of site (no and/or swaps, no return-value or True/False changes, no removed statements), and runs the whole test file for every mutant, so it gets slow on big suites.

Which prompt makes an AI write tests that kill survivors?

Paste the survivor list back to the assistant along with the code. This is an untested template; I wrote it for this post and did not run it on a live model.

Here is a Python module and its current pytest file.
A mutation check changed one thing at a time and the tests still passed
for these mutants:

<paste the SURVIVED lines>

For each survivor:
1. Say what input would behave differently between the original and the
   mutant. If no input can (an equivalent mutant), say so and explain why.
2. Otherwise add or change a test that fails on the mutant and passes on
   the original. Use parametrize for boundary values.
Rules: do not modify the module. Do not add tests that only repeat an
existing assertion. Pin exact expected values; no assertions like
"result > 0". Show me each new test with a one-line comment naming the
mutant it targets.

The "equivalent mutant" instruction matters. Without it, a model will invent a contrived test for every line, and some can't exist.

What did the stronger suite change?

I wrote a second file, test_strong.py, in the shape that prompt asks for: boundary pairs (1 and 1.5 kg, 5 and 5.5 kg, quantity 5 and 10), exact expected values, and match= on the error messages. Then I ran the script again:

test_strong.py: 36 mutants, 31 killed, 5 survived, score 86%
  SURVIVED line 7: LtE -> Lt
  SURVIVED line 9: LtE -> Lt
  SURVIVED line 15: constant 2 -> 3
  SURVIVED line 23: constant 2 -> 3
  SURVIVED line 26: constant 2 -> 3

Boundary tests removed most of the comparison survivors. Five remained, and they split into two different kinds.

Equivalent mutants (lines 7 and 9). Take weight_kg <= 1 and change it to < 1. At exactly 1 kg, the original takes the first branch and returns 40. The mutant skips to the middle band and returns 40 + (1 - 1) * 20, which is also 40. Same for 5 kg: 120 either way. No test can tell them apart, so don't try. This is why a 100% target is wrong.

Real gaps (lines 15, 23 and 26). These are the round constants. Every price in test_strong.py multiplied out to a clean number, except one case on the 5% branch, which is why line 25 died. A test with a messy price on the other branches was missing. I added three parametrized cases with fractional prices (19.999) plus a 1.3333 kg shipping case, saved as test_final.py:

test_final.py: 36 mutants, 34 killed, 2 survived, score 94%
  SURVIVED line 7: LtE -> Lt
  SURVIVED line 9: LtE -> Lt

The two survivors are the equivalent ones. Everything else is killed. Suite size: 22 collected test cases in test_final.py (19 in test_strong.py) against nine in the weak file, and pytest finishes each in well under a second.

How do I review AI-written tests by hand before the mutation run?

Mutation testing is the slow check. These are the fast ones, and they catch most weak suites on sight. Scan each generated test for:

1. Exactly one value per branch. If you see 6 for the 5% tier and 12 for the 10% tier, ask what happens at 4, 5, 9 and 10.
2. Assertions that can't fail. assert result, assert result is not None, assert isinstance(result, float). These pass for wrong answers.
3. Expected values computed with the code's own formula. assert shipping_fee(3) == 40 + (3 - 1) * 20 repeats the bug if the formula is wrong. Work out expected values independently.
4. Mocks everywhere. If the function under test is mocked, nothing real is being tested. Mock the network and the clock, not your own logic.
5. No error-path assertions. pytest.raises(ValueError) without match= passes even if the wrong ValueError fires.
6. Tests that read like the implementation. A test named test_shipping_medium with one call says nothing about the rule. Parametrize with ids that describe the rule.

For prompts that make an assistant read code more skeptically in general, see AI code review prompts that actually catch bugs. For a broader view of AI-assisted testing on a real stack, Claude Code for QA engineers covers pytest generation from endpoints and OpenAPI specs.

Should I use mutmut or write my own script?

For a real project, use a maintained tool. The mutmut README (PyPI page read on 2026-10-09, version 3.8.0) says you install it with pip install mutmut and start with mutmut run; it runs pytest on the tests or test folder and tries to work out which code to mutate. It remembers finished work so you can stop and resume, and mutmut browse opens a terminal UI to inspect survivors. Two practical limits from the same page: it needs a system with fork support, so on Windows you run it inside WSL, and mutmut apply writes a mutant to disk, so commit first.

I installed mutmut 3.8.0 in the scratch environment but did not run it, so I can't tell you what score it would give this module. Its operator set differs from the script above, so scores from the two tools are not comparable anyway.

The hand-rolled script earns its place in two situations: when you want to see exactly what a "mutant" is before trusting a tool, and when you want a ten-second check on one small module inside a CI step you control.

A workflow that fits in a pull request

1. Ask the assistant for tests, giving it the module and your conventions. A short project file helps; see how to write a CLAUDE.md for any project.
2. Run the suite. Read it against the six-point scan above.
3. Run coverage. Treat a low number as a signal; treat a high number as nothing.
4. Run the mutation check on the module only, not the whole repo.
5. Sort every survivor into "real gap" or "equivalent". Feed only the real gaps back with the prompt above.
6. Keep the final suite. Delete tests that duplicate each other; a suite with the same assertion three times is slower to maintain and kills nothing extra.

This turns the AI from a test generator you hope is right into a collaborator whose output gets measured. The same habit applies to everything it writes; the vibe coding anti-patterns post covers where skipping the measurement goes wrong.

What this check does not cover

A 94% mutation score on a 20-line module with three mutation types says little about your service. It doesn't test whether the requirements are right: if the shop really charges 2 kg at 70, not 60, every test here is a perfectly killed mutant of a wrong rule. It also doesn't cover concurrency, I/O or integration behaviour. Mutation testing measures how well your tests defend the code you have. It can't tell you whether the code does what the business meant.