Agent Checkpointing in Python: Resume After a Crash Without Repeating Side Effects

An agent that sends one email and then dies before it can note that it sent it will send that email again after a restart. That is the bug hiding in most "save state after each step" designs. This post builds agent checkpointing in Python with SQLite, kills the process with a real kill -9, and shows both the duplicate and the fix.

Everything below uses a stub model (a fixed three-step plan) and a fake mail service that writes to a file. No API key is involved and nothing here calls a live model. The kill, resume and counts are real output from my run on 2026-10-31 (Python 3.14, standard library only).

What do you actually need to save?

Less than you think. An agent loop is: send messages to the model, get a tool call, run it, append the result, repeat. If you store those appended items in order, you can rebuild the whole conversation after a crash.

So the checkpoint is an event log:

Resume is then: load events for the run id, turn them into messages, look at the last event, and continue. No pickled objects, no framework. If you want typed state, a graph and built-in checkpointers, see LangGraph stateful agents; the problem below applies to those too.

How do you build the event log in SQLite?

One table, with (run_id, seq) as the primary key so two writers cannot both claim the same position.

db = sqlite3.connect(DB, isolation_level=None)
db.execute("PRAGMA journal_mode=WAL")
db.execute("PRAGMA synchronous=FULL")
db.execute(
    "CREATE TABLE IF NOT EXISTS events("
    "run_id TEXT, seq INTEGER, kind TEXT, payload TEXT, PRIMARY KEY(run_id, seq))"
)

def append(run_id, rows):
    """Insert one or more events atomically."""
    db.execute("BEGIN")
    try:
        for kind, payload in rows:
            seq = db.execute(
                "SELECT COALESCE(MAX(seq)+1, 0) FROM events WHERE run_id=?", (run_id,)
            ).fetchone()[0]
            db.execute("INSERT INTO events VALUES (?,?,?,?)",
                       (run_id, seq, kind, json.dumps(payload)))
        db.execute("COMMIT")
    except Exception:
        db.execute("ROLLBACK")
        raise

Two settings matter. synchronous=FULL makes SQLite wait for the data to reach disk before a commit returns, which is what you want for a log you will trust after a crash. append takes a list so you can write several events in one transaction: all or nothing.

The naive loop and the window it leaves open

The first version most people write checkpoints after each completed step:

reply = model(messages)                       # decide
output = TOOLS[reply["tool"]](reply["args"])  # act
append(run_id, [("call", reply), ("result", {"output": output})])  # save

It looks safe. The call and result land in one transaction, so the log never holds half a step. But look at the gap between the second and third lines. If the process dies after the tool returns and before append commits, the tool has acted on the world and the log says nothing happened.

I tested that exact window. The script kills itself with os.kill(os.getpid(), signal.SIGKILL) right after send_email returns:

=== kill -9 after send_email ran, before result saved (MODE=naive) ===
first run exit code: -9  (killed by SIGKILL)
final: Report summarised and sent.
emails sent: 2   model calls: 5

Two emails. The resumed run rebuilt state from the log (which ended after the summary step), asked the model for the next action, got send_email again, and ran it. It also cost an extra model call: 5 instead of 4.

This is not exotic. OOM kills, deploys, a laptop lid closing and a container being rescheduled all look the same to your process. A graceful try/finally never runs.

How do you fix it: write the intent first

Split the step into two durable facts: "I decided to do this" and "this is what happened". Write the first before acting.

call = {"tool": reply["tool"], "args": reply["args"]}
append(run_id, [("call", call)])               # intent first
key = f"{run_id}:{call_seq}"                   # stable across restarts
output = TOOLS[call["tool"]](call["args"], key)
append(run_id, [("result", {"output": output})])

On resume, a call event with no result after it is a dangling call. You know exactly what the agent was doing, and you do not need to ask the model again. You re-run the same call with the same key:

if last_kind == "call":   # we died between intent and result
    call, call_seq = last, last_seq
    print(f"resuming dangling call seq={call_seq} tool={call['tool']}")

The key is run_id:seq. It is derived from the log position, so it is identical on every attempt. A random UUID generated at call time would be new after the restart and defeat the point.

Here is the same kill with that design:

=== kill -9 after send_email ran, before result saved (MODE=journal) ===
first run exit code: -9  (killed by SIGKILL)
resuming dangling call seq=5 tool=send_email
final: Report summarised and sent.
emails sent: 1   model calls: 4

One email, four model calls (three tool decisions plus the final answer), and the log tells you why: "resuming dangling call".

The deduplication happens in the fake mail service, which refuses a second send with a key it has already seen. That is the part you do not control with a real provider, so check what yours supports. The next post, making tool calls idempotent so agents can retry safely, covers what to do when it supports nothing.

What happens when the crash is between steps?

That is the easy case, and it is worth confirming. I killed the process right after step 1 (search_notes) had fully saved:

=== kill -9 after step 1 fully saved (MODE=journal) ===
first run exit code: -9  (killed by SIGKILL)
final: Report summarised and sent.
emails sent: 1   model calls: 4

The resumed run loads three events, sees the last one is a result, and carries on with step 2. Completed steps are never re-run and never re-asked of the model. Model calls stay at 4, the same as an uninterrupted run, which is the baseline I also ran and got the same counts.

Which tools can safely be re-run on resume?

The dangling-call rule needs a policy per tool. Decide it when you write the tool, not at 2 a.m.

The last row is the honest one. If you cannot tell whether the action happened, a duplicate may be worse than a gap, and that call belongs with a person. The approval gate post shows a place to hang that decision.

What should you not checkpoint?

• Secrets. Tool args get written to disk as JSON. Keep API keys out of args, and redact tokens before logging.
• Huge outputs. Store a pointer (file path, object id) and a short summary, not a 2 MB page. The log is replayed into the context window on resume, so the same token budget concerns apply.
• Model internals. You do not need hidden reasoning to resume. You need the decisions and their results.

What this design does not cover

• One process. SQLite handles one writer well. Two workers resuming the same run need a lease (a locked_until column) so only one picks it up. I did not build or test that.
• Disk loss. synchronous=FULL protects against process death and, on most setups, power loss. It does not protect against the disk going away. Back the file up.
• Schema changes. If you change the event shape, old runs in flight will not parse. Add a version field early.
• Real models. I used a stub, so I did not test what a live model does when you restart mid-conversation. Because the design replays stored decisions and never regenerates them, the model's variability should not matter, but that is reasoning, not a measurement.
• Long waits. If a run needs to sleep for three days or wait on a human, use a workflow engine; see durable agents with Temporal.

For where this fits in a launch review, the production checklist has a line for it, and plain Python loop versus framework covers when to stay this small.

Checklist

1. Every run has a stable id before anything happens.
2. State is an append-only event log, written with a commit per event or step.
3. Intent (call) is written before the tool runs; result after.
4. Each tool declares how it behaves on a dangling call (re-run, re-run with key, look up, or stop).
5. The idempotency key comes from run_id:seq, never from a random value or the clock.
6. Test it with a real kill -9 at the dangerous point, not a raised exception.