Chapter 40: Durable Agent Loops

"It crashed, so resume where it left off" — resume a three-month-old destructive turn with nobody watching and you'll wish it had just failed.

Quick Track (Master the core in 5 minutes)

1. Checkpoint at phase boundaries, debounced — not every iteration, not only at the end
2. A durable marker index makes crashed turns discoverable without scanning every session
3. Resumption needs three gates: enabled, staleness window, attempt cap
4. A recovered turn is always unattended, whatever the original source was
5. Persist the attempt count before the model call, or a crash loop resets it forever

10-Minute Path: 40.1-40.3 -> 40.5 -> Kocoro OSS Lab

Public-source snapshot: implementation details are examples from public Kocoro origin/main at commit 4ec6772, reviewed 2026-07-27. Treat the invariant as the lesson and re-check constants against current source.

40.1 Starting with a Turn That Woke Up Three Months Late

An agent is midway through a destructive operation. The user asked for it, approved it, and is sitting right there. Then the daemon is killed — an upgrade, an OOM, a laptop lid.

The turn is checkpointed. Good. That's the whole point of durability.

Three months later, the user upgrades. The daemon starts, finds a durable checkpoint, and does exactly what it was built to do: it resumes. The destructive operation runs. Nobody is watching. The intent behind it expired a quarter ago, along with the context that made it correct — the files it referenced, the branch it targeted, the reason it was a good idea.

This is a real failure mode, and the fix isn't "checkpoint less." It's that resumability without an expiry is a time bomb with a durable fuse.

40.2 Where to Checkpoint

The naive options are both wrong. Checkpoint every iteration and you pay serialization cost on a hot path for state that's about to change again. Checkpoint only at the end and a crash loses everything, which is the situation you were trying to fix.

The useful answer is phase boundaries: the points where the turn has just produced something worth keeping and is about to do something that might not return. Model call completed. Tool batch finished. Compaction summary written — that last one especially, because the summary was expensive and losing it means paying for it twice.

Add a debounce so a burst of fast boundaries doesn't turn into a burst of writes. A couple of seconds is enough; the point of the checkpoint is surviving a crash, not being current to the millisecond.

And write the session before the risky thing, not after. Chapter 34 covered this from the other direction — the session is persisted before loop.Run precisely so a mid-execution crash leaves a record rather than a hole. Durability is a property you buy in advance or not at all.

40.3 How a Crashed Turn Gets Found Again

A checkpoint nobody discovers is just disk usage.

The obvious approach — scan every session at startup and look for unfinished ones — costs more the longer the system has been in use, which is exactly backwards. Steady-state startup should be cheap.

So maintain a durable marker index: a small record per interrupted turn, written alongside the session. Startup reads the index, not the corpus. The invariant to hold is a biconditional:

on-disk InProgress = true ⟺ a marker exists

Every persistence path has to maintain it — the full save, and every read-modify-write patch. Miss one path and you get orphaned markers pointing at turns that finished, or finished-looking turns that were actually interrupted. Both are worse than no recovery at all, because they make the recovery system itself untrustworthy.

40.4 Three Gates, and the One That Matters Most

Discovery is the easy half. Deciding whether to execute is where the design lives.

Enabled. A boolean, default on, that turns automatic continuation off entirely while leaving markers in place. Operators need a way to say "find them but don't run them," particularly during an incident.

Staleness window. The gate from §40.1. The public default is four hours, and the reasoning is written into the config:

Default 4 covers crash/upgrade restarts within a work session; anything older executes a stale user intent with nobody present (a months-old interrupted destructive turn must never fire on upgrade). When it binds, the checkpoint is abandoned with an interrupted_turn_abandoned event instead of resumed.

Note the shape of that argument. Four hours isn't a round number picked for comfort — it's "one work session," the span over which a user's intent plausibly still holds. Beyond it, the checkpoint doesn't represent something the user still wants; it represents something they wanted, once, in a context that no longer exists.

Note also that abandonment is an event, not a silent drop. A checkpoint that expires should be visible. Silent abandonment and silent resumption are both ways of doing something surprising without telling anyone.

Attempt cap. Default three. And here is the detail that makes it work: the attempt count is persisted before the model call, not after. Increment after and a turn that crashes the daemon during the call will never record its attempt — so it retries forever, crashing on every startup, and your recovery system is now a boot loop.

40.5 A Recovered Turn Is Always Unattended

This is the subtlest part, and it's a security property rather than a correctness one.

A turn originally started from an interactive desktop session was attended. A human was present, approvals could round-trip to a real person, and the permission system correctly allowed things that require consent.

When that same turn resumes at daemon start, nobody is there — but the session's recorded source still says "desktop." Classify by source and the recovered turn inherits attended privileges with no human to exercise them. The public runtime pins this explicitly:

IsUnattendedRun is always true: a recovered turn executes at daemon start with no human present, regardless of the session's original source. Without this marker a desktop/kocoro-source checkpoint would classify as attended and bypass the unattended auto-approval deny-list.

The general rule: attendance is a property of the moment of execution, not of the session's origin. Any system that derives permission from stored provenance rather than present reality will eventually grant privileges to an empty room.

The same logic says the resume request must take the original route key, so it contends for the same lock as concurrent inbound traffic. Recovery that bypasses normal routing can run concurrently with a live turn on the same session — two writers, one session, no lock between them.

40.6 Snapshot Evidence

These describe one dated implementation, not a universal contract.

40.7 What Durability Does Not Give You

Resuming a turn is not the same as making it idempotent, and conflating the two is how recovery causes the outage it was meant to prevent.

If the interrupted turn had already sent an email, charged a card, or pushed a branch, the checkpoint records that the tool was called — it cannot record whether the side effect landed. Resume blindly and you may send twice. The defence is at the tool layer, not the recovery layer: idempotency keys on anything with an external effect, so a repeat is a no-op rather than a duplicate.

Durability also does not survive a broken deploy. A checkpoint written by one version and resumed by another is only safe if the message shapes are compatible. Version your persisted state, and treat an unreadable checkpoint as abandon-with-an-event rather than crash-on-startup.

And it does not replace cancellation. A user who cancels wants the turn gone, not checkpointed and helpfully resurrected on the next daemon start. Cancellation must clear the marker.

40.8 Common Pitfalls

Incrementing the attempt count after the call. Produces a boot loop that looks like a crash bug.

No staleness window. The §40.1 failure. Any system with durable resumption and no expiry has this bug latent right now.

Classifying attendance by session source. Grants a permission set to nobody.

Maintaining the marker in Save but not in the patch paths. The invariant holds until the one code path that doesn't maintain it runs, and then it's silently false.

Resuming outside the normal route lock. Two writers on one session, discovered later as mysterious interleaved history.

Treating "the tool was called" as "the effect happened." Checkpoints record intent, not outcome.

Kocoro OSS Lab (10 minutes)

1. Read the InterruptedResumeMaxAgeHours comment. Note that it names the failure it prevents, in one clause, in the config struct — where the person changing the value will see it.
2. Find IsUnattendedRun and write down which specific tools its true keeps out of an unattended run.
3. In your own system, list the tools with external side effects. For each, ask what happens if the turn that called it resumes. If you can't answer, you don't have durable recovery — you have durable duplication.

Key Points

1. Checkpoint at phase boundaries, debounced. Not every iteration, not just the end.
2. A marker index makes discovery cheap. And the on-disk-flag ⟺ marker biconditional must hold on every persistence path.
3. Three gates: enabled, staleness, attempts. The staleness window is the one most systems are missing.
4. Attendance belongs to the moment of execution. A recovered turn is unattended regardless of where the session came from.
5. Resumption is not idempotency. External side effects need keys at the tool layer; the checkpoint can't tell you whether the email went out.

Next: Chapter 41 covers the other direction — not recovering a turn that stopped, but changing one that's still running.