Chapter 42: Agent Timeouts and Watchdogs

“Nothing happened for 90 seconds” is not a failure description until you can name what the loop was waiting for.

Quick Track (Master the core in 5 minutes)

1. Model the blocking phase explicitly; wall-clock duration alone cannot distinguish a hang from legitimate work
2. Count idle time only while waiting for a remote LLM response, including the final force-stop synthesis
3. Soft idle reports once per phase instance; hard idle cancels with a typed cause and cleanup headroom
4. A stream-gap watchdog is a separate clock — it watches time between chunks inside the transport
5. If phase ownership becomes untrustworthy, disable timing actions and surface the structural bug

Public-source snapshot: implementation details are examples from public Kocoro origin/main at commit 4ec6772, reviewed 2026-07-27. Treat the invariant as the lesson and re-check constants against current source.

42.1 Starting with One Timer That Killed the Wrong Work

An agent asks an image service to render a large asset. Eight minutes is slow but legitimate; the tool owns a ten-minute timeout and is still making progress.

On another run, the LLM stream sends half a sentence and then the TCP connection goes silent. No final event, no error, no new bytes. Waiting eight minutes here is not patience. It is a hang.

Put one “turn timeout” around both and you must choose which run to break. Ninety seconds kills valid tools. Ten minutes leaves dead streams occupying workers. Reset the timer on every activity and a loop that alternates useless work can stay alive forever.

The problem is not picking the right duration. It is that different components own different kinds of waiting.

42.2 A Watchdog Needs a Phase Model

The loop exposes an explicit TurnPhase instead of inferring state from which timer was most recently touched:

Only phases that are strictly waiting on a remote model response count as idle. A Bash command, approval dialog, or local history rewrite may be quiet from the loop's point of view, but it has a different owner and a different valid duration.

This gives every timeout one question:

Which component promised progress in this phase, and what observation proves it?

If there is no owner, adding another timer hides the design gap. If two timers own the same wait, they will eventually disagree about which error the user should see.

42.3 Soft and Hard Idle Are Two Decisions

The public defaults are 90 seconds for soft idle and 540 seconds for hard idle. Those numbers describe one transport stack, not universal patience.

Soft idle emits status: no LLM activity for this duration, in this phase. It does not cancel. It fires once for a particular phase transition, identified by a monotonically increasing sequence number. Leave awaiting_llm, retry, then re-enter awaiting_llm, and the new phase instance can report again.

Hard idle emits a final status and cancels the run with ErrHardIdleTimeout. The 540-second default leaves 60 seconds below the 600-second transport ceiling so cancellation can propagate and cleanup can run before the outer HTTP layer gives up on its own.

That separation matters operationally. Soft is visibility. Hard is policy. You can change one without pretending the other changed too.

A typed cancellation cause matters as well. User cancellation and watchdog cancellation both close a context, but they are not the same outcome. The caller needs to render “you stopped this” differently from “the provider stopped making progress,” and recovery logic needs to know whether a retry would violate user intent.

42.4 Stream Idle Is a Different Clock

The turn watchdog knows how long the loop has been in awaiting_llm. It does not own the mechanics of a blocked SSE body.

The gateway therefore runs a per-chunk timer inside CompleteStream. Every received line resets it. At half the configured gap it logs a warning; at the full gap it cancels the scanner and closes the response body. The snapshot default is 90 seconds.

This catches a failure the outer turn timer handles poorly: a connection that delivered some content and then stopped producing bytes without returning an error. Retrying the same hung upstream in non-streaming mode can simply buy another full transport wait, so the loop treats the stream-gap error as a partial outcome. It preserves text already received, inserts an explicit placeholder when there was none, records a deadline-exceeded status, and exits.

The clocks answer different questions:

• Turn idle: has this LLM phase existed too long?
• Stream idle: has this particular transport produced a chunk recently?
• Tool timeout: has this tool exceeded its own execution contract?

Do not reset one from the events of another. A screenshot tool finishing should not forgive a previously stuck provider stream, and a provider token should not extend a tool's deadline.

42.5 Nested Calls Still Need Ownership

Compaction is a local phase, so the watchdog should not count the CPU work that shapes history. But compaction also calls an LLM to persist learnings or generate a summary. That nested remote call does need coverage.

The phase tracker handles this with EnterTransient(PhaseAwaitingLLM), which returns an idempotent restore closure:

restore := tracker.EnterTransient(PhaseAwaitingLLM)
summary, err := client.Complete(ctx, request)
restore()

The outer phase remains compacting; only the blocking remote section temporarily belongs to the LLM watchdog. Forget the restore and the loop stays in the wrong phase. Skip the transient entirely and a hung summarizer becomes invisible.

This is why manual “pause watchdog / resume watchdog” calls spread through business logic are brittle. The phase owns the timing policy. Nested work borrows a phase and returns it.

42.6 What If the Phase Tracker Lies?

There is a dangerous temptation here: call the tracker “fail closed” and cancel whenever its state is inconsistent.

But an inconsistent tracker cannot tell you whether the loop is waiting on a model or legitimately executing a tool. Acting on it can kill valid work. The snapshot chooses the opposite production trade: any structural violation marks the tracker invalid, logs a warning, and makes watchdog observers disable themselves for the rest of the run. Under tests or SHANNON_PHASE_STRICT=1, the same violation panics so development cannot normalize it.

So the behavior is:

development / strict run: structural bug → panic
ordinary production run: structural bug → warning + watchdog disabled

That can leave a real hang alive. The alternative can cancel a valid destructive or expensive operation on untrustworthy evidence. Neither is free; the important thing is to choose explicitly and make the lost coverage visible.

The restore closure is idempotent, which removes one class of ownership bug. The exit assertion catches a forgotten transient. And top-level Enter during an open transient is itself a violation. These checks turn “timer feels weird” into a concrete phase-layering error.

42.7 Timeout Is an Outcome, Not Just an Error

When hard idle fires, cancellation must reach four seams:

1. the in-flight provider or stream;
2. any speculative tool work started from stream deltas;
3. checkpoint/persistence so the turn is not left falsely complete;
4. the caller, with phase and partial-output status.

Returning context canceled is not enough. Without the typed cause, the UI cannot distinguish watchdog from user intent. Without partial text, a nine-minute response vanishes because the final ten seconds hung. Without a terminal run status, the persistence layer from Chapter 40 and its callers cannot distinguish a timeout-truncated turn from clean completion.

Likewise, the soft event should include the active phase and measured idle duration. “Agent slow” gives an operator nothing. “No LLM activity for 90s, phase=force_stop” says the final synthesis call is the owner.

Watchdogs should bound silence, not conceal it.

42.8 Snapshot Evidence

These describe one dated implementation and transport stack, not universal timeout values.

42.9 Common Pitfalls

One timeout for the whole turn. It kills legitimate tools or waits too long on dead providers.

Resetting on any activity. Unrelated progress can keep a genuinely stuck owner alive forever.

Excluding all of compaction. The local wrapper is not idle-counted, but its nested summary call still needs LLM coverage.

Counting approval wait as agent idle. A human owns that pause. Cancelling it on the model's timer is category error.

Using phase names without transition identity. Re-entering awaiting_llm never re-arms the soft warning.

Falling back after a stream-gap timeout automatically. The same upstream may hang again, doubling latency and spend.

Cancelling from invalid phase data. Once ownership is untrustworthy, a timer no longer knows what it is killing.

Returning a generic cancellation. The caller loses cause, phase, partial output, and the ability to choose recovery.

Key Points

1. Time has an owner. A duration is meaningful only inside a known blocking phase.
2. Soft is visibility; hard is policy. Keep them independently configurable and observable.
3. Stream gaps need a transport clock. Turn elapsed time cannot replace time-between-chunks.
4. Nested remote work borrows the remote phase. Phase ownership beats scattered suspend/resume calls.
5. Untrustworthy timing data must not trigger confident action. Surface the structural bug and choose the production trade deliberately.

Next: Chapter 43 handles a different kind of non-progress — a loop that is active, producing events, and still going nowhere.