Chapter 36: Tool Result Budget and Spill

"Just truncate the big ones" — truncate and the model re-reads. Spill it to disk with a pointer and it doesn't have to.

Quick Track (Master the core in 5 minutes)

1. Truncation destroys; spilling relocates. Keep a preview and a retrievable pointer
2. Two budgets: one per result, one per turn — the second is what parallel batches blow
3. The replacement must persist across turns, or you re-spill the same result forever
4. Some tools self-bound and are exempt; the aggregate target is best-effort, not a ceiling
5. Count runes, not bytes, or your preview cuts a character in half

Public-source snapshot: implementation details are examples from public Kocoro origin/main at commit 4ec6772, reviewed 2026-07-27. Treat the invariant as the lesson and re-check constants against current source.

36.1 Starting with Ten Search Results That Cost More Than the Answer

Ten calls to a synthetic MCP repo_search tool go out in one parallel batch. It inherits the runtime's default result policy, and each call comes back with roughly 30,000 characters of matches.

repo_search("handleRequest") → 31,204 chars
repo_search("func New")      → 28,910 chars
repo_search("TODO")          → 34,332 chars
repo_search("import")        → 26,180 chars
repo_search("error")         → 32,441 chars
repo_search("context")       → 29,608 chars
repo_search("session")       → 30,774 chars
repo_search("tool_use")      → 27,955 chars
repo_search("cache")         → 33,116 chars
repo_search("message")       → 25,480 chars
                               ─────────────
                               300,000 chars ≈ 86K tokens

The synthetic name matters. Kocoro's built-in grep declares a lower, roughly 20,000-character tool-specific limit and would spill these results earlier. A default-policy MCP tool exposes the aggregate failure cleanly: every result stays below 50,000 while their sum does not.

Not one of those individually looks alarming. Together they consume nearly half of a 200K-token window, arriving in a single turn, for a question the model will answer by reading maybe twenty lines out of all of it.

Your first instinct is to truncate — cap each result at, say, 10,000 characters and move on. That works exactly once. The model reads the truncated result, doesn't find what it needs, and calls repo_search again with a narrower pattern. Now you've paid twice and you still don't have the content.

Truncation destroys information the model may need. Spilling relocates it. That's the whole distinction this chapter turns on.

36.2 Preview Plus Pointer

The mechanism is simple: when a result exceeds a threshold, write the full payload to a runtime-owned file and replace the in-prompt content with two things — a preview of the opening, and a path the model can read later.

In the public runtime the default threshold is 50,000 characters and the preview is the leading 2,000 runes. A tool can declare a lower limit, or an unlimited/self-bounded policy, and that policy wins. So the model still sees what kind of thing it got and roughly what's in it, and if that's enough it moves on. If it isn't, file_read on the spill path retrieves the rest — at a moment when the model actually knows what it's looking for.

That last clause is the real win. The retrieval happens after the model has narrowed the question. You've turned "carry 300K characters just in case" into "carry 20K now, fetch precisely what's needed later."

Note the preview is measured in runes, not bytes. Slice a UTF-8 string at byte 2,000 and you can land mid-character, producing a broken rune the model sees as garbage — and in CJK text, byte-based counting also gives you roughly a third of the preview you intended.

36.3 The Budget That Parallel Execution Blows

A per-result threshold is not enough, and §36.1 is why. Every one of those ten default-policy search results was under 50,000. None triggered per-result spilling. The turn was still 300,000 characters.

So there's a second budget: a per-turn aggregate, 200,000 characters in the snapshot. After the batch completes, the runtime spills the largest eligible results until the total fits under it — largest-first, because that clears the most budget for the fewest spill operations.

There's a floor on this too: nothing smaller than 5,000 characters gets spilled during aggregate reduction. Below that size, the spill overhead — a file write, a path in the prompt, a probable file_read later — costs more than the content you removed.

The aggregate target is best-effort, not a hard ceiling. A turn can legitimately end above it: some tools declare themselves unlimited and are skipped when choosing spill candidates, and a spill that fails leaves the content in place rather than dropping it. Treat exceeding the target as a signal worth investigating, not as an impossible state your code can assume away.

36.4 The Exemption That Looks Like a Bug

file_read is exempt from spilling. That looks wrong until you see what it does instead: it self-bounds at 500,000 runes inside the tool, with an explicit truncation marker.

The reasoning is worth extracting. Spilling file_read would mean writing a file's contents to another file and handing the model a path — when the model already had a path, the one it just read. You'd have accomplished nothing except adding a layer of indirection and a wasted round-trip.

So the rule generalizes: a tool that already returns a retrievable reference doesn't need spilling; it needs a self-imposed bound. Spilling is for tools whose output exists nowhere else. Reads, by definition, have a source.

This is why the exemption is a property the tool declares, not a name on a list somewhere in the dispatcher. The tool knows whether its output is recoverable.

36.5 The Replacement Has to Outlive the Turn

Here is the part that's easy to miss and expensive to get wrong.

You spill a result on turn 12. On turn 13, the conversation history is rebuilt to send to the model. If that rebuild uses the original tool result rather than the spilled replacement, the 60,000 characters come straight back into the prompt — and you spill them again, and again, every turn, forever.

The fix is that the replacement is persisted state, not a per-turn transformation. A record of which results were replaced and with what, carried across checkpoints and terminal saves, so a rebuilt history reconstructs the spilled version rather than the original.

The Seen half of that state matters as much as the replacement map: it distinguishes "this result was never spilled" from "this result was spilled and here is its stand-in." Without it, a missing entry is ambiguous, and ambiguity here means either re-spilling or silently losing content.

Any prompt-shaping operation that isn't persisted will be redone every turn. That applies to spilling, to compaction, and to every rewrite in Chapter 37.

36.6 Snapshot Evidence

These describe one dated implementation, not a universal contract.

36.7 When Not to Spill

Spilling has a cost the threshold hides: it converts content the model already has into content the model must ask for. Every spill is a potential extra round-trip.

So don't spill small results — the floor exists for exactly this reason. Don't spill a result the model is certain to need in full; if the next step is obviously "summarize this document," relocating it just adds a fetch. And don't spill when the content is the answer rather than the evidence: a tool whose output is what the user asked for should reach them, not a preview and a path.

There's also a lifecycle question people skip. Spill files accumulate. They're session-scoped and they need cleanup, and they contain whatever the tool returned — which may be exactly the data your permission model works to keep contained. A spill directory is a copy of tool output sitting on disk in plaintext. Scope it, permission it, and delete it.

36.8 Common Pitfalls

Truncating instead of spilling. The model re-runs the tool with a narrower query. You pay twice and lose the original.

Only budgeting per result. Ten default-policy results at 30K each pass every per-result spill check and still blow the turn. §36.1 is this bug.

Not persisting the replacement. Re-spills every turn, forever, and the symptom looks like a mysterious repeated slowdown rather than a bug.

Byte-slicing the preview. Breaks multi-byte characters and silently gives CJK users a third of the preview length.

Spilling tools that self-bound. Indirection with no benefit — the content already had a retrievable address.

Treating the aggregate as a hard cap. Exempt tools and failed spills both exit above it. Code that assumes otherwise will be surprised in production.

Key Points

1. Truncation destroys, spilling relocates. Keep a preview and a pointer so retrieval stays possible.
2. Budget per result and per turn. Parallel batches pass every per-result check and still overflow.
3. Persist the replacement. An unpersisted prompt-shaping operation gets redone every single turn.
4. Tools that self-bound don't need spilling. If the output already has an address, relocating it buys nothing.
5. The aggregate is a target, not a ceiling. Exempt tools and failed spills legitimately exceed it.

Next: Chapter 37 handles the results that stay in the prompt — degrading them by age instead of moving them out.