Chapter 39: Prompt Cache Stability

The provider does not cache meanings. It caches a prefix, and the first different byte ends the reuse.

Quick Track (Master the core in 5 minutes)

1. Cacheability is a serialization contract — semantic equivalence does not help
2. Allocate scarce breakpoints by stability: cross-user, tool-schema, rolling-history, per-session
3. Put volatile values after the last stable boundary; do not let a timestamp poison the tools prefix
4. Enforce permissions at execution time so authorization changes do not reorder or remove schemas
5. Fork from the exact request that was sent, append only, and instrument every intentional rewrite

Public-source snapshot: implementation details are examples from public Kocoro origin/main at commit 4ec6772, reviewed 2026-07-27. Treat the invariant as the lesson and re-check constants against current source.

39.1 Starting with the $0.67 Rewrite That Changed Nothing

Chapter 35 ended with a small-looking truncation bug. One oversized user message was clipped to a boundary derived from the current history. Add a follow-up, the history grows, the boundary moves, and the message is clipped at a different byte.

The visible meaning barely changes. The cache sees a new prefix.

That one moving cut point re-billed the whole message as fresh cache creation: roughly $0.67 per follow-up turn in the recorded incident. Nothing got more accurate. Nothing got faster. The system simply serialized the same logical input differently every time.

This is the useful way to think about prompt caching: not as an optimization the provider may happen to discover, but as an interface your request builder must preserve. If identical logical state can produce different bytes, you do not have a cacheable prompt. You have a cache lottery.

39.2 The Cache Key Ends at the First Difference

Imagine two consecutive requests:

request N:   system | tools | history A B C | current turn
request N+1: system | tools | history A B C | current turn | follow-up

That is the ideal shape. Request N is a byte-for-byte prefix of request N+1, so the expensive stable portion can be reused.

Now let one innocent thing drift near the front:

request N:   system | tools [a,b,c] | history ...
request N+1: system | tools [b,a,c] | history ...
                              ^
                         reuse ends here

The tools are the same. The set is the same. The model would understand either order. None of that matters to a prefix cache.

The drift sources are mundane: iterating a map, filtering schemas after a Skill activates, injecting the current time into the system prompt, serializing {} as null, or mutating a historical message during compaction. The discipline is therefore mundane too: deterministic ordering, canonical encodings, explicit stable/volatile regions, and tests over serialized bytes rather than object equality.

39.3 Four Breakpoints Are an Allocation Problem

The public snapshot has four cache breakpoints to spend. Kocoro places content and emits one plain-text <!-- cache_break --> marker; Cloud translates the layout into provider cache_control blocks.

The numbering is historical, not positional: BP #4 appears before BP #3 on the wire. BP #3 is not “the first user message” either. Persisted history strips the scaffold, so exactly one <!-- cache_break --> exists in a request however long the session becomes.

This is not trivia. Put the rolling marker on the wrong message and you lose history reuse. Preserve an old marker by stripping a current one and you may change the block bytes you were trying to match. Add a fifth logical region and the provider does not grant you a fifth slot — something else must move or merge.

The architectural lesson is allocate boundaries by who shares the bytes and how often they change:

cross-user stable → session-stable → rolling history → current-turn volatile

Every value belongs to one of those lifetimes. If you cannot say which one, it will eventually land in the most expensive wrong place.

39.4 Stable and Volatile Are About Position

The system block must be byte-identical across users who genuinely share it. Per-user MCP names, deferred-tool listings, and other configuration-derived values cannot live there; they move to the per-session prefix or the volatile tail.

The current date, working directory, memory recall, raw user text, Skill listing, and language directive live after the <!-- cache_break --> marker. They are uncached on the turn that creates them. On the next turn, the rolling history boundary can absorb them.

That placement is more important than the labels. A <!-- volatile --> marker inside the system prompt sounds cleaner, but in the recorded implementation those bytes still appeared before the tool breakpoint. A minute-changing timestamp there invalidated the tools cache every minute. “Volatile” written in a comment does not make bytes invisible; only their on-wire position does.

39.5 Authorization Must Not Reshape the Prefix

Chapter 38 separated discovery from authorization. The same separation protects the cache.

When an active Skill declares allowed-tools, the tempting implementation is to remove every disallowed schema before the next model call. That makes the prompt look secure, but the tools array now changes mid-run. BP #2 misses, and every byte after it loses reuse too.

The snapshot keeps the tools array stable and checks activeSkillFilter at execution time, immediately before a call runs. A denied tool produces a denial result; it never executes. Security still has an owner, but authorization no longer mutates the schema prefix.

That distinction is load-bearing:

• Deferral decides which schemas the model sees initially.
• Authorization decides which calls may execute.
• Caching requires both mechanisms to preserve their declared byte boundaries.

Merge those three into one “tool filtering” step and each starts breaking the other two.

39.6 Fork the Request; Do Not Reconstruct It

Post-turn suggestions and speculative calls want the main request's warm prefix. Rebuilding that request from higher-level state is a trap: one defaulted field, reordered schema, or different thinking budget is enough to make the fork cold.

The safer shape is BuildForkedRequest(main, opts): start from the exact request captured at dispatch, allocate a fresh Messages slice, copy the Thinking pointer target, and append only the fork-specific messages. The tools slice remains shared and must be treated as read-only. The function permits only the divergences it names.

After it returns, do not “optimize” the fork by lowering MaxTokens, changing temperature, filtering tools, or clamping thinking. Those values participate in the cache contract. A fork that is cheaper per cold call but misses the parent prefix can be more expensive overall.

This is the broader pattern from Chapter 34: capture the real artifact at the public seam. A reconstructed approximation is not evidence that the bytes sent to the provider were preserved.

39.7 Measure Drift, Not Just Hits

A cache miss tells you that reuse failed. It does not tell you where the first byte changed or why.

So, when SHANNON_CACHE_DEBUG=1 enables cache-debug telemetry, intentional in-place rewrites emit the action, message index, and old/new hashes. Request logs can then be joined to the rewrite immediately before them: Tier 1 compression, observation-window pruning, image stripping, or query-time budgeting. No rewrite event plus a changing prefix hash points you toward nondeterministic serialization instead.

Also keep attribution separate from policy. cache_source labels Desktop, TUI, channel, schedule, helper, and one-shot traffic so their cost and cache ratios can be compared. In this snapshot it does not select the TTL. Treating an attribution field as a policy switch creates a second, undocumented cache system.

Track cache reads, cache creation, latency, model calls per user turn, and task success together. A high hit rate can coexist with too many calls; a low creation/read ratio can hide a workload that stopped completing. Cache metrics are operational signals, not the product.

39.8 Snapshot Evidence

These describe one dated implementation and provider layout, not a universal cache contract.

39.9 Common Pitfalls

Testing objects instead of wire bytes. Two maps can be deeply equal and serialize in a different order. Snapshot the actual request representation.

Filtering schemas for permission. Secure at execution, cold at BP #2. Keep authorization and schema presentation separate.

Putting “volatile” data before a stable breakpoint. The name does nothing. A timestamp before tools invalidates tools.

Reconstructing a fork. Defaults drift. Capture the dispatched request and append.

Mutating the fork after construction. A smaller output limit can alter a cache-key field and destroy the reuse the fork existed to obtain.

Preserving every old marker. More markers are not automatically more reuse. Under a fixed cap, freeing a slot can mutate the very block you hoped to match.

Treating cache_source as TTL policy. It is attribution in this snapshot. Policy belongs to the service that owns the cache controls.

Watching hit rate alone. Cost, latency, call count, and completion quality decide whether the cache design is helping.

Key Points

1. Cacheability is a serialization contract. Meaning-equivalent requests can still be byte-different misses.
2. Breakpoints are scarce architecture. Allocate them by sharing scope and change frequency.
3. Authorization belongs at execution. Do not mutate a stable tools array to express a run-time denial.
4. Fork from the dispatched artifact. Append only; reconstruction and post-build customization invite drift.
5. Every intentional rewrite needs attribution. A miss without a cause is not observable enough to fix.

Next: Chapter 40 applies the same “persist the real state” discipline to turns that must survive a process crash.

Continue Reading

Byte-Stability Tests for Prompt Caching describes tool-schema freezing, deterministic ordering, and request snapshot tests. Use it alongside this chapter to examine how your request builder preserves cache reuse.