Chapter 38: Deferred Tool Loading and Tool Search

"Defer the schemas once there are too many tools" — except tool count is the wrong unit, and the tool you defer wrong costs you six seconds on every session's first reply.

Quick Track (Master the core in 5 minutes)

1. Budget tokens, not tool count — thirty tiny schemas can be cheaper than five fat ones
2. Two triggers, not one: budget exceeded or a cold always-defer category is present
3. Some tools must never defer — the session-opening ones, where a search round-trip is pure latency
4. The warm set is session-scoped and starts cold; key it by schema fingerprint, not name
5. Deferral hides schemas. It does not deny permission — those are different systems

10-Minute Path: 38.1-38.3 -> 38.5 -> Kocoro OSS Lab

Public-source snapshot: implementation details are examples from public Kocoro origin/main at commit 4ec6772, reviewed 2026-07-27. Treat the invariant as the lesson and re-check constants against current source.

38.1 Starting with a Prompt That Was 60% Table of Contents

You connect four MCP servers. A GitHub server, a Postgres server, a Slack server, a browser automation server. Between them, 74 tools.

Now look at what you're sending on every single request:

system prompt        1,800 tokens
tool schemas        23,400 tokens   ← 74 tools, full JSON Schema
conversation         4,100 tokens
─────────────────────────────────
                    29,300 tokens

Twenty-three thousand tokens of tool definitions, re-sent on every iteration of every turn, to answer a question that touches two tools. The model is reading a 74-item catalogue to pick one line from it, and you're paying for the catalogue every time.

The obvious fix is to stop sending schemas the model doesn't need, and let it ask for them. That's deferred loading, and the mechanism is simple. The interesting part is that almost every naive version of it makes things worse, in ways that only show up in production.

38.2 Tool Count Is the Wrong Unit

The first instinct is a count threshold: more than thirty tools, switch to deferred mode.

But schemas are not the same size. A read_file schema with two string parameters is a couple hundred tokens. A browser automation tool with fifteen optional parameters, nested selector objects, and enumerated wait conditions can be several thousand on its own. Thirty of the first are cheaper than five of the second.

So budget the thing you actually care about. The public runtime estimates schema tokens directly and compares against a budget:

schemaTokenBudget = 8000      // ~28K chars of compact schema JSON
charsPerTokenSchema = 3.5     // conservative ratio, mirrors the context estimator

Eight thousand tokens, derived as roughly 28K characters of compact schema JSON at a deliberately conservative characters-per-token ratio. Note the conservatism is intentional — under-estimating your schema cost means blowing the budget you were trying to protect, so the ratio errs toward assuming schemas are expensive.

Measure what you're spending, not what you're counting. A count threshold will be wrong the day someone connects one verbose server.

38.3 Two Triggers, Because Budget Alone Misses a Case

Here is where it gets more interesting than "check the budget." The actual condition is a disjunction:

deferredMode := len(coldDeferred) > 0 &&
    (shouldDefer(a.tools, a.tools.SortedNames(), schemaTokenBudget) ||
        hasCategoricalDeferred(coldDeferred))

Budget exceeded, or the cold set contains a tool from an always-defer category.

Why the second clause? Because some tools are expensive and almost never used in a given context. macOS GUI automation, scheduling, headless process control — a one-shot CLI task touches none of them, but their schemas ride along in every request anyway. The always-defer set covers exactly that:

Keeping them in the deferred set means cold-start tools[] ships ~5K fewer tokens; sessions that DO need them pay one extra tool_search round-trip per session.

That's the trade stated plainly. Five thousand tokens saved on every cold start, against one extra round-trip for the sessions that actually need a GUI tool. For a CLI workload where most sessions never touch the desktop, that's obviously worth it — and note that it's a judgment about your workload, not a universal truth. The right always-defer list for a desktop automation product would be nearly the inverse.

38.4 The Tools You Must Never Defer

Now the part that's easy to get wrong, and the reason this chapter has a number in its opening line.

Gateway tools are deferred-eligible by default. But two of them are explicitly exempt, and the reasoning is worth reading in full:

web_search/web_fetch are the most common opener of a NEW session (e.g. "what's the news on X") — deferring them forces the model into an extra tool_search round-trip before it can call the tool at all, adding ~6s of observed latency to the very first reply of a session, every session, since the WorkingSet warm cache is session-scoped and starts cold each time.

Read that carefully, because there are two distinct facts in it.

First: the warm cache is session-scoped and starts cold. Whatever the model learned about available tools last session is gone. Every new session pays full price for its first deferred lookup.

Second: the cost lands on the first reply. Not on some deep iteration where a user has already committed attention — on the very first thing they see after asking. Six seconds of nothing, before any output, on the interaction that sets their impression of whether the system is fast.

So the rule isn't "defer the expensive schemas." It's defer the schemas whose retrieval cost lands somewhere the user isn't waiting. A tool that opens sessions is the worst possible thing to defer, no matter how big its schema is.

memory_recall is exempt for a related but distinct reason: it's the fallback path when implicit memory injection turned up nothing. Defer it and the observed failure mode was the model hallucinating an older version of the schema on its first attempt, burning a turn. The fix costs about 1.5K extra schema bytes — which, and this is the key point, ride along in the cacheable system prefix. Paid once per session, not once per turn.

That last detail is the one to carry: when a schema lives in the cached prefix, its cost is amortized across the whole session. "Expensive" schemas in a stable prefix are much cheaper than they look.

38.5 The Warm Set, and Why It Keys on Fingerprint

Once the model has searched for a tool and received its schema, sending it again is free in attention terms and cheap in tokens. So you keep it — that's the warm set, a session-scoped working set of schemas the model has already pulled.

The subtle failure: an MCP server updates. Its tool now takes a different parameter shape. Your warm set still holds the old schema, the model calls with old-shaped arguments, and the tool rejects them — or worse, silently accepts and does the wrong thing.

Which is why the warm set is invalidated by schema fingerprint, not by name. If the underlying effective toolset changes, warm entries derived from the old definitions stop being valid. Name-keyed caching is the bug; content-keyed caching is the fix.

38.6 Deferral Is Not Authorization

This one is worth being blunt about, because it looks like security and isn't.

Hiding a schema from the model does not stop the tool from being called. The model can still emit a call by name — it may have seen the name in the search catalogue, in a skill description, or simply guessed it. Deferral is a retrieval optimization over capability descriptions. It shapes what the model sees, not what it may do.

Permission is a separate system, enforced at execution time. If your only defence against a dangerous tool is that its schema wasn't in the prompt, you have no defence. Chapter 25 and Chapter 33 cover the actual authorization layer; this chapter is only about token economics.

The same separation appears in skill restrictions, incidentally: allowed-tools is enforced as execution-time denial rather than schema filtering, specifically so the tools array stays byte-stable for the prompt cache. Two systems, two mechanisms, deliberately not merged — which is the same discipline Chapter 39 is built on.

38.7 Snapshot Evidence

These describe one dated implementation, not a universal contract.

38.8 Common Pitfalls

Budgeting by tool count. Covered above. It breaks the day one verbose server connects.

Deferring a session opener. The six-second tax, paid on the first reply, every session. Look at what actually starts sessions in your logs before choosing what to defer.

Name-keyed warm entries. Works until a server updates its schema, then produces malformed calls that look like model errors.

Forgetting the warm set starts cold. Reasoning about steady-state behaviour and forgetting that every new session begins at zero is how the six-second problem gets missed in testing — it never shows up in a long manual session.

Treating deferral as a security boundary. It is not one. Denial happens at execution.

Deferring so aggressively the model can't discover anything. If the search catalogue only lists names, and the names are cryptic, the model cannot form a good query. Names in a fallback catalogue are load-bearing — make them descriptive.

Kocoro OSS Lab (10 minutes)

1. Read the neverDeferTools comment and note the structure of the argument: an observed latency number, attached to a specific user-visible moment, attached to a specific cache property.
2. Look at alwaysDeferTools and ask which entries would be wrong for a desktop automation product. The list encodes a workload assumption.
3. Estimate your own schema token spend: serialize your tool definitions to compact JSON, divide characters by 3.5. Compare against your context window. If it's more than a few percent, you have this problem.

Key Points

1. Budget tokens, not tool count. Schema sizes vary by an order of magnitude.
2. Two triggers. Budget exceeded, or a cold always-defer category present — the second catches expensive-but-unused tools that never trip the budget.
3. Never defer a session opener. The retrieval cost lands on the first reply, on a cache that starts cold every session.
4. Cached-prefix schemas are cheaper than they look. Cost amortizes across the session, so "expensive but always loaded" can beat "cheap but deferred."
5. Deferral is retrieval, not authorization. Denial belongs at execution time, and merging the two gives you neither.

Next: Chapter 39 is the discipline that makes the cached prefix worth protecting in the first place.