Chapter 45: Computer-Use Context Management

The current viewport is evidence. Eleven obsolete viewports are a tax — unless the user asked you to compare all twelve.

Quick Track (Master the core in 5 minutes)

1. Computer use grows two histories at once: text observations and images
2. Use four independent controls: per-observation cap, text window, browser-image window, global image ceiling
3. Scope browser pruning by tool identity so user uploads and batch-vision tasks keep a looser budget
4. Replace stale content with explicit placeholders while preserving tool_use_id pairing
5. Make filters idempotent and observable; pruning the prompt is not deleting the backing artifact

Public-source snapshot: implementation details are examples from public Kocoro origin/main at commit 4ec6772, reviewed 2026-07-27. Re-check constants against current source.

45.1 Starting with Eleven Screens the Agent Cannot Act On

A computer-use agent opens a settings panel, takes a screenshot, reads the accessibility tree, clicks a tab, and repeats. After twelve actions, the next model request contains:

12 screenshots
12 page / accessibility observations
12 assistant decisions
11 viewports that no longer exist

Those eleven older screenshots are expensive and usually useless. The model cannot click any of them. The UI has moved on.

But “usually” is doing important work. If the user's task is “compare these twelve screenshots,” every image is the product. A global “keep one image” rule that is perfect for browser navigation destroys batch vision.

The same mismatch exists in text. A DOM dump can be 80,000 characters even when it is the newest observation. Keeping three of them does not stop one oversized page from dominating the prompt.

Computer-use context is not one budget. It is several payload classes with different half-lives.

45.2 Four Controls, Four Failure Modes

The public snapshot separates four controls:

The numbers are workload choices. The separation is architectural.

A per-item cap cannot bound item count. A sliding window cannot bound one pathological item. A browser-specific image window cannot protect a batch-image run. A global ceiling cannot know that the latest viewport is more actionable than a user upload from six turns ago.

Each control should have one owner, one configuration knob, and one public-seam test. Combining them into max_context_items produces a number nobody can tune without breaking another workload.

45.3 Cap at Capture; Age at Assembly

The 24,000-rune control binds when a browser or GUI tool result first enters context. It keeps the front of the observation and appends a self-describing marker:

[browser observation truncated: 81342 chars total]

The marker is included inside the cap, so the returned observation never quietly exceeds the configured maximum. Counting runes instead of bytes avoids splitting Chinese, Japanese, or emoji and keeps “24,000 characters” meaningful across languages.

The text window acts later, while assembling each request. It keeps the three newest GUI observations at full fidelity and replaces older string results with a one-line stub:

[elided browser observation: browser_snapshot, was 23811 chars]

These controls solve different time dimensions. Capture-time truncation says, “this single state was too large even when fresh.” Aging says, “this state used to matter and no longer deserves full fidelity.”

This producer-specific aging runs before generic context-pressure compression. That is why Chapter 37's browser Tier-2 floor is scoped to the generic compressor: it stops that mechanism from collapsing an actionable snapshot to metadata, while this window deliberately ages obsolete GUI states earlier.

That is the same distinction Chapter 36 made between result size and turn total, applied to observations instead of generic tool output.

45.4 Browser Images Are Not User Images

Screenshots are nested inside tool_result blocks, but not every image in a conversation came from a browser. A user may upload reference art. file_read may return a diagram. A vision task may intentionally carry dozens of images.

So browser screenshot pruning first identifies tool calls in the GUI family and follows their tool_use_id into matching results. Only those images fall under the “keep the newest one” window. User uploads and non-GUI tool images remain untouched by that pass.

Then the global filter applies the looser ceiling of 50 image-bearing messages across top-level and nested images. Running the narrow filter first matters: the global pass sees an already-thinned browser history and spends its generous budget on the workloads that actually need it.

The rule generalizes beyond images: scope an aggressive budget to the producer whose workload justifies it. A cheap global classifier is not a substitute for knowing where the data came from.

45.5 Preserve the Conversation Contract

Removing a screenshot block outright can break the provider message contract. The assistant's tool_use still exists, but its matching tool_result has changed shape or vanished.

The filters therefore preserve the result block and its tool_use_id, replacing only the stale payload with explicit text:

[previous screenshot removed to save context]

The model now knows information was removed, the provider still sees a valid call/result pair, and an audit trace can explain why a later decision lacked the old pixels.

The text filter follows the same rule. It changes the string inside a result, not the surrounding turn structure. This is the pairing invariant from Chapter 37, specialized for GUI observations.

Silent deletion is the worst option. It saves tokens while making the transcript lie about what the model was allowed to see.

45.6 Idempotency Protects the Cache

These filters run before repeated model calls. If an already-stubbed observation is rewritten into a slightly different stub on every iteration, the prefix changes every iteration and Chapter 39 starts missing at the same message forever.

The text stub has a recognizable prefix, so subsequent passes skip it. Image replacements are stable text blocks. With cache-debug telemetry enabled, every first-time mutation emits a cache-compaction event (obs_window, browser_img_strip, or the global img_strip) with old/new hashes; no-op revisits emit nothing.

That gives the intended lifecycle:

full observation → one deterministic placeholder → byte-stable thereafter

Idempotency here is not just code cleanliness. It is a billing property.

45.7 Prompt Retention Is Not Artifact Retention

All four controls operate on the request's message view. They do not, by themselves, delete a screenshot file, erase the full audit event, or revoke a user upload.

Those are different policies:

• Prompt retention: what the model receives on the next iteration
• Session/audit retention: what operators can reconstruct later
• Temporary-file cleanup: when local screenshot bytes leave disk
• User-asset ownership: whether an uploaded file remains available to the user

Tie them together and “save context” can become “destroy evidence.” Keep them separate and you can prune an old viewport from the model while retaining the trace needed to debug a bad click.

The model still needs disclosure. A placeholder states what was removed and why. Operators need telemetry. Storage needs its own lifecycle. One pruning function should not pretend to own all three.

45.8 Tune from Tasks, Not Token Anxiety

The current viewport is usually enough for a click, but not always. A form may show an error only after navigation. A comparison workflow may need the previous screen. A canvas task may need visual continuity across several steps.

So tune with real sequences:

1. navigation across several pages;
2. a modal that must be compared before and after;
3. a batch of user-provided images;
4. a huge multilingual accessibility tree;
5. repeated filtering of the same history;
6. resume from persisted messages after older images were stubbed.

Measure task success, model calls, input tokens, cache drift, and re-observation rate. If the agent keeps taking the same screenshot again, the window is too tight or the placeholder removed the wrong signal. If success is stable and context falls sharply, the budget is doing real work.

Do not derive a universal limit from one clean demo. Computer use is where the interface changes faster than your intuition.

45.9 Snapshot Evidence

These describe one dated implementation and workload, not universal observation limits.

45.10 Common Pitfalls

One image limit for every workload. Browser navigation wants the newest viewport; batch vision may need every upload.

A window without a per-item cap. One enormous DOM dump still dominates the prompt.

A cap without a window. Thousands of individually legal observations still accumulate.

Counting bytes as characters. Multibyte text gets less useful context and may be cut mid-rune.

Dropping the whole result block. Breaks tool-call pairing and can make the provider reject the request.

Silent pruning. The transcript implies the model saw evidence that was actually removed.

Non-idempotent placeholders. Repeated assembly redirties the same prefix and thrashes the cache.

Deleting backing artifacts during prompt pruning. Context policy becomes an accidental retention policy.

Tuning on one task. A limit that excels at clicking through forms may destroy visual comparison.

Key Points

1. Computer use has several context lifecycles. Text, GUI screenshots, and user images do not age the same way.
2. Cap size and count separately. One huge observation and many normal observations are different failures.
3. Scope aggressive pruning by producer. Browser history can be tight without punishing batch vision.
4. Preserve pairing and disclose loss. Replace payloads with explicit, stable placeholders.
5. Prompt pruning is a view, not deletion. Audit, files, and user assets keep their own lifecycle owners.

This closes Part 10's loop: context compaction, tool-result budgeting, cache stability, steering, and time discipline all meet in the observation-heavy workflow introduced in Chapter 28.