CH 05 · Prompt Caching and Context Management · 09/15
Read as one pageReal code: OpenCode marks the stable edges
pi spends a marker on the system text, the last tool, and the latest user message. OpenCode has a harder budget. applyCachePolicy runs before the provider body is built, and it may attach at most four CacheHint tags. A CacheHint is the in-memory tag that later becomes cache_control on the wire. The default policy, AUTO, asks for the last tool, the system parts, and a tail of one message. Manual tags already on the request count against the four.
const AUTO: CachePolicyObject = {
tools: true,
system: true,
messages: { tail: 1 }, // [1]
}
const NONE: CachePolicyObject = {}
const BREAKPOINT_CAP = 4
Source: anomalyco/opencode, packages/ai/src/cache-policy.ts (MIT)
const markSystemBoundaries = (system: LLMRequest["system"], hint: CacheHint, budget: Budget): LLMRequest["system"] => {
if (system.length === 0) return system
let changed = false
const next = system.map((part, index) => {
if ((index !== 0 && index !== system.length - 1) || part.cache || budget.remaining === 0) return part // [2]
budget.remaining -= 1
changed = true
return { ...part, cache: hint }
})
return changed ? next : system
}
Source: anomalyco/opencode, packages/ai/src/cache-policy.ts (MIT)
const markMessages = (
messages: ReadonlyArray<Message>,
strategy: NonNullable<CachePolicyObject["messages"]>,
hint: CacheHint,
budget: Budget,
): ReadonlyArray<Message> => {
if (messages.length === 0) return messages
if (strategy === "latest-user-message")
return markMessageAt(messages, lastIndexOfRole(messages, "user"), hint, budget)
if (strategy === "latest-assistant")
return markMessageAt(messages, lastIndexOfRole(messages, "assistant"), hint, budget)
let start = messages.length
let remaining = strategy.tail // [3]
while (remaining > 0 && start > 0) {
start -= 1
if (effortUpdate(messages[start]!) === undefined) remaining -= 1 // [4]
}
let next = messages
for (let i = start; i < messages.length; i++)
if (effortUpdate(messages[i]!) === undefined) next = markMessageAt(next, i, hint, budget)
return next
}
Source: anomalyco/opencode, packages/ai/src/cache-policy.ts (MIT)
What to notice
- [1]
tail: 1means one conversation breakpoint, on the newest real message. The tools and the system text take the earlier slots.applyCachePolicymarks in that order: last tool, then system boundaries, then messages, and it stops whenBREAKPOINT_CAP - countHints(request)hits zero. - [2] A system part is marked only when it is the first or the last. A part that already has a hint is skipped, and a spent budget stops the walk. The last mark still covers the prefix through the end of the system block.
- [3] For the default object form, the loop walks backward
strategy.tailmessages. Withtail: 1, that is the newest message that is not an effort update. - [4]
effortUpdateis a system message whose only part has type"effort".effort-updates.tssays a change to top-level reasoning effort invalidates the prompt cache, so OpenCode records it as one of these messages instead of editing the prefix. The counter and the following loop both step over them, and the one tail slot lands on a real conversation message.
OpenCode's Anthropic lowering says the API accepts at most four cache_control breakpoints per request and returns a 400 past that, then drops the extras.
Read it yourself: In packages/ai/src/cache-policy.ts, read applyCachePolicy and then selectPolicy. cacheFormat is checked first. After that, a route missing from RESPECTS_INLINE_HINTS gets NONE, so automatic placement adds nothing. A manual hint already on the request stays. The comment above the set says those providers ignore the tag and cache a prefix without it. Then open packages/ai/src/protocols/anthropic-messages.ts and search for ANTHROPIC_BREAKPOINT_CAP. You should find the counter that drops a marker once four have been written.
Finished this idea?
Your learning progress stays in this browser.