Skip to slide
057 / 170

CH 05 · Prompt Caching and Context Management · 09/15

Read as one page

Real code: OpenCode marks the stable edges

pi spends a marker on the system text, the last tool, and the latest user message. OpenCode has a harder budget. applyCachePolicy runs before the provider body is built, and it may attach at most four CacheHint tags. A CacheHint is the in-memory tag that later becomes cache_control on the wire. The default policy, AUTO, asks for the last tool, the system parts, and a tail of one message. Manual tags already on the request count against the four.

const AUTO: CachePolicyObject = {
  tools: true,
  system: true,
  messages: { tail: 1 }, // [1]
}

const NONE: CachePolicyObject = {}
const BREAKPOINT_CAP = 4

Source: anomalyco/opencode, packages/ai/src/cache-policy.ts (MIT)

const markSystemBoundaries = (system: LLMRequest["system"], hint: CacheHint, budget: Budget): LLMRequest["system"] => {
  if (system.length === 0) return system
  let changed = false
  const next = system.map((part, index) => {
    if ((index !== 0 && index !== system.length - 1) || part.cache || budget.remaining === 0) return part // [2]
    budget.remaining -= 1
    changed = true
    return { ...part, cache: hint }
  })
  return changed ? next : system
}

Source: anomalyco/opencode, packages/ai/src/cache-policy.ts (MIT)

const markMessages = (
  messages: ReadonlyArray<Message>,
  strategy: NonNullable<CachePolicyObject["messages"]>,
  hint: CacheHint,
  budget: Budget,
): ReadonlyArray<Message> => {
  if (messages.length === 0) return messages
  if (strategy === "latest-user-message")
    return markMessageAt(messages, lastIndexOfRole(messages, "user"), hint, budget)
  if (strategy === "latest-assistant")
    return markMessageAt(messages, lastIndexOfRole(messages, "assistant"), hint, budget)
  let start = messages.length
  let remaining = strategy.tail // [3]
  while (remaining > 0 && start > 0) {
    start -= 1
    if (effortUpdate(messages[start]!) === undefined) remaining -= 1 // [4]
  }
  let next = messages
  for (let i = start; i < messages.length; i++)
    if (effortUpdate(messages[i]!) === undefined) next = markMessageAt(next, i, hint, budget)
  return next
}

Source: anomalyco/opencode, packages/ai/src/cache-policy.ts (MIT)

What to notice

  1. [1] tail: 1 means one conversation breakpoint, on the newest real message. The tools and the system text take the earlier slots. applyCachePolicy marks in that order: last tool, then system boundaries, then messages, and it stops when BREAKPOINT_CAP - countHints(request) hits zero.
  2. [2] A system part is marked only when it is the first or the last. A part that already has a hint is skipped, and a spent budget stops the walk. The last mark still covers the prefix through the end of the system block.
  3. [3] For the default object form, the loop walks backward strategy.tail messages. With tail: 1, that is the newest message that is not an effort update.
  4. [4] effortUpdate is a system message whose only part has type "effort". effort-updates.ts says a change to top-level reasoning effort invalidates the prompt cache, so OpenCode records it as one of these messages instead of editing the prefix. The counter and the following loop both step over them, and the one tail slot lands on a real conversation message.

OpenCode's Anthropic lowering says the API accepts at most four cache_control breakpoints per request and returns a 400 past that, then drops the extras.

Read it yourself: In packages/ai/src/cache-policy.ts, read applyCachePolicy and then selectPolicy. cacheFormat is checked first. After that, a route missing from RESPECTS_INLINE_HINTS gets NONE, so automatic placement adds nothing. A manual hint already on the request stays. The comment above the set says those providers ignore the tag and cache a prefix without it. Then open packages/ai/src/protocols/anthropic-messages.ts and search for ANTHROPIC_BREAKPOINT_CAP. You should find the counter that drops a marker once four have been written.

Finished this idea?

Your learning progress stays in this browser.