Skip to slide
056 / 170

CH 05 · Prompt Caching and Context Management · 08/15

Read as one page

Real code: pi places the cache breakpoint

Your apply_change_cache_safely never tells the provider where the reusable prefix ends. It only refuses to edit old items. pi's Anthropic adapter, the code that turns one session into a Messages API request, sends an explicit marker. Watch getCacheControl, which builds the marker, and the block at the end of convertMessages, which hangs it on the latest user message.

function getCacheControl(
  model: Model<"anthropic-messages">,
  cacheRetention?: CacheRetention,
  env?: ProviderEnv,
): { retention: CacheRetention; cacheControl?: CacheControlEphemeral } {
  const retention = resolveCacheRetention(cacheRetention, env);
  if (retention === "none") { // [1]
    return { retention };
  }
  const ttl = retention === "long" && getAnthropicCompat(model).supportsLongCacheRetention ? "1h" : undefined; // [2]
  return {
    retention,
    cacheControl: { type: "ephemeral", ...(ttl && { ttl }) }, // [3]
  };
}

Source: earendil-works/pi, packages/ai/src/api/anthropic-messages.ts (MIT)

  // Add cache_control to the last user or system message to cache conversation history
  if (cacheControl && params.length > 0) {
    const lastMessage = params[params.length - 1];
    if (lastMessage.role === "user" || lastMessage.role === "system") { // [4]
      if (Array.isArray(lastMessage.content)) {
        const lastBlock = lastMessage.content[lastMessage.content.length - 1];
        if (
          lastBlock &&
          (lastBlock.type === "text" ||
            lastBlock.type === "image" ||
            lastBlock.type === "tool_result" ||
            lastBlock.type === "tool_addition" ||
            lastBlock.type === "tool_removal")
        ) {
          (lastBlock as any).cache_control = cacheControl; // [5]
        }
      } else if (typeof lastMessage.content === "string") {
        lastMessage.content = [
          {
            type: "text",
            text: lastMessage.content,
            cache_control: cacheControl,
          },
        ] as any;
      }
    }
  }

Source: earendil-works/pi, packages/ai/src/api/anthropic-messages.ts (MIT)

What to notice

  1. [1] CacheRetention is the caller's setting: "none", "short", or "long". resolveCacheRetention returns that value when one was passed. Otherwise only PI_CACHE_RETENTION=long selects "long". Every other value, including an unset variable and none, selects "short". "none" as the argument returns no marker.
  2. [2] ttl is the lifetime written on the marker. It becomes "1h" only for "long" retention, and only when the model supports that. supportsLongCacheRetention defaults to true. "short" leaves ttl unset, so the spread omits the field.
  3. [3] The marker is a cache_control object with type: "ephemeral". That object is the cache breakpoint: the provider may reuse the prefix up to it.
  4. [4] The history breakpoint is written only when the last message is a user or system message. Tool results are pushed as a user message, so a turn that just ran a tool gets the marker. A request that ends on the assistant does not.
  5. [5] The marker sits on the last block of that message, not on every block. On the next turn, if you only appended, the prefix through this block still matches.

buildParams places the same object on the system text, and convertTools places it on the last tool when the model allows a marker there. pi freezes the initial tool list, breakpoint included, only when the model allows mid-conversation system messages and tool changes and the session already has tools. Later tools then arrive as tool_addition blocks. Both flags default to false. If either is off, or the session started with no tools, getCurrentTools sends the current list. Rewriting that list would change the prefix the append just protected.

Read it yourself: Open packages/ai/src/api/anthropic-messages.ts and search for cache_control. Find the system-text write in buildParams, the tools.length - 1 check in convertTools, and nativeToolChanges just above convertMessages. You should see three marker sites, not one per message.

Finished this idea?

Your learning progress stays in this browser.