CH 05 · Prompt Caching and Context Management · 08/15
Read as one pageReal code: pi places the cache breakpoint
Your apply_change_cache_safely never tells the provider where the reusable prefix ends. It only refuses to edit old items. pi's Anthropic adapter, the code that turns one session into a Messages API request, sends an explicit marker. Watch getCacheControl, which builds the marker, and the block at the end of convertMessages, which hangs it on the latest user message.
function getCacheControl(
model: Model<"anthropic-messages">,
cacheRetention?: CacheRetention,
env?: ProviderEnv,
): { retention: CacheRetention; cacheControl?: CacheControlEphemeral } {
const retention = resolveCacheRetention(cacheRetention, env);
if (retention === "none") { // [1]
return { retention };
}
const ttl = retention === "long" && getAnthropicCompat(model).supportsLongCacheRetention ? "1h" : undefined; // [2]
return {
retention,
cacheControl: { type: "ephemeral", ...(ttl && { ttl }) }, // [3]
};
}
Source: earendil-works/pi, packages/ai/src/api/anthropic-messages.ts (MIT)
// Add cache_control to the last user or system message to cache conversation history
if (cacheControl && params.length > 0) {
const lastMessage = params[params.length - 1];
if (lastMessage.role === "user" || lastMessage.role === "system") { // [4]
if (Array.isArray(lastMessage.content)) {
const lastBlock = lastMessage.content[lastMessage.content.length - 1];
if (
lastBlock &&
(lastBlock.type === "text" ||
lastBlock.type === "image" ||
lastBlock.type === "tool_result" ||
lastBlock.type === "tool_addition" ||
lastBlock.type === "tool_removal")
) {
(lastBlock as any).cache_control = cacheControl; // [5]
}
} else if (typeof lastMessage.content === "string") {
lastMessage.content = [
{
type: "text",
text: lastMessage.content,
cache_control: cacheControl,
},
] as any;
}
}
}
Source: earendil-works/pi, packages/ai/src/api/anthropic-messages.ts (MIT)
What to notice
- [1]
CacheRetentionis the caller's setting:"none","short", or"long".resolveCacheRetentionreturns that value when one was passed. Otherwise onlyPI_CACHE_RETENTION=longselects"long". Every other value, including an unset variable andnone, selects"short"."none"as the argument returns no marker. - [2]
ttlis the lifetime written on the marker. It becomes"1h"only for"long"retention, and only when the model supports that.supportsLongCacheRetentiondefaults to true."short"leavesttlunset, so the spread omits the field. - [3] The marker is a
cache_controlobject withtype: "ephemeral". That object is the cache breakpoint: the provider may reuse the prefix up to it. - [4] The history breakpoint is written only when the last message is a user or system message. Tool results are pushed as a user message, so a turn that just ran a tool gets the marker. A request that ends on the assistant does not.
- [5] The marker sits on the last block of that message, not on every block. On the next turn, if you only appended, the prefix through this block still matches.
buildParams places the same object on the system text, and convertTools places it on the last tool when the model allows a marker there. pi freezes the initial tool list, breakpoint included, only when the model allows mid-conversation system messages and tool changes and the session already has tools. Later tools then arrive as tool_addition blocks. Both flags default to false. If either is off, or the session started with no tools, getCurrentTools sends the current list. Rewriting that list would change the prefix the append just protected.
Read it yourself: Open packages/ai/src/api/anthropic-messages.ts and search for cache_control. Find the system-text write in buildParams, the tools.length - 1 check in convertTools, and nativeToolChanges just above convertMessages. You should see three marker sites, not one per message.
Finished this idea?
Your learning progress stays in this browser.