Chapter 05 · 16 min read · Updated
Prompt Caching and Context Management
Prompt caching reuses an unchanged prefix at a fraction of the cost. Compaction and truncation keep long sessions under the context ceiling.
Prefer one idea per screen? Read it as 15 slides →
In this chapter15
- 05.1Concept explanation
- 05.2Prompt caching: paying once for the stable part
- 05.3What breaks the cache
- 05.4Context management: staying under the ceiling
- 05.5Why it matters
- 05.6How it works: ordering and compacting
- 05.7Code MVP: cache-aware ordering and a compactor
- 05.8Real code: pi places the cache breakpoint
- 05.9Real code: OpenCode marks the stable edges
- 05.10Real code: Codex appends and keeps one cache key
- 05.11Real code: pi will not orphan a tool result
- 05.12Real code: Codex rebuilds history around a summary
- 05.13Read the source yourself
- 05.14Connecting to the bigger picture
- 05.15Key takeaways
05.1
Concept explanation
Chapter 4 left us with a problem: the prompt grows every turn, cost grows quadratically, and eventually the context window fills up. This chapter covers the two families of fixes that every serious harness uses. One attacks the cost curve (prompt caching). The other attacks the window ceiling (compaction and friends). They are different problems with different solutions, so keep them separate in your head.
05.2
Prompt caching: paying once for the stable part
When the model processes a prompt, it builds up an internal state (the key-value cache, or KV cache) for those tokens. Prompt caching means the provider saves that computed state for a prefix of your prompt the first time it sees it, and reuses it on later calls that share the exact same prefix. You skip recomputing the cached part.
The critical rule, and it is unforgiving: cache hits only work for exact prefix matches. The provider can reuse the work only up to the first token that differs. So the structural advice from Chapter 3 pays off here:
- Put static content (instructions, tool definitions, examples) at the front.
- Put variable content (the latest user message, the newest tool result) at the end.
What caches well in an agent session: the system prompt (identical every turn) and the bulk of the conversation history (unchanged from the previous turn). What never caches: the latest user message and the most recent tool result, because they are always new.
The payoff is large. Cached prefix tokens cost roughly a tenth of fresh input tokens. This is why the quadratic growth in tokens sent does not become quadratic growth in cost: most of what you send each turn is a cached prefix you are billed a tenth for. Codex's lead engineer puts it precisely: with cache hits, sampling the model is linear rather than quadratic. That one optimization is the difference between a usable agent and an unaffordable one.
05.3
What breaks the cache
Because hits need exact prefix matches, a handful of innocent-looking actions cause expensive cache misses:
- Changing the
toolslist mid-conversation. - Switching the model (this changes the model-specific instructions near the front of the prompt).
- Changing the sandbox configuration, approval mode, or working directory.
- Rewriting an earlier
AGENTS.mdmessage would invalidate the cached prefix. Codex does not rewrite it.render_diffappends a new message, so the old prefix still matches and only the new message is uncached.
The Codex team learned this the hard way: an early bug enumerated MCP tools in an inconsistent order, which silently broke caching on every turn. The fix for necessary mid-session changes is clever and worth stealing: instead of editing an earlier message (which would change the prefix and bust the cache), append a new message at the end describing the change. For example, if the working directory changes, Codex inserts a fresh <environment_context> message at the tail rather than rewriting the original. The prefix stays intact; the cache survives.
A practical habit falls right out of this: lock your AGENTS.md before a long session. If you do change it, Codex appends the new text instead of rewriting the old message, so the cached prefix still matches. The new message is large, and you pay full price for it the first time.
05.4
Context management: staying under the ceiling
Caching saves money but does not shrink the prompt. To stay under the context window, harnesses use three moves, listed in the Deep Research survey as the standard menu:
- Extend the window. The blunt approach: use a model with a bigger window (Gemini offers up to a million tokens). Simple, but expensive and not always available.
- Compress intermediate steps. Summarize older parts of the conversation so they take fewer tokens.
- Use external storage. Write intermediate results to files or a database and pull them back only when needed (Chapter 9's file-based memory).
The workhorse is move 2, called compaction. When the token count crosses a threshold, the harness replaces the long history with a shorter representative summary, freeing space while keeping the agent's understanding of what happened. Anthropic's harness clears the oldest tool outputs first (those are the bulkiest and least likely to matter later), then summarizes the conversation if needed; your requests and key code snippets are preserved.
Codex started with a manual /compact command and later moved to automatic compaction when an auto_compact_limit is exceeded. The newest Responses API even has a dedicated compaction endpoint that returns a special compaction item carrying an opaque encrypted blob that preserves the model's latent understanding, more faithful than a plain text summary.
Two related techniques round out the toolkit:
- Truncation. Individual tool outputs can be huge (a 5,000-line test log). Rather than dropping them, harnesses keep the head and tail and elide the middle, with a marker like
... (4,800 lines omitted) .... The beginning and end usually carry the signal; the middle is noise. (We build this in Chapter 6.) - Context diffing. Codex tracks a reference context and, when little has changed between turns, avoids re-injecting the full context. Less to send, better cache hit rates.
One honest caveat the sources raise: compaction can lose detail. A summary is lossy by definition, and on a long, intricate task that loss can hurt later reasoning. So compaction is a necessary evil, not a free lunch; it is one more reason to keep sessions focused.
05.5
Why it matters
Caching and compaction are what make agents practical. Without caching, the Chapter 4 cost curve makes any non-trivial session painfully expensive. Without compaction, the agent simply hits a wall mid-task and cannot continue. Together they turn the agent loop from a neat demo into something you can run all day. They are also where a lot of the "why is my agent slow or expensive?" answers live, so understanding them pays off the moment you operate a real agent.
05.6
How it works: ordering and compacting
Two small mechanisms. First, a builder that guarantees the stable prefix never changes so the cache keeps hitting. Second, a compactor that summarizes old turns when the history gets too big, while always preserving the original request and the most recent exchanges.
05.7
Code MVP: cache-aware ordering and a compactor
"""
chapter 05: caching-aware ordering and compaction.
Two tools: keep the prompt prefix stable (so caching keeps hitting), and
compact old history when it grows past a token budget.
Reuses estimate_items_tokens from Chapter 4.
"""
def estimate_tokens(text: str) -> int:
return max(1, len(str(text)) // 4)
def estimate_items_tokens(items: list) -> int:
return sum(estimate_tokens(item.get("content", "")) for item in items)
# --- 1. Keep mid-session changes cache-friendly by APPENDING, not editing ---
def apply_change_cache_safely(history: list, change_description: str) -> list:
"""A config change (new cwd, new sandbox mode) is added as a NEW item at
the end, so the existing prefix is untouched and stays cached."""
history.append({"role": "developer", "type": "message",
"content": f"<context_update>{change_description}</context_update>"})
return history
# --- 2. Compaction: summarize old turns once we cross a token budget ---
class Compactor:
def __init__(self, max_tokens: int = 4000, keep_recent: int = 4):
self.max_tokens = max_tokens # window budget (toy value)
self.keep_recent = keep_recent # always keep the last N items verbatim
def summarize(self, items: list) -> str:
"""Stand-in for a real summarization model call. A production harness
would ask the model to compress these items; here we just sketch it."""
kinds = {}
for it in items:
kinds[it.get("type", "message")] = kinds.get(it.get("type", "message"), 0) + 1
breakdown = ", ".join(f"{n} {k}" for k, n in kinds.items())
return f"[summary of {len(items)} earlier items: {breakdown}]"
def compact(self, history: list) -> list:
if estimate_items_tokens(history) <= self.max_tokens:
return history # under budget: nothing to do
# ALWAYS preserve the very first item (the original user request)...
first = history[:1]
# ...and the most recent items (freshest, least safe to summarize).
recent = history[-self.keep_recent:]
middle = history[1:-self.keep_recent] if len(history) > self.keep_recent + 1 else []
if not middle:
return history
summary_item = {"role": "developer", "type": "message",
"content": self.summarize(middle)}
# Oldest tool outputs are the bulk; replacing them frees the most space.
return first + [summary_item] + recent
if __name__ == "__main__":
history = [{"role": "user", "type": "message", "content": "fix the build"}]
# Simulate a long session: many bulky tool results pile up.
for i in range(30):
history.append({"role": "assistant", "type": "tool_call",
"content": f"run step {i}"})
history.append({"role": "tool", "type": "tool_result",
"content": "X" * 800}) # a big log (~200 tokens each)
print("before:", estimate_items_tokens(history), "tokens,", len(history), "items")
compacted = Compactor(max_tokens=4000, keep_recent=4).compact(history)
print("after :", estimate_items_tokens(compacted), "tokens,", len(compacted), "items")
print("first item preserved:", compacted[0]["content"])
Run it and the history collapses from thousands of tokens to a few hundred, while the original request ("fix the build") and the last few exchanges survive intact. That is compaction in miniature: throw away the bulky middle, keep the bookends.
The slides after this one are those two jobs in production code. You kept a prefix stable and summarized a middle. pi, OpenCode, and Codex have to do that without orphaning a tool result, and without moving the bytes the cache already paid for.
05.8
Real code: pi places the cache breakpoint
Your apply_change_cache_safely never tells the provider where the reusable prefix ends. It only refuses to edit old items. pi's Anthropic adapter, the code that turns one session into a Messages API request, sends an explicit marker. Watch getCacheControl, which builds the marker, and the block at the end of convertMessages, which hangs it on the latest user message.
function getCacheControl(
model: Model<"anthropic-messages">,
cacheRetention?: CacheRetention,
env?: ProviderEnv,
): { retention: CacheRetention; cacheControl?: CacheControlEphemeral } {
const retention = resolveCacheRetention(cacheRetention, env);
if (retention === "none") { // [1]
return { retention };
}
const ttl = retention === "long" && getAnthropicCompat(model).supportsLongCacheRetention ? "1h" : undefined; // [2]
return {
retention,
cacheControl: { type: "ephemeral", ...(ttl && { ttl }) }, // [3]
};
}
Source: earendil-works/pi, packages/ai/src/api/anthropic-messages.ts (MIT)
// Add cache_control to the last user or system message to cache conversation history
if (cacheControl && params.length > 0) {
const lastMessage = params[params.length - 1];
if (lastMessage.role === "user" || lastMessage.role === "system") { // [4]
if (Array.isArray(lastMessage.content)) {
const lastBlock = lastMessage.content[lastMessage.content.length - 1];
if (
lastBlock &&
(lastBlock.type === "text" ||
lastBlock.type === "image" ||
lastBlock.type === "tool_result" ||
lastBlock.type === "tool_addition" ||
lastBlock.type === "tool_removal")
) {
(lastBlock as any).cache_control = cacheControl; // [5]
}
} else if (typeof lastMessage.content === "string") {
lastMessage.content = [
{
type: "text",
text: lastMessage.content,
cache_control: cacheControl,
},
] as any;
}
}
}
Source: earendil-works/pi, packages/ai/src/api/anthropic-messages.ts (MIT)
What to notice
- [1]
CacheRetentionis the caller's setting:"none","short", or"long".resolveCacheRetentionreturns that value when one was passed. Otherwise onlyPI_CACHE_RETENTION=longselects"long". Every other value, including an unset variable andnone, selects"short"."none"as the argument returns no marker. - [2]
ttlis the lifetime written on the marker. It becomes"1h"only for"long"retention, and only when the model supports that.supportsLongCacheRetentiondefaults to true."short"leavesttlunset, so the spread omits the field. - [3] The marker is a
cache_controlobject withtype: "ephemeral". That object is the cache breakpoint: the provider may reuse the prefix up to it. - [4] The history breakpoint is written only when the last message is a user or system message. Tool results are pushed as a user message, so a turn that just ran a tool gets the marker. A request that ends on the assistant does not.
- [5] The marker sits on the last block of that message, not on every block. On the next turn, if you only appended, the prefix through this block still matches.
buildParams places the same object on the system text, and convertTools places it on the last tool when the model allows a marker there. pi freezes the initial tool list, breakpoint included, only when the model allows mid-conversation system messages and tool changes and the session already has tools. Later tools then arrive as tool_addition blocks. Both flags default to false. If either is off, or the session started with no tools, getCurrentTools sends the current list. Rewriting that list would change the prefix the append just protected.
Read it yourself: Open packages/ai/src/api/anthropic-messages.ts and search for cache_control. Find the system-text write in buildParams, the tools.length - 1 check in convertTools, and nativeToolChanges just above convertMessages. You should see three marker sites, not one per message.
05.9
Real code: OpenCode marks the stable edges
pi spends a marker on the system text, the last tool, and the latest user message. OpenCode has a harder budget. applyCachePolicy runs before the provider body is built, and it may attach at most four CacheHint tags. A CacheHint is the in-memory tag that later becomes cache_control on the wire. The default policy, AUTO, asks for the last tool, the system parts, and a tail of one message. Manual tags already on the request count against the four.
const AUTO: CachePolicyObject = {
tools: true,
system: true,
messages: { tail: 1 }, // [1]
}
const NONE: CachePolicyObject = {}
const BREAKPOINT_CAP = 4
Source: anomalyco/opencode, packages/ai/src/cache-policy.ts (MIT)
const markSystemBoundaries = (system: LLMRequest["system"], hint: CacheHint, budget: Budget): LLMRequest["system"] => {
if (system.length === 0) return system
let changed = false
const next = system.map((part, index) => {
if ((index !== 0 && index !== system.length - 1) || part.cache || budget.remaining === 0) return part // [2]
budget.remaining -= 1
changed = true
return { ...part, cache: hint }
})
return changed ? next : system
}
Source: anomalyco/opencode, packages/ai/src/cache-policy.ts (MIT)
const markMessages = (
messages: ReadonlyArray<Message>,
strategy: NonNullable<CachePolicyObject["messages"]>,
hint: CacheHint,
budget: Budget,
): ReadonlyArray<Message> => {
if (messages.length === 0) return messages
if (strategy === "latest-user-message")
return markMessageAt(messages, lastIndexOfRole(messages, "user"), hint, budget)
if (strategy === "latest-assistant")
return markMessageAt(messages, lastIndexOfRole(messages, "assistant"), hint, budget)
let start = messages.length
let remaining = strategy.tail // [3]
while (remaining > 0 && start > 0) {
start -= 1
if (effortUpdate(messages[start]!) === undefined) remaining -= 1 // [4]
}
let next = messages
for (let i = start; i < messages.length; i++)
if (effortUpdate(messages[i]!) === undefined) next = markMessageAt(next, i, hint, budget)
return next
}
Source: anomalyco/opencode, packages/ai/src/cache-policy.ts (MIT)
What to notice
- [1]
tail: 1means one conversation breakpoint, on the newest real message. The tools and the system text take the earlier slots.applyCachePolicymarks in that order: last tool, then system boundaries, then messages, and it stops whenBREAKPOINT_CAP - countHints(request)hits zero. - [2] A system part is marked only when it is the first or the last. A part that already has a hint is skipped, and a spent budget stops the walk. The last mark still covers the prefix through the end of the system block.
- [3] For the default object form, the loop walks backward
strategy.tailmessages. Withtail: 1, that is the newest message that is not an effort update. - [4]
effortUpdateis a system message whose only part has type"effort".effort-updates.tssays a change to top-level reasoning effort invalidates the prompt cache, so OpenCode records it as one of these messages instead of editing the prefix. The counter and the following loop both step over them, and the one tail slot lands on a real conversation message.
OpenCode's Anthropic lowering says the API accepts at most four cache_control breakpoints per request and returns a 400 past that, then drops the extras.
Read it yourself: In packages/ai/src/cache-policy.ts, read applyCachePolicy and then selectPolicy. cacheFormat is checked first. After that, a route missing from RESPECTS_INLINE_HINTS gets NONE, so automatic placement adds nothing. A manual hint already on the request stays. The comment above the set says those providers ignore the tag and cache a prefix without it. Then open packages/ai/src/protocols/anthropic-messages.ts and search for ANTHROPIC_BREAKPOINT_CAP. You should find the counter that drops a marker once four have been written.
05.10
Real code: Codex appends and keeps one cache key
Your helper appends a <context_update> so the old bytes stay put. Codex picks that string in prompt_cache_key. When AGENTS.md or the working directory changes, it records a new message instead of patching the old one.
fn prompt_cache_key(&self, responses_metadata: &CodexResponsesMetadata) -> String {
if let Some(prompt_cache_key) = &self.prompt_cache_key_override {
return prompt_cache_key.clone(); // [1]
}
if let SessionSource::Internal(source) = &self.state.session_source
&& let Some(parent_thread_id) = responses_metadata.parent_thread_id
{
return format!("{source}:{parent_thread_id}"); // [2]
}
responses_metadata.session_id.clone() // [3]
}
Source: openai/codex, codex-rs/core/src/client.rs (Apache-2.0)
if matches!(previous, PreviousSectionState::Known(previous) if previous == ¤t) {
return (None, Vec::new()); // [4]
}
let previous_may_contain_instructions = match previous {
PreviousSectionState::Known(previous) => previous.text.is_some(),
PreviousSectionState::Unknown => true,
PreviousSectionState::Absent => false,
};
let instructions = match (&self.instructions, previous_may_contain_instructions) {
(Some(instructions), true) => UserInstructions {
directory: instructions.directory.clone(),
text: format!("{REPLACEMENT_NOTICE}\n\n{}", instructions.text), // [5]
},
(Some(instructions), false) => instructions.clone(),
(None, true) => UserInstructions {
directory: None,
text: REMOVAL_NOTICE.to_string(),
},
(None, false) => return (Some(current), Vec::new()),
};
(
Some(current),
vec![WorldStateUpdate::fragment(instructions)], // [6]
)
Source: openai/codex, codex-rs/core/src/context/world_state/agents_md.rs (Apache-2.0)
What to notice
- [1] This branch returns
prompt_cache_key_overridewhen one was set. A guardian review session can force that override toguardian:{parent_thread_id}. An ephemeral fork of a root session can force it to the forked session's id. Every other session uses the later branches inprompt_cache_key. - [2] An internal session that has a parent thread uses
{source}:{parent_thread_id}.sourceprints asguardianormemory_consolidation. Internal sessions of that same kind under that parent share this key. It is not the parent'ssession_id. A normal parent session falls through to its own id, with noguardian:prefix. - [3] The remaining case uses this session's id.
build_responses_requestsets the request field fromprompt_cache_key()on every turn. A root session sends this key as the Responses session id. - [4]
render_diffcompares the currentAGENTS.mdtext with the previous snapshot. No change means no new history item, so the prefix is left alone. - [5] A real edit does not rewrite the earlier instructions message. It builds a new
UserInstructionsvalue whose text starts withREPLACEMENT_NOTICE: "These AGENTS.md instructions replace all previously provided AGENTS.md instructions." The old message stays byte for byte, so the cached prefix still matches. - [6]
WorldStateUpdate::fragmentis that new user message. When the diff is non-empty, the history store pushes it onto the end ofself.items.
A working-directory change is appended the same way. In environment.rs, has_same_diff_value returns false when cwd, status, error, the primary flag, or the shell differs, and render_diff emits a new <environment_context> fragment. A date, timezone, network, or filesystem change can emit one even when cwd is unchanged.
Read it yourself: In codex-rs/core/src/client.rs, open build_responses_request and search for let prompt_cache_key = Some(self.prompt_cache_key. The next field in that function is the shorthand prompt_cache_key,. A search for prompt_cache_key: lands on a comparison of two requests, not on this assignment. Then open codex-rs/core/src/session/mod.rs and read record_step_world_state_if_changed. record_conversation_items runs only when the rendered diff is non-empty. Follow that into codex-rs/core/src/context_manager/history.rs and find the push onto self.items.
05.11
Real code: pi will not orphan a tool result
Your Compactor.compact keeps item 0 and the last keep_recent items. Suppose keep_recent is 3 and the history is the user request plus three call/result pairs, seven items. The last three start on a tool result whose call sits in the summarized middle. The model then sees a result with no call. pi will not cut there.
prepareCompaction calls findProjectedCutPoint. A ProjectedSessionEntry is one session entry after context edits have been applied. keepRecentTokens is how many recent tokens stay verbatim.
export const DEFAULT_COMPACTION_SETTINGS: CompactionSettings = {
enabled: true,
reserveTokens: 16384, // [1]
keepRecentTokens: 20000, // [2]
};
Source: earendil-works/pi, packages/coding-agent/src/core/compaction/compaction.ts (MIT)
export function shouldCompact(contextTokens: number, contextWindow: number, settings: CompactionSettings): boolean {
if (!settings.enabled) return false;
return contextTokens > contextWindow - settings.reserveTokens; // [3]
}
Source: earendil-works/pi, packages/coding-agent/src/core/compaction/compaction.ts (MIT)
function isCutPointMessage(message: AgentMessage): boolean {
switch (message.role) {
case "user":
case "assistant":
case "bashExecution":
case "custom":
case "branchSummary":
case "compactionSummary":
return true;
case "toolResult":
return false; // [4]
}
return false;
}
Source: earendil-works/pi, packages/coding-agent/src/core/compaction/compaction.ts (MIT)
let accumulatedTokens = 0;
let exceededBudget = false;
let cutIndex = cutPoints[0];
for (let i = endIndex - 1; i >= startIndex; i--) {
const messageTokens = entries[i].messages.reduce((sum, message) => sum + estimateTokens(message), 0);
if (messageTokens === 0) continue;
accumulatedTokens += messageTokens;
if (accumulatedTokens >= keepRecentTokens) {
exceededBudget = true;
cutIndex = cutPoints.find((candidate) => candidate >= i) ?? cutPoints[cutPoints.length - 1]; // [5]
break;
}
}
Source: earendil-works/pi, packages/coding-agent/src/core/compaction/compaction.ts (MIT)
The summary itself is a real model call, not a count of item types. This is the instruction, with the middle sections elided:
const SUMMARIZATION_PROMPT = `The messages above are a conversation to summarize. Create a structured context checkpoint summary that another LLM will use to continue the work.
Use this EXACT format:
## Goal
[What is the user trying to accomplish? Can be multiple items if the session covers different tasks.]
## Constraints & Preferences
- [Any constraints, preferences, or requirements mentioned by user]
- [Or "(none)" if none were mentioned]
// ... (Progress and Key Decisions)
## Next Steps
1. [Ordered list of what should happen next]
## Critical Context
- [Any data, examples, or references needed to continue]
// ...
Keep each section concise. Preserve exact file paths, function names, and error messages.`;
Source: earendil-works/pi, packages/coding-agent/src/core/compaction/compaction.ts (MIT)
What to notice
- [1]
reserveTokensis headroom, 16384 by default. Compaction starts when the context is inside that reserve of the window, so the model still has room to answer. Your toy waits until a fixedmax_tokensand has no separate reserve. - [2]
keepRecentTokensis 20000 by default, kept verbatim. Your toy keeps a count of items, so four huge logs can still blow the budget. - [3] The check is
contextTokens > contextWindow - reserveTokens. Disabled settings skip it. The trigger does not itself delete anything. - [4] A
toolResultis not a legal cut, and neither is asystemmessage, which falls through to the finalreturn false. User, assistant,bashExecution,custom,branchSummary, andcompactionSummaryare legal cuts. The tool result stays with the assistant entry that issued the call, on whichever side of the cut that entry lands. - [5] The walk goes backward from the newest entry. When the estimate crosses
keepRecentTokensat indexi, the cut is the nearest legal point at or afteri. A trailing tool result has no later legal point, so??uses the last one: the assistant message in front of those results. Call and result stay together.
getSummarizationFailure drops a summary whose stopReason is "length". That text is partial, so it must not become the checkpoint. Your summarize never calls a model, so it cannot fail this way.
Read it yourself: In packages/coding-agent/src/core/compaction/compaction.ts, read the rest of SUMMARIZATION_PROMPT, then getSummarizationFailure. Then read prepareCompaction and see what isSplitTurn does. The turn's prefix is summarized on its own, and the messages from the cut onward stay verbatim.
05.12
Real code: Codex rebuilds history around a summary
pi keeps a recent tail verbatim and summarizes what fell off. Codex's local compaction is harsher, and it is closer to a handoff than to your bookends. The compaction turn sends a prompt, takes the assistant's reply as the summary, and builds a new history. Recent tool calls are not copied. They have to survive inside the summary.
SUMMARIZATION_PROMPT is the default user text. config.compact_prompt replaces it when set. The reply is stored as summary_suffix, then summary_text is SUMMARY_PREFIX, a newline, and that reply.
You are performing a CONTEXT CHECKPOINT COMPACTION. Create a handoff summary for another LLM that will resume the task.
Include:
- Current progress and key decisions made
- Important context, constraints, or user preferences
- What remains to be done (clear next steps)
- Any critical data, examples, or references needed to continue
Be concise, structured, and focused on helping the next LLM seamlessly continue the work.
Source: openai/codex, codex-rs/prompts/templates/compact/prompt.md (Apache-2.0)
fn build_compacted_history_with_limit(
mut history: Vec<ResponseItemEnvelope>,
user_messages: &[CompactedUserMessage<'_>],
summary_text: &str,
max_tokens: usize,
) -> Vec<ResponseItemEnvelope> {
let mut selected_messages = Vec::new();
if max_tokens > 0 {
let mut remaining = max_tokens;
for message in user_messages.iter().rev() { // [1]
if remaining == 0 {
break; // [2]
}
let tokens = approx_token_count(&message.message);
Source: openai/codex, codex-rs/core/src/compact.rs (Apache-2.0)
selected_messages.push(ResponseItemEnvelope {
item,
metadata: harness_metadata,
});
if tokens > remaining {
break;
}
remaining = remaining.saturating_sub(tokens);
}
selected_messages.reverse(); // [3]
}
history.extend(selected_messages);
Source: openai/codex, codex-rs/core/src/compact.rs (Apache-2.0)
history.push(ResponseItemEnvelope::new(ContextualUserFragment::into(
CompactionSummary::new(summary_text), // [4]
)));
history
}
Source: openai/codex, codex-rs/core/src/compact.rs (Apache-2.0)
What to notice
- [1]
user_messagescomes fromcollect_annotated_user_messages. That helper keeps real user messages and drops anything that already starts withSUMMARY_PREFIX, so an old summary is not kept as if the user had typed it. The loop walks newest user message first. Assistant turns and tool results are not in this list. - [2]
max_tokenshere isCOMPACT_USER_MESSAGE_MAX_TOKENS, 20000. Thisbreakfires when that budget is already zero, so an older user message is not taken. Between these two excerpts, a message that fits and is text only is copied, and any other message keeps a truncated text prefix. A message that costs more than what is left is still pushed, and the laterbreakstops the walk. - [3]
reverseputs the kept user messages back into time order. They were collected newest first. - [4]
CompactionSummaryis a user-role fragment, pushed last by this function. The caller passed an empty initial context intobuild_compacted_history.build_compaction_replacement_historythen rebuilds the full initial context: every world-state section plus the developer instructions. Prefix items go at the front. The rest follow the four rules ininsert_initial_context_before_last_real_user_or_summary. The first choice is the slot before the last real user message, agent message, or retained tool output, so the summary stays last.
Your summarize invents a one-line count of item types. This prompt asks for progress, constraints, next steps, and the data the next model needs.
Read it yourself: Open codex-rs/prompts/templates/compact/summary_prefix.md and compare it with SUMMARY_PREFIX in codex-rs/core/src/compact.rs, on the line that formats summary_text. Then read insert_initial_context_before_last_real_user_or_summary and check the four placement rules in its comment. In codex-rs/core/src/tasks/compact.rs, see which function runs when the provider's remote compaction support is V2 instead of this local prompt.
05.13
Read the source yourself
Read these in order. Each link is the commit this chapter quotes.
- pi's marker sites. packages/ai/src/api/anthropic-messages.ts. Search for
cache_control. Count how many request shapes get a marker, and which message role is refused one. - OpenCode's budget.
packages/ai/src/cache-policy.ts. ReadselectPolicy. After thecacheFormatchecks, a route missing fromRESPECTS_INLINE_HINTSgetsNONE, so automatic placement adds nothing. A manual hint already on the request stays. The comment above the set says those providers ignore the tag. - Codex appending a change. environment.rs. Find
has_same_diff_valueand the<cwd>write insideRenderedEnvironments. Confirm a directory change cannot edit an older<environment_context>message, because the diff only builds a new fragment. - pi's cut and the failed summary. compaction.ts. Read
findProjectedCutPointthroughisSplitTurn, thengetSummarizationFailure. Note whichstopReasonvalues must not be saved. - Codex's rebuilt history. compact.rs and prompt.md. Trace one user message from
collect_annotated_user_messagesto thepushofCompactionSummary.
Exercise: Open Gemini CLI's chatCompressionService.ts (Apache-2.0). In findCompressSplitPoint, an entry is a split point only when its role is user and it has no functionResponse. Model entries are never split points. Separately, the function will not return contents.length (compress everything) when the last entry is a model turn that still has a functionCall. Then find the status returned when newTokenCount > originalTokenCount. That comparison is the rebuilt history, which is the summary plus a short acknowledgement plus the kept tail, against the original history. The status is COMPRESSION_FAILED_INFLATED_TOKEN_COUNT. pi's cut and this split share one rule: do not separate a call from its result, and do not keep a summary that failed to shrink the history.
05.14
Connecting to the bigger picture
Chapter 4 measured the problem; this chapter solves it. The apply_change_cache_safely pattern protects the prefix that Chapter 3's PromptBuilder carefully put first. The Compactor keeps the loop from Chapter 2 from ever hitting the window ceiling. Chapter 9's file-based memory is the third context-management move (external storage). And Chapter 11's subagents are arguably the best context tool of all: a subagent does noisy work in its own window and returns a tiny summary, so the main context never sees the mess.
05.15
Key takeaways
- Prompt caching reuses computed state for an exact prompt prefix, at roughly a tenth the cost. It turns the quadratic cost curve back into a linear one.
- Cache hits need exact prefix matches, so put stable content first and variable content last, and never edit earlier messages mid-session; append new ones instead.
- Common cache-busters: changing tools, switching models, or changing the sandbox. A working-directory change and an
AGENTS.mdedit are appended as new messages, so the old prefix still matches and only the new message is uncached. Lock instructions before long sessions. - Compaction keeps you under the context window by summarizing old turns (oldest tool outputs first), while preserving the original request and recent exchanges.
- Truncation (head plus tail) tames giant tool outputs, and context diffing avoids re-sending unchanged context. Compaction is lossy, so keep sessions focused.
Original sources: OpenAI's Unrolling the Codex agent loop, Anthropic's How Claude Code works, and the Deep Research Agents survey (arXiv:2506.18096).
0 of 15 slides completed
Your learning progress stays in this browser.
Self-check · 5 questions
01 / 05
Why does prompt caching turn a quadratic cost curve back into a linear one?