Skip to content

05.1

Concept explanation

Chapter 4 left us with a problem: the prompt grows every turn, cost grows quadratically, and eventually the context window fills up. This chapter covers the two families of fixes that every serious harness uses. One attacks the cost curve (prompt caching). The other attacks the window ceiling (compaction and friends). They are different problems with different solutions, so keep them separate in your head.

05.2

Prompt caching: paying once for the stable part

When the model processes a prompt, it builds up an internal state (the key-value cache, or KV cache) for those tokens. Prompt caching means the provider saves that computed state for a prefix of your prompt the first time it sees it, and reuses it on later calls that share the exact same prefix. You skip recomputing the cached part.

The critical rule, and it is unforgiving: cache hits only work for exact prefix matches. The provider can reuse the work only up to the first token that differs. So the structural advice from Chapter 3 pays off here:

  • Put static content (instructions, tool definitions, examples) at the front.
  • Put variable content (the latest user message, the newest tool result) at the end.

What caches well in an agent session: the system prompt (identical every turn) and the bulk of the conversation history (unchanged from the previous turn). What never caches: the latest user message and the most recent tool result, because they are always new.

The payoff is large. Cached prefix tokens cost roughly a tenth of fresh input tokens. This is why the quadratic growth in tokens sent does not become quadratic growth in cost: most of what you send each turn is a cached prefix you are billed a tenth for. Codex's lead engineer puts it precisely: with cache hits, sampling the model is linear rather than quadratic. That one optimization is the difference between a usable agent and an unaffordable one.

05.3

What breaks the cache

Because hits need exact prefix matches, a handful of innocent-looking actions cause expensive cache misses:

  • Changing the tools list mid-conversation.
  • Switching the model (this changes the model-specific instructions near the front of the prompt).
  • Changing the sandbox configuration, approval mode, or working directory.
  • Rewriting an earlier AGENTS.md message would invalidate the cached prefix. Codex does not rewrite it. render_diff appends a new message, so the old prefix still matches and only the new message is uncached.

The Codex team learned this the hard way: an early bug enumerated MCP tools in an inconsistent order, which silently broke caching on every turn. The fix for necessary mid-session changes is clever and worth stealing: instead of editing an earlier message (which would change the prefix and bust the cache), append a new message at the end describing the change. For example, if the working directory changes, Codex inserts a fresh <environment_context> message at the tail rather than rewriting the original. The prefix stays intact; the cache survives.

A practical habit falls right out of this: lock your AGENTS.md before a long session. If you do change it, Codex appends the new text instead of rewriting the old message, so the cached prefix still matches. The new message is large, and you pay full price for it the first time.

Prompt caching is a prefix match
Prompt caching is a prefix match

05.4

Context management: staying under the ceiling

Caching saves money but does not shrink the prompt. To stay under the context window, harnesses use three moves, listed in the Deep Research survey as the standard menu:

  1. Extend the window. The blunt approach: use a model with a bigger window (Gemini offers up to a million tokens). Simple, but expensive and not always available.
  2. Compress intermediate steps. Summarize older parts of the conversation so they take fewer tokens.
  3. Use external storage. Write intermediate results to files or a database and pull them back only when needed (Chapter 9's file-based memory).

The workhorse is move 2, called compaction. When the token count crosses a threshold, the harness replaces the long history with a shorter representative summary, freeing space while keeping the agent's understanding of what happened. Anthropic's harness clears the oldest tool outputs first (those are the bulkiest and least likely to matter later), then summarizes the conversation if needed; your requests and key code snippets are preserved.

Codex started with a manual /compact command and later moved to automatic compaction when an auto_compact_limit is exceeded. The newest Responses API even has a dedicated compaction endpoint that returns a special compaction item carrying an opaque encrypted blob that preserves the model's latent understanding, more faithful than a plain text summary.

Two related techniques round out the toolkit:

  • Truncation. Individual tool outputs can be huge (a 5,000-line test log). Rather than dropping them, harnesses keep the head and tail and elide the middle, with a marker like ... (4,800 lines omitted) .... The beginning and end usually carry the signal; the middle is noise. (We build this in Chapter 6.)
  • Context diffing. Codex tracks a reference context and, when little has changed between turns, avoids re-injecting the full context. Less to send, better cache hit rates.

One honest caveat the sources raise: compaction can lose detail. A summary is lossy by definition, and on a long, intricate task that loss can hurt later reasoning. So compaction is a necessary evil, not a free lunch; it is one more reason to keep sessions focused.

05.5

Why it matters

Caching and compaction are what make agents practical. Without caching, the Chapter 4 cost curve makes any non-trivial session painfully expensive. Without compaction, the agent simply hits a wall mid-task and cannot continue. Together they turn the agent loop from a neat demo into something you can run all day. They are also where a lot of the "why is my agent slow or expensive?" answers live, so understanding them pays off the moment you operate a real agent.

05.6

How it works: ordering and compacting

Two small mechanisms. First, a builder that guarantees the stable prefix never changes so the cache keeps hitting. Second, a compactor that summarizes old turns when the history gets too big, while always preserving the original request and the most recent exchanges.

05.7

Code MVP: cache-aware ordering and a compactor

"""
chapter 05: caching-aware ordering and compaction.
Two tools: keep the prompt prefix stable (so caching keeps hitting), and
compact old history when it grows past a token budget.
Reuses estimate_items_tokens from Chapter 4.
"""

def estimate_tokens(text: str) -> int:
    return max(1, len(str(text)) // 4)

def estimate_items_tokens(items: list) -> int:
    return sum(estimate_tokens(item.get("content", "")) for item in items)

# --- 1. Keep mid-session changes cache-friendly by APPENDING, not editing ---
def apply_change_cache_safely(history: list, change_description: str) -> list:
    """A config change (new cwd, new sandbox mode) is added as a NEW item at
    the end, so the existing prefix is untouched and stays cached."""
    history.append({"role": "developer", "type": "message",
                    "content": f"<context_update>{change_description}</context_update>"})
    return history

# --- 2. Compaction: summarize old turns once we cross a token budget ---
class Compactor:
    def __init__(self, max_tokens: int = 4000, keep_recent: int = 4):
        self.max_tokens = max_tokens      # window budget (toy value)
        self.keep_recent = keep_recent    # always keep the last N items verbatim

    def summarize(self, items: list) -> str:
        """Stand-in for a real summarization model call. A production harness
        would ask the model to compress these items; here we just sketch it."""
        kinds = {}
        for it in items:
            kinds[it.get("type", "message")] = kinds.get(it.get("type", "message"), 0) + 1
        breakdown = ", ".join(f"{n} {k}" for k, n in kinds.items())
        return f"[summary of {len(items)} earlier items: {breakdown}]"

    def compact(self, history: list) -> list:
        if estimate_items_tokens(history) <= self.max_tokens:
            return history                       # under budget: nothing to do

        # ALWAYS preserve the very first item (the original user request)...
        first = history[:1]
        # ...and the most recent items (freshest, least safe to summarize).
        recent = history[-self.keep_recent:]
        middle = history[1:-self.keep_recent] if len(history) > self.keep_recent + 1 else []

        if not middle:
            return history
        summary_item = {"role": "developer", "type": "message",
                        "content": self.summarize(middle)}
        # Oldest tool outputs are the bulk; replacing them frees the most space.
        return first + [summary_item] + recent

if __name__ == "__main__":
    history = [{"role": "user", "type": "message", "content": "fix the build"}]
    # Simulate a long session: many bulky tool results pile up.
    for i in range(30):
        history.append({"role": "assistant", "type": "tool_call",
                        "content": f"run step {i}"})
        history.append({"role": "tool", "type": "tool_result",
                        "content": "X" * 800})  # a big log (~200 tokens each)

    print("before:", estimate_items_tokens(history), "tokens,", len(history), "items")
    compacted = Compactor(max_tokens=4000, keep_recent=4).compact(history)
    print("after :", estimate_items_tokens(compacted), "tokens,", len(compacted), "items")
    print("first item preserved:", compacted[0]["content"])

Run it and the history collapses from thousands of tokens to a few hundred, while the original request ("fix the build") and the last few exchanges survive intact. That is compaction in miniature: throw away the bulky middle, keep the bookends.

The slides after this one are those two jobs in production code. You kept a prefix stable and summarized a middle. pi, OpenCode, and Codex have to do that without orphaning a tool result, and without moving the bytes the cache already paid for.

05.8

Real code: pi places the cache breakpoint

Your apply_change_cache_safely never tells the provider where the reusable prefix ends. It only refuses to edit old items. pi's Anthropic adapter, the code that turns one session into a Messages API request, sends an explicit marker. Watch getCacheControl, which builds the marker, and the block at the end of convertMessages, which hangs it on the latest user message.

function getCacheControl(
  model: Model<"anthropic-messages">,
  cacheRetention?: CacheRetention,
  env?: ProviderEnv,
): { retention: CacheRetention; cacheControl?: CacheControlEphemeral } {
  const retention = resolveCacheRetention(cacheRetention, env);
  if (retention === "none") { // [1]
    return { retention };
  }
  const ttl = retention === "long" && getAnthropicCompat(model).supportsLongCacheRetention ? "1h" : undefined; // [2]
  return {
    retention,
    cacheControl: { type: "ephemeral", ...(ttl && { ttl }) }, // [3]
  };
}

Source: earendil-works/pi, packages/ai/src/api/anthropic-messages.ts (MIT)

  // Add cache_control to the last user or system message to cache conversation history
  if (cacheControl && params.length > 0) {
    const lastMessage = params[params.length - 1];
    if (lastMessage.role === "user" || lastMessage.role === "system") { // [4]
      if (Array.isArray(lastMessage.content)) {
        const lastBlock = lastMessage.content[lastMessage.content.length - 1];
        if (
          lastBlock &&
          (lastBlock.type === "text" ||
            lastBlock.type === "image" ||
            lastBlock.type === "tool_result" ||
            lastBlock.type === "tool_addition" ||
            lastBlock.type === "tool_removal")
        ) {
          (lastBlock as any).cache_control = cacheControl; // [5]
        }
      } else if (typeof lastMessage.content === "string") {
        lastMessage.content = [
          {
            type: "text",
            text: lastMessage.content,
            cache_control: cacheControl,
          },
        ] as any;
      }
    }
  }

Source: earendil-works/pi, packages/ai/src/api/anthropic-messages.ts (MIT)

What to notice

  1. [1] CacheRetention is the caller's setting: "none", "short", or "long". resolveCacheRetention returns that value when one was passed. Otherwise only PI_CACHE_RETENTION=long selects "long". Every other value, including an unset variable and none, selects "short". "none" as the argument returns no marker.
  2. [2] ttl is the lifetime written on the marker. It becomes "1h" only for "long" retention, and only when the model supports that. supportsLongCacheRetention defaults to true. "short" leaves ttl unset, so the spread omits the field.
  3. [3] The marker is a cache_control object with type: "ephemeral". That object is the cache breakpoint: the provider may reuse the prefix up to it.
  4. [4] The history breakpoint is written only when the last message is a user or system message. Tool results are pushed as a user message, so a turn that just ran a tool gets the marker. A request that ends on the assistant does not.
  5. [5] The marker sits on the last block of that message, not on every block. On the next turn, if you only appended, the prefix through this block still matches.

buildParams places the same object on the system text, and convertTools places it on the last tool when the model allows a marker there. pi freezes the initial tool list, breakpoint included, only when the model allows mid-conversation system messages and tool changes and the session already has tools. Later tools then arrive as tool_addition blocks. Both flags default to false. If either is off, or the session started with no tools, getCurrentTools sends the current list. Rewriting that list would change the prefix the append just protected.

Read it yourself: Open packages/ai/src/api/anthropic-messages.ts and search for cache_control. Find the system-text write in buildParams, the tools.length - 1 check in convertTools, and nativeToolChanges just above convertMessages. You should see three marker sites, not one per message.

05.9

Real code: OpenCode marks the stable edges

pi spends a marker on the system text, the last tool, and the latest user message. OpenCode has a harder budget. applyCachePolicy runs before the provider body is built, and it may attach at most four CacheHint tags. A CacheHint is the in-memory tag that later becomes cache_control on the wire. The default policy, AUTO, asks for the last tool, the system parts, and a tail of one message. Manual tags already on the request count against the four.

const AUTO: CachePolicyObject = {
  tools: true,
  system: true,
  messages: { tail: 1 }, // [1]
}

const NONE: CachePolicyObject = {}
const BREAKPOINT_CAP = 4

Source: anomalyco/opencode, packages/ai/src/cache-policy.ts (MIT)

const markSystemBoundaries = (system: LLMRequest["system"], hint: CacheHint, budget: Budget): LLMRequest["system"] => {
  if (system.length === 0) return system
  let changed = false
  const next = system.map((part, index) => {
    if ((index !== 0 && index !== system.length - 1) || part.cache || budget.remaining === 0) return part // [2]
    budget.remaining -= 1
    changed = true
    return { ...part, cache: hint }
  })
  return changed ? next : system
}

Source: anomalyco/opencode, packages/ai/src/cache-policy.ts (MIT)

const markMessages = (
  messages: ReadonlyArray<Message>,
  strategy: NonNullable<CachePolicyObject["messages"]>,
  hint: CacheHint,
  budget: Budget,
): ReadonlyArray<Message> => {
  if (messages.length === 0) return messages
  if (strategy === "latest-user-message")
    return markMessageAt(messages, lastIndexOfRole(messages, "user"), hint, budget)
  if (strategy === "latest-assistant")
    return markMessageAt(messages, lastIndexOfRole(messages, "assistant"), hint, budget)
  let start = messages.length
  let remaining = strategy.tail // [3]
  while (remaining > 0 && start > 0) {
    start -= 1
    if (effortUpdate(messages[start]!) === undefined) remaining -= 1 // [4]
  }
  let next = messages
  for (let i = start; i < messages.length; i++)
    if (effortUpdate(messages[i]!) === undefined) next = markMessageAt(next, i, hint, budget)
  return next
}

Source: anomalyco/opencode, packages/ai/src/cache-policy.ts (MIT)

What to notice

  1. [1] tail: 1 means one conversation breakpoint, on the newest real message. The tools and the system text take the earlier slots. applyCachePolicy marks in that order: last tool, then system boundaries, then messages, and it stops when BREAKPOINT_CAP - countHints(request) hits zero.
  2. [2] A system part is marked only when it is the first or the last. A part that already has a hint is skipped, and a spent budget stops the walk. The last mark still covers the prefix through the end of the system block.
  3. [3] For the default object form, the loop walks backward strategy.tail messages. With tail: 1, that is the newest message that is not an effort update.
  4. [4] effortUpdate is a system message whose only part has type "effort". effort-updates.ts says a change to top-level reasoning effort invalidates the prompt cache, so OpenCode records it as one of these messages instead of editing the prefix. The counter and the following loop both step over them, and the one tail slot lands on a real conversation message.

OpenCode's Anthropic lowering says the API accepts at most four cache_control breakpoints per request and returns a 400 past that, then drops the extras.

Read it yourself: In packages/ai/src/cache-policy.ts, read applyCachePolicy and then selectPolicy. cacheFormat is checked first. After that, a route missing from RESPECTS_INLINE_HINTS gets NONE, so automatic placement adds nothing. A manual hint already on the request stays. The comment above the set says those providers ignore the tag and cache a prefix without it. Then open packages/ai/src/protocols/anthropic-messages.ts and search for ANTHROPIC_BREAKPOINT_CAP. You should find the counter that drops a marker once four have been written.

05.10

Real code: Codex appends and keeps one cache key

Your helper appends a <context_update> so the old bytes stay put. Codex picks that string in prompt_cache_key. When AGENTS.md or the working directory changes, it records a new message instead of patching the old one.

    fn prompt_cache_key(&self, responses_metadata: &CodexResponsesMetadata) -> String {
        if let Some(prompt_cache_key) = &self.prompt_cache_key_override {
            return prompt_cache_key.clone(); // [1]
        }

        if let SessionSource::Internal(source) = &self.state.session_source
            && let Some(parent_thread_id) = responses_metadata.parent_thread_id
        {
            return format!("{source}:{parent_thread_id}"); // [2]
        }

        responses_metadata.session_id.clone() // [3]
    }

Source: openai/codex, codex-rs/core/src/client.rs (Apache-2.0)

        if matches!(previous, PreviousSectionState::Known(previous) if previous == &current) {
            return (None, Vec::new()); // [4]
        }

        let previous_may_contain_instructions = match previous {
            PreviousSectionState::Known(previous) => previous.text.is_some(),
            PreviousSectionState::Unknown => true,
            PreviousSectionState::Absent => false,
        };
        let instructions = match (&self.instructions, previous_may_contain_instructions) {
            (Some(instructions), true) => UserInstructions {
                directory: instructions.directory.clone(),
                text: format!("{REPLACEMENT_NOTICE}\n\n{}", instructions.text), // [5]
            },
            (Some(instructions), false) => instructions.clone(),
            (None, true) => UserInstructions {
                directory: None,
                text: REMOVAL_NOTICE.to_string(),
            },
            (None, false) => return (Some(current), Vec::new()),
        };
        (
            Some(current),
            vec![WorldStateUpdate::fragment(instructions)], // [6]
        )

Source: openai/codex, codex-rs/core/src/context/world_state/agents_md.rs (Apache-2.0)

What to notice

  1. [1] This branch returns prompt_cache_key_override when one was set. A guardian review session can force that override to guardian:{parent_thread_id}. An ephemeral fork of a root session can force it to the forked session's id. Every other session uses the later branches in prompt_cache_key.
  2. [2] An internal session that has a parent thread uses {source}:{parent_thread_id}. source prints as guardian or memory_consolidation. Internal sessions of that same kind under that parent share this key. It is not the parent's session_id. A normal parent session falls through to its own id, with no guardian: prefix.
  3. [3] The remaining case uses this session's id. build_responses_request sets the request field from prompt_cache_key() on every turn. A root session sends this key as the Responses session id.
  4. [4] render_diff compares the current AGENTS.md text with the previous snapshot. No change means no new history item, so the prefix is left alone.
  5. [5] A real edit does not rewrite the earlier instructions message. It builds a new UserInstructions value whose text starts with REPLACEMENT_NOTICE: "These AGENTS.md instructions replace all previously provided AGENTS.md instructions." The old message stays byte for byte, so the cached prefix still matches.
  6. [6] WorldStateUpdate::fragment is that new user message. When the diff is non-empty, the history store pushes it onto the end of self.items.

A working-directory change is appended the same way. In environment.rs, has_same_diff_value returns false when cwd, status, error, the primary flag, or the shell differs, and render_diff emits a new <environment_context> fragment. A date, timezone, network, or filesystem change can emit one even when cwd is unchanged.

Read it yourself: In codex-rs/core/src/client.rs, open build_responses_request and search for let prompt_cache_key = Some(self.prompt_cache_key. The next field in that function is the shorthand prompt_cache_key,. A search for prompt_cache_key: lands on a comparison of two requests, not on this assignment. Then open codex-rs/core/src/session/mod.rs and read record_step_world_state_if_changed. record_conversation_items runs only when the rendered diff is non-empty. Follow that into codex-rs/core/src/context_manager/history.rs and find the push onto self.items.

05.11

Real code: pi will not orphan a tool result

Your Compactor.compact keeps item 0 and the last keep_recent items. Suppose keep_recent is 3 and the history is the user request plus three call/result pairs, seven items. The last three start on a tool result whose call sits in the summarized middle. The model then sees a result with no call. pi will not cut there.

prepareCompaction calls findProjectedCutPoint. A ProjectedSessionEntry is one session entry after context edits have been applied. keepRecentTokens is how many recent tokens stay verbatim.

export const DEFAULT_COMPACTION_SETTINGS: CompactionSettings = {
  enabled: true,
  reserveTokens: 16384, // [1]
  keepRecentTokens: 20000, // [2]
};

Source: earendil-works/pi, packages/coding-agent/src/core/compaction/compaction.ts (MIT)

export function shouldCompact(contextTokens: number, contextWindow: number, settings: CompactionSettings): boolean {
  if (!settings.enabled) return false;
  return contextTokens > contextWindow - settings.reserveTokens; // [3]
}

Source: earendil-works/pi, packages/coding-agent/src/core/compaction/compaction.ts (MIT)

function isCutPointMessage(message: AgentMessage): boolean {
  switch (message.role) {
    case "user":
    case "assistant":
    case "bashExecution":
    case "custom":
    case "branchSummary":
    case "compactionSummary":
      return true;
    case "toolResult":
      return false; // [4]
  }
  return false;
}

Source: earendil-works/pi, packages/coding-agent/src/core/compaction/compaction.ts (MIT)

  let accumulatedTokens = 0;
  let exceededBudget = false;
  let cutIndex = cutPoints[0];
  for (let i = endIndex - 1; i >= startIndex; i--) {
    const messageTokens = entries[i].messages.reduce((sum, message) => sum + estimateTokens(message), 0);
    if (messageTokens === 0) continue;
    accumulatedTokens += messageTokens;
    if (accumulatedTokens >= keepRecentTokens) {
      exceededBudget = true;
      cutIndex = cutPoints.find((candidate) => candidate >= i) ?? cutPoints[cutPoints.length - 1]; // [5]
      break;
    }
  }

Source: earendil-works/pi, packages/coding-agent/src/core/compaction/compaction.ts (MIT)

The summary itself is a real model call, not a count of item types. This is the instruction, with the middle sections elided:

  const SUMMARIZATION_PROMPT = `The messages above are a conversation to summarize. Create a structured context checkpoint summary that another LLM will use to continue the work.

  Use this EXACT format:

  ## Goal
  [What is the user trying to accomplish? Can be multiple items if the session covers different tasks.]

  ## Constraints & Preferences
  - [Any constraints, preferences, or requirements mentioned by user]
  - [Or "(none)" if none were mentioned]

  // ... (Progress and Key Decisions)

  ## Next Steps
  1. [Ordered list of what should happen next]

  ## Critical Context
  - [Any data, examples, or references needed to continue]

  // ...

  Keep each section concise. Preserve exact file paths, function names, and error messages.`;

Source: earendil-works/pi, packages/coding-agent/src/core/compaction/compaction.ts (MIT)

What to notice

  1. [1] reserveTokens is headroom, 16384 by default. Compaction starts when the context is inside that reserve of the window, so the model still has room to answer. Your toy waits until a fixed max_tokens and has no separate reserve.
  2. [2] keepRecentTokens is 20000 by default, kept verbatim. Your toy keeps a count of items, so four huge logs can still blow the budget.
  3. [3] The check is contextTokens > contextWindow - reserveTokens. Disabled settings skip it. The trigger does not itself delete anything.
  4. [4] A toolResult is not a legal cut, and neither is a system message, which falls through to the final return false. User, assistant, bashExecution, custom, branchSummary, and compactionSummary are legal cuts. The tool result stays with the assistant entry that issued the call, on whichever side of the cut that entry lands.
  5. [5] The walk goes backward from the newest entry. When the estimate crosses keepRecentTokens at index i, the cut is the nearest legal point at or after i. A trailing tool result has no later legal point, so ?? uses the last one: the assistant message in front of those results. Call and result stay together.

getSummarizationFailure drops a summary whose stopReason is "length". That text is partial, so it must not become the checkpoint. Your summarize never calls a model, so it cannot fail this way.

Read it yourself: In packages/coding-agent/src/core/compaction/compaction.ts, read the rest of SUMMARIZATION_PROMPT, then getSummarizationFailure. Then read prepareCompaction and see what isSplitTurn does. The turn's prefix is summarized on its own, and the messages from the cut onward stay verbatim.

05.12

Real code: Codex rebuilds history around a summary

pi keeps a recent tail verbatim and summarizes what fell off. Codex's local compaction is harsher, and it is closer to a handoff than to your bookends. The compaction turn sends a prompt, takes the assistant's reply as the summary, and builds a new history. Recent tool calls are not copied. They have to survive inside the summary.

SUMMARIZATION_PROMPT is the default user text. config.compact_prompt replaces it when set. The reply is stored as summary_suffix, then summary_text is SUMMARY_PREFIX, a newline, and that reply.

You are performing a CONTEXT CHECKPOINT COMPACTION. Create a handoff summary for another LLM that will resume the task.

Include:
- Current progress and key decisions made
- Important context, constraints, or user preferences
- What remains to be done (clear next steps)
- Any critical data, examples, or references needed to continue

Be concise, structured, and focused on helping the next LLM seamlessly continue the work.

Source: openai/codex, codex-rs/prompts/templates/compact/prompt.md (Apache-2.0)

fn build_compacted_history_with_limit(
    mut history: Vec<ResponseItemEnvelope>,
    user_messages: &[CompactedUserMessage<'_>],
    summary_text: &str,
    max_tokens: usize,
) -> Vec<ResponseItemEnvelope> {
    let mut selected_messages = Vec::new();
    if max_tokens > 0 {
        let mut remaining = max_tokens;
        for message in user_messages.iter().rev() { // [1]
            if remaining == 0 {
                break; // [2]
            }
            let tokens = approx_token_count(&message.message);

Source: openai/codex, codex-rs/core/src/compact.rs (Apache-2.0)

            selected_messages.push(ResponseItemEnvelope {
                item,
                metadata: harness_metadata,
            });
            if tokens > remaining {
                break;
            }
            remaining = remaining.saturating_sub(tokens);
        }
        selected_messages.reverse(); // [3]
    }
    history.extend(selected_messages);

Source: openai/codex, codex-rs/core/src/compact.rs (Apache-2.0)

    history.push(ResponseItemEnvelope::new(ContextualUserFragment::into(
        CompactionSummary::new(summary_text), // [4]
    )));

    history
}

Source: openai/codex, codex-rs/core/src/compact.rs (Apache-2.0)

What to notice

  1. [1] user_messages comes from collect_annotated_user_messages. That helper keeps real user messages and drops anything that already starts with SUMMARY_PREFIX, so an old summary is not kept as if the user had typed it. The loop walks newest user message first. Assistant turns and tool results are not in this list.
  2. [2] max_tokens here is COMPACT_USER_MESSAGE_MAX_TOKENS, 20000. This break fires when that budget is already zero, so an older user message is not taken. Between these two excerpts, a message that fits and is text only is copied, and any other message keeps a truncated text prefix. A message that costs more than what is left is still pushed, and the later break stops the walk.
  3. [3] reverse puts the kept user messages back into time order. They were collected newest first.
  4. [4] CompactionSummary is a user-role fragment, pushed last by this function. The caller passed an empty initial context into build_compacted_history. build_compaction_replacement_history then rebuilds the full initial context: every world-state section plus the developer instructions. Prefix items go at the front. The rest follow the four rules in insert_initial_context_before_last_real_user_or_summary. The first choice is the slot before the last real user message, agent message, or retained tool output, so the summary stays last.

Your summarize invents a one-line count of item types. This prompt asks for progress, constraints, next steps, and the data the next model needs.

Read it yourself: Open codex-rs/prompts/templates/compact/summary_prefix.md and compare it with SUMMARY_PREFIX in codex-rs/core/src/compact.rs, on the line that formats summary_text. Then read insert_initial_context_before_last_real_user_or_summary and check the four placement rules in its comment. In codex-rs/core/src/tasks/compact.rs, see which function runs when the provider's remote compaction support is V2 instead of this local prompt.

05.13

Read the source yourself

Read these in order. Each link is the commit this chapter quotes.

  1. pi's marker sites. packages/ai/src/api/anthropic-messages.ts. Search for cache_control. Count how many request shapes get a marker, and which message role is refused one.
  2. OpenCode's budget. packages/ai/src/cache-policy.ts. Read selectPolicy. After the cacheFormat checks, a route missing from RESPECTS_INLINE_HINTS gets NONE, so automatic placement adds nothing. A manual hint already on the request stays. The comment above the set says those providers ignore the tag.
  3. Codex appending a change. environment.rs. Find has_same_diff_value and the <cwd> write inside RenderedEnvironments. Confirm a directory change cannot edit an older <environment_context> message, because the diff only builds a new fragment.
  4. pi's cut and the failed summary. compaction.ts. Read findProjectedCutPoint through isSplitTurn, then getSummarizationFailure. Note which stopReason values must not be saved.
  5. Codex's rebuilt history. compact.rs and prompt.md. Trace one user message from collect_annotated_user_messages to the push of CompactionSummary.

Exercise: Open Gemini CLI's chatCompressionService.ts (Apache-2.0). In findCompressSplitPoint, an entry is a split point only when its role is user and it has no functionResponse. Model entries are never split points. Separately, the function will not return contents.length (compress everything) when the last entry is a model turn that still has a functionCall. Then find the status returned when newTokenCount > originalTokenCount. That comparison is the rebuilt history, which is the summary plus a short acknowledgement plus the kept tail, against the original history. The status is COMPRESSION_FAILED_INFLATED_TOKEN_COUNT. pi's cut and this split share one rule: do not separate a call from its result, and do not keep a summary that failed to shrink the history.

05.14

Connecting to the bigger picture

Chapter 4 measured the problem; this chapter solves it. The apply_change_cache_safely pattern protects the prefix that Chapter 3's PromptBuilder carefully put first. The Compactor keeps the loop from Chapter 2 from ever hitting the window ceiling. Chapter 9's file-based memory is the third context-management move (external storage). And Chapter 11's subagents are arguably the best context tool of all: a subagent does noisy work in its own window and returns a tiny summary, so the main context never sees the mess.

05.15

Key takeaways

  • Prompt caching reuses computed state for an exact prompt prefix, at roughly a tenth the cost. It turns the quadratic cost curve back into a linear one.
  • Cache hits need exact prefix matches, so put stable content first and variable content last, and never edit earlier messages mid-session; append new ones instead.
  • Common cache-busters: changing tools, switching models, or changing the sandbox. A working-directory change and an AGENTS.md edit are appended as new messages, so the old prefix still matches and only the new message is uncached. Lock instructions before long sessions.
  • Compaction keeps you under the context window by summarizing old turns (oldest tool outputs first), while preserving the original request and recent exchanges.
  • Truncation (head plus tail) tames giant tool outputs, and context diffing avoids re-sending unchanged context. Compaction is lossy, so keep sessions focused.

Original sources: OpenAI's Unrolling the Codex agent loop, Anthropic's How Claude Code works, and the Deep Research Agents survey (arXiv:2506.18096).


0 of 15 slides completed

Your learning progress stays in this browser.

Self-check · 5 questions

01 / 05

Why does prompt caching turn a quadratic cost curve back into a linear one?

Related chapters