CH 04 · Tokens, Context Windows, and the Quadratic Problem · 04/13
Read as one pageReal code: Codex rebuilds the prompt from all of history
The toy's simulate_session (coming up in the Code MVP) adds 1,500 tokens every turn and charges for the whole running total. Codex, OpenAI's coding agent written in Rust, keeps that total in a ContextManager. Each model call in run_turn copies the manager and calls for_prompt. Chapter 2 walks the loop around that call. This is the copy.
/// Returns the history prepared for sending to the model. This applies a proper
/// normalization and drops un-suited items. Unsupported image and audio content
/// is stripped from messages and tool outputs according to `input_modalities`.
pub(crate) fn for_prompt(self, input_modalities: &[InputModality]) -> Vec<ResponseItem> { // [1]
self.for_prompt_annotated(input_modalities)
.into_iter()
.map(ResponseItemEnvelope::into_item)
.collect()
}
/// Returns normalized history envelopes for internal consumers that must retain metadata.
pub(crate) fn for_prompt_annotated( // [2]
mut self,
input_modalities: &[InputModality],
) -> Vec<ResponseItemEnvelope> {
self.normalize_history(input_modalities); // [3]
Arc::unwrap_or_clone(self.items) // [4]
}
Source: openai/codex, codex-rs/core/src/context_manager/history.rs (Apache-2.0)
normalize_history is the filter that runs first. Its comment lists every reason an item changes.
/// This function enforces a couple of invariants on the in-memory history:
/// 1. every call (function/custom) has a corresponding output entry
/// 2. every output has a corresponding call entry or names an external tool event
/// 3. unsupported image and audio content is stripped from messages and tool outputs
fn normalize_history(&mut self, input_modalities: &[InputModality]) { // [5]
let items = Arc::make_mut(&mut self.items);
// all function/tool calls must have a corresponding output
normalize::ensure_call_outputs_present(items); // [6]
// Paired outputs must have a corresponding call; named external outputs stand alone.
normalize::remove_orphan_outputs(items);
// strip images when model does not support them
normalize::strip_images_when_unsupported(input_modalities, items); // [7]
// strip audio when model does not support it
normalize::strip_audio_when_unsupported(input_modalities, items);
}
Source: openai/codex, codex-rs/core/src/context_manager/history.rs (Apache-2.0)
What to notice
- [1]
for_promptconsumes the copy (the session keeps its own) and returns aVec<ResponseItem>. AResponseItemis one API entry: a message, a tool call, a tool result, and a few other kinds. The return value is the whole list. - [2]
for_prompt_annotatedkeeps each item's envelope, the metadata wrapper.for_promptthrows the wrapper away and keeps the item. - [3] Normalization runs on the stored list before anything is returned.
- [4]
Arc::unwrap_or_clone(self.items)hands back every stored item. There is no "last few turns" slice. - [5] The comment's three rules are the filter: a call needs an output, an output needs a call (or it names an external tool), and image or audio the model cannot take is removed. Age is not a rule.
- [6] The pairing helpers fix missing outputs and orphan outputs. They do not drop old turns.
- [7]
input_modalitiesis what this model accepts. An image stays when the model accepts images.
Compaction can replace the stored items with a shorter summary before the next copy. Until it does, the list only grows, so pass n includes everything from earlier passes plus the new items. That is the quadratic curve.
On a WebSocket, get_incremental_items can shrink the upload. The baseline is the previous request's input plus the output items the server already returned. If the new input starts with that baseline, the websocket body sends only the suffix, with previous_response_id. OpenAI's conversation state guide says those earlier input tokens are still billed.
Read it yourself: open get_incremental_items and prepare_websocket_request just under it. List every case that refuses the suffix, so the full input is sent. A history that got shorter, which compaction does, is one of them.
Finished this idea?
Your learning progress stays in this browser.