Chapter 04 · 13 min read · Updated
Tokens, Context Windows, and the Quadratic Problem
Every turn replays the full conversation, so tokens sent grow quadratically as a session lengthens, capped by the context window.
Prefer one idea per screen? Read it as 13 slides →
In this chapter13
- 04.1Concept explanation
- 04.2Real code: one model's limits and prices in LiteLLM
- 04.3The quadratic problem (the insight most people miss)
- 04.4Real code: Codex rebuilds the prompt from all of history
- 04.5Why it matters
- 04.6How it works: estimating before you spend
- 04.7Code MVP: a token and cost estimator
- 04.8Real code: three harnesses guess tokens from length
- 04.9Real code: pi prices the provider's real usage
- 04.10Real code: OpenCode shrinks the reply to fit
- 04.11Read the source yourself
- 04.12Connecting to the bigger picture
- 04.13Key takeaways
04.1
Concept explanation
Before the model can do anything with your prompt, the prompt is turned into numbers. This step is called tokenization: the text is chopped into tokens (chunks roughly the size of a short word or word-piece), and each token is mapped to an integer that indexes the model's vocabulary. A rough rule of thumb the harnesses use is about 4 characters per token. The model then samples output tokens one at a time, and those get decoded back into text. The token-by-token streaming you see in the terminal is literally this sampling process exposed to you.
Two facts about tokens shape the entire economics of agents.
Fact one: the model must read every input token before it can write a single output token. A bloated AGENTS.md, a huge file you read, a verbose tool result: all of it adds latency and cost to every inference call for the rest of the session, not just the next one.
Fact two: there is a hard ceiling called the context window. This is the maximum number of tokens the model can use for one inference call, and it counts both input and output. An agent that makes hundreds of tool calls in one turn can run straight into that ceiling. Managing it is one of the harness's core jobs.
04.2
Real code: one model's limits and prices in LiteLLM
The toy you build later in this chapter hard-codes two things a real harness has to look up: how big the window is, and what each token costs. LiteLLM, a Python library that puts one interface in front of many model providers, keeps both for every model in a single JSON file. Here is its entry for Claude Sonnet 4.5. Watch which numbers describe the window and which describe money.
"claude-sonnet-4-5": {
"cache_creation_input_token_cost": 3.75e-06,
// ...
"cache_read_input_token_cost": 3e-07, // [1]
"input_cost_per_token": 3e-06, // [2]
// ... (batch prices, deprecation date)
"litellm_provider": "anthropic",
"max_input_tokens": 200000, // [3]
"max_output_tokens": 64000, // [4]
"max_tokens": 64000, // [5]
"mode": "chat",
"output_cost_per_token": 1.5e-05,
// ... (batch price, web search prices, feature flags)
"supports_prompt_caching": true,
// ...
"prompt_cache_min_tokens": 1024,
"source": "https://platform.claude.com/docs/en/about-claude/pricing"
},
Source: BerriAI/litellm, model_prices_and_context_window.json (MIT)
What to notice
- [1] Every price is in dollars per single token, so the numbers look tiny. Multiply by a million to get the usual form:
3e-07is $0.30 per million tokens read from the prompt cache (Chapter 5 covers the cache itself). - [2]
3e-06is $3 per million fresh input tokens. A cached token costs one tenth of that, the same ratio as the toy'scached_rate = 0.1. Further down,output_cost_per_tokenis $15 per million, five times the input price, which is why the toy makesoutput_ratebigger thaninput_rate. - [3]
max_input_tokensis 200,000, the number LiteLLM stores as this model's input cap. It is not, by itself, Fact two's combined ceiling. - [4]
max_output_tokensis a separate 64,000. The file does not say whether those 64,000 come out of the 200,000. A later slide shows OpenCode fitting the reply into the room the prompt leaves. - [5] The template marks
max_tokensLEGACY: it copiesmax_output_tokenswhen the provider has one, and otherwise copiesmax_input_tokens. On this row it copies the 64,000 output cap, so reading it as the window misses the 200,000 input cap.
Read it yourself: open the same file at the sample_spec entry, the template at the top that documents every field, and read the three lines about max_input_tokens, max_output_tokens and max_tokens. Then open generic_cost_per_token in litellm/litellm_core_utils/llm_cost_calc/utils.py. It looks up this entry with get_model_info and returns two numbers. Find out what they are, and where cache-hit tokens get priced.
04.3
The quadratic problem (the insight most people miss)
Recall from Chapter 2 that every new turn re-sends the entire conversation history. Now combine that with tokens costing money per call, and you get a nasty surprise.
Here is the growth pattern, straight from the Codex deep-dive:
Turn 1: [system] + [user msg 1] -> ~2k tokens
Turn 5: [system] + 4 turns + tool calls -> ~15k tokens
Turn 20: [system] + 19 turns + dozens of tool calls -> ~80k tokens
Turn 50: [system] + 49 turns + hundreds of tool calls -> potentially millions
Each turn re-sends everything before it. So the total amount of data sent over the lifetime of a session is quadratic in the number of turns. Double the session length and you roughly quadruple the total tokens sent, and therefore the cost.
If that word "quadratic" feels abstract, here is the everyday version. Imagine a meeting where the rule is that before anyone speaks, they must first read the entire transcript of the meeting so far, out loud. The first person reads almost nothing. The tenth person reads nine speeches. The fiftieth person reads forty-nine. The meeting does not get linearly longer; it balloons. That is your agent session.
This is not a bug. The model genuinely needs the history to reason coherently from turn to turn; it has no memory otherwise. But the consequences are real and worth stating plainly:
- Long sessions are disproportionately expensive. The cost is not linear in length.
- Editing-heavy sessions burn context fastest. Lots of
apply_patchcalls means lots of large diffs piling up in history. A session that does heavy file editing exhausts the window faster than one doing light Q&A. - One session should do one focused task. This is the single most important workflow habit that falls out of understanding the loop. Do not run a marathon session that does everything; start fresh sessions, or delegate to subagents (Chapter 11), for distinct jobs.
04.4
Real code: Codex rebuilds the prompt from all of history
The toy's simulate_session (coming up in the Code MVP) adds 1,500 tokens every turn and charges for the whole running total. Codex, OpenAI's coding agent written in Rust, keeps that total in a ContextManager. Each model call in run_turn copies the manager and calls for_prompt. Chapter 2 walks the loop around that call. This is the copy.
/// Returns the history prepared for sending to the model. This applies a proper
/// normalization and drops un-suited items. Unsupported image and audio content
/// is stripped from messages and tool outputs according to `input_modalities`.
pub(crate) fn for_prompt(self, input_modalities: &[InputModality]) -> Vec<ResponseItem> { // [1]
self.for_prompt_annotated(input_modalities)
.into_iter()
.map(ResponseItemEnvelope::into_item)
.collect()
}
/// Returns normalized history envelopes for internal consumers that must retain metadata.
pub(crate) fn for_prompt_annotated( // [2]
mut self,
input_modalities: &[InputModality],
) -> Vec<ResponseItemEnvelope> {
self.normalize_history(input_modalities); // [3]
Arc::unwrap_or_clone(self.items) // [4]
}
Source: openai/codex, codex-rs/core/src/context_manager/history.rs (Apache-2.0)
normalize_history is the filter that runs first. Its comment lists every reason an item changes.
/// This function enforces a couple of invariants on the in-memory history:
/// 1. every call (function/custom) has a corresponding output entry
/// 2. every output has a corresponding call entry or names an external tool event
/// 3. unsupported image and audio content is stripped from messages and tool outputs
fn normalize_history(&mut self, input_modalities: &[InputModality]) { // [5]
let items = Arc::make_mut(&mut self.items);
// all function/tool calls must have a corresponding output
normalize::ensure_call_outputs_present(items); // [6]
// Paired outputs must have a corresponding call; named external outputs stand alone.
normalize::remove_orphan_outputs(items);
// strip images when model does not support them
normalize::strip_images_when_unsupported(input_modalities, items); // [7]
// strip audio when model does not support it
normalize::strip_audio_when_unsupported(input_modalities, items);
}
Source: openai/codex, codex-rs/core/src/context_manager/history.rs (Apache-2.0)
What to notice
- [1]
for_promptconsumes the copy (the session keeps its own) and returns aVec<ResponseItem>. AResponseItemis one API entry: a message, a tool call, a tool result, and a few other kinds. The return value is the whole list. - [2]
for_prompt_annotatedkeeps each item's envelope, the metadata wrapper.for_promptthrows the wrapper away and keeps the item. - [3] Normalization runs on the stored list before anything is returned.
- [4]
Arc::unwrap_or_clone(self.items)hands back every stored item. There is no "last few turns" slice. - [5] The comment's three rules are the filter: a call needs an output, an output needs a call (or it names an external tool), and image or audio the model cannot take is removed. Age is not a rule.
- [6] The pairing helpers fix missing outputs and orphan outputs. They do not drop old turns.
- [7]
input_modalitiesis what this model accepts. An image stays when the model accepts images.
Compaction can replace the stored items with a shorter summary before the next copy. Until it does, the list only grows, so pass n includes everything from earlier passes plus the new items. That is the quadratic curve.
On a WebSocket, get_incremental_items can shrink the upload. The baseline is the previous request's input plus the output items the server already returned. If the new input starts with that baseline, the websocket body sends only the suffix, with previous_response_id. OpenAI's conversation state guide says those earlier input tokens are still billed.
Read it yourself: open get_incremental_items and prepare_websocket_request just under it. List every case that refuses the suffix, so the full input is sent. A history that got shorter, which compaction does, is one of them.
04.5
Why it matters
This chapter is the "why" behind half the techniques in the rest of the guide. Prompt caching (Chapter 5) exists to break the quadratic cost curve. Compaction (Chapter 5) exists to keep you under the window ceiling. Subagents (Chapter 11) exist so a noisy research task can run in its own context and report back a small summary instead of dumping ten thousand tokens into your main thread. Budgets (Chapter 12) exist so a retry loop cannot quietly run the meter to a hundred dollars. Every one of those is a response to the two facts and the quadratic curve above.
It also matters for your wallet directly. The "Everything About Codex" guide reports that a debugging-heavy task that loops ten or twenty times, each lap replaying the full history, is where bills spike. The mitigation it recommends is exactly what this chapter motivates: cap turns, set token limits, keep scope bounded.
04.6
How it works: estimating before you spend
You do not need an exact tokenizer to reason about this; a good estimate is enough to build budgets and warnings. Real harnesses use byte-based heuristics (roughly 4 characters per token) for fast counting, and only fall back to a precise tokenizer when it matters. Here is how to think about the cost of a whole session.
If a turn's prompt has P tokens and produces O output tokens, that turn's cost is roughly P * input_rate + O * output_rate. Because P grows each turn, summing over a session gives you that quadratic shape. The estimator below makes this concrete and even shows the difference caching makes (foreshadowing Chapter 5).
04.7
Code MVP: a token and cost estimator
"""
chapter 04: token and cost estimation.
A fast, approximate token counter plus a session cost model that makes
the quadratic growth visible. This becomes the capstone's budget meter.
"""
from dataclasses import dataclass
CHARS_PER_TOKEN = 4 # the standard rough heuristic used by real harnesses
def estimate_tokens(text: str) -> int:
"""Cheap approximation: ~4 characters per token."""
return max(1, len(text) // CHARS_PER_TOKEN)
def estimate_items_tokens(items: list) -> int:
"""Total tokens across a list of structured history items (Ch 3)."""
return sum(estimate_tokens(str(item.get("content", ""))) for item in items)
@dataclass
class CostModel:
input_rate: float = 1.0 # cost per 1k fresh input tokens (arbitrary units)
output_rate: float = 4.0 # output usually costs more than input
cached_rate: float = 0.1 # cached input is ~10x cheaper (Chapter 5)
def turn_cost(self, prompt_tokens: int, output_tokens: int,
cached_tokens: int = 0) -> float:
fresh = prompt_tokens - cached_tokens
return (fresh / 1000) * self.input_rate \
+ (cached_tokens / 1000) * self.cached_rate \
+ (output_tokens / 1000) * self.output_rate
def simulate_session(turns: int, tokens_added_per_turn: int = 1500,
output_per_turn: int = 300, use_cache: bool = False):
"""Show how total cost grows with session length."""
model = CostModel()
history_tokens = 0
total = 0.0
for t in range(1, turns + 1):
history_tokens += tokens_added_per_turn # the prompt keeps growing
# With caching, everything except this turn's new content is cached.
cached = history_tokens - tokens_added_per_turn if use_cache else 0
total += model.turn_cost(history_tokens, output_per_turn, cached)
return round(total, 2)
if __name__ == "__main__":
for n in (5, 10, 20, 50):
no_cache = simulate_session(n, use_cache=False)
cached = simulate_session(n, use_cache=True)
print(f"{n:>3} turns | no cache: {no_cache:>8} | with cache: {cached:>7}")
Run it and watch the "no cache" column grow far faster than the turn count, while the "with cache" column grows much more gently. That gap is the entire argument for Chapter 5. The numbers are in arbitrary units, but the shape is what matters: without caching, doubling the turns roughly quadruples the cost; with caching, the per-turn marginal cost grows slowly.
04.8
Real code: three harnesses guess tokens from length
You just wrote estimate_tokens: characters divided by 4, rounded down, never less than 1. OpenCode (a TypeScript coding agent), Codex, and pi (a small toolkit; pi-ai is its provider library) guess the same way, with no vocabulary loaded. Watch the divisor, the rounding, and whatever is not plain text.
OpenCode's Token.estimate is the function the rest of its session code calls.
export * as Token from "./token.js"
const CHARS_PER_TOKEN = 4
export const estimate = (input: string) => Math.max(0, Math.round(input.length / CHARS_PER_TOKEN)) // [1]
Source: anomalyco/opencode, packages/core/src/util/token.ts (MIT)
Codex counts bytes and rounds up. saturating_add adds and, past the largest value the type can hold, stays there, so a huge length cannot wrap around to a small one.
const APPROX_BYTES_PER_TOKEN: usize = 4; // [2]
// ... (helpers that cut a long string down to a head and a tail)
pub fn approx_token_count(text: &str) -> usize {
let len = text.len(); // [3]
len.saturating_add(APPROX_BYTES_PER_TOKEN.saturating_sub(1)) / APPROX_BYTES_PER_TOKEN // [4]
}
Source: openai/codex, codex-rs/utils/string/src/truncate.rs (Apache-2.0)
pi divides by 3.5 and walks one Message, a single transcript entry, by role.
const CHARS_PER_TOKEN = 3.5; // [5]
const ESTIMATED_IMAGE_CHARS = 4800; // [6]
// ... (estimateTextTokens rounds up: Math.ceil(length / 3.5))
export function estimateMessageTokens(message: Message): number {
let chars = 0;
if (message.role === "system") {
return (
estimateTextTokens(getSystemMessageText(message)) +
estimateToolsTokens(message.toolsAdded) + // [7]
estimateToolsTokens(message.toolsRemoved)
);
}
if (message.role === "user") return estimateTextAndImageContentTokens(message.content);
if (message.role === "toolResult") return estimateTextAndImageContentTokens(message.content);
for (const block of message.content) {
if (block.type === "text") {
chars += block.text.length;
} else if (block.type === "thinking") {
chars += block.thinking.length; // [8]
} else {
chars += block.name.length + safeJsonStringify(block.arguments).length; // [9]
}
}
return Math.ceil(chars / CHARS_PER_TOKEN);
}
Source: earendil-works/pi, packages/ai/src/utils/estimate.ts (MIT)
What to notice
- [1]
Math.roundsends a half up: 6 characters become 2 tokens, and your//gives 1. One character rounds to 0. Yours returns 1. - [2] The unit is bytes. The next function,
approx_bytes_for_tokens(lines 92-94, past this range), multiplies a token budget by 4 so tool-output truncation knows how many bytes to keep. - [3]
text.len()is bytes.éis two bytes, so other languages come out higher than a character count. - [4] Adding 3 before dividing by 4 rounds up: 5 bytes become 2 tokens. Your
//turns 5 characters into 1. - [5] 100 characters is 25 tokens at your divisor of 4, and
Math.ceil(100 / 3.5)is 29. - [6] The skipped helper adds 4,800 characters per image (1,372 tokens) and never inspects the pixels. Your counter only sees
content. - [7]
toolsAddedis the full definitions that become available.toolsRemovedis only the names that go away. Both are JSON-encoded and divided by 3.5. A tool schema outsidecontentadds nothing to your sum. An item with nocontentstill counts as 1, becauseestimate_tokens("")ismax(1, 0). - [8]
thinkingis reasoning text stored apart from the visible answer, and it counts. - [9] A tool call counts its name plus its arguments. A
content-only sum misses both.
Read it yourself: open estimateContextTokens in the same file. When an earlier assistant reply carries a real Usage, pi keeps that number and estimates only the messages after it.
04.9
Real code: pi prices the provider's real usage
CostModel.turn_cost multiplies your own guess by arbitrary rates per thousand tokens. pi waits until the provider answers, then prices the counts the provider reports. Those counts sit on a Usage value attached to the assistant message.
export interface Usage {
input: number; // [1]
output: number;
cacheRead: number;
cacheWrite: number;
/** Subset of `cacheWrite` written with 1h retention. Only Anthropic reports this split. */
cacheWrite1h?: number; // [2]
// ... (reasoning tokens are a subset of output, so they are already counted)
reasoning?: number;
totalTokens: number;
cost: { // [3]
input: number;
output: number;
cacheRead: number;
cacheWrite: number;
total: number;
};
}
Source: earendil-works/pi, packages/ai/src/types.ts (MIT)
calculateCost fills usage.cost from the model's price table. The rates are US dollars per million tokens, the same kind of numbers as the LiteLLM entry earlier, written per million instead of per token.
export function calculateCost(model: AnyModel, usage: Usage): Usage["cost"] {
const inputTokens = usage.input + usage.cacheRead + usage.cacheWrite; // [4]
let rates: ModelCostRates = model.cost;
let matchedThreshold = -1;
for (const tier of model.cost.tiers ?? []) { // [5]
if (inputTokens > tier.inputTokensAbove && tier.inputTokensAbove > matchedThreshold) {
rates = tier;
matchedThreshold = tier.inputTokensAbove;
}
}
// Anthropic charges 2x base input for 1h cache writes.
const longWrite = usage.cacheWrite1h ?? 0; // [6]
const shortWrite = usage.cacheWrite - longWrite;
usage.cost.input = (rates.input / 1000000) * usage.input; // [7]
usage.cost.output = (rates.output / 1000000) * usage.output;
usage.cost.cacheRead = (rates.cacheRead / 1000000) * usage.cacheRead;
usage.cost.cacheWrite = (rates.cacheWrite * shortWrite + rates.input * 2 * longWrite) / 1000000;
usage.cost.total = usage.cost.input + usage.cost.output + usage.cost.cacheRead + usage.cost.cacheWrite;
return usage.cost;
}
Source: earendil-works/pi, packages/ai/src/models.ts (MIT)
What to notice
- [1]
inputis fresh tokens,outputis tokens written,cacheReadis a cache hit,cacheWriteis a cache fill. Yourturn_costhas no write price, so a cache fill looks free. - [2]
cacheWrite1his the part ofcacheWritestored for one hour. Anthropic reports that split.reasoningis already insideoutput, so leave it out of the sum. - [3]
costis written here, in dollars.totalTokensis the provider's own total when it sends one. - [4] The tier check adds all three input kinds. A cached token still counts toward how big the prompt was.
- [5]
inputTokensAboveis where a higher price starts. The highest step the prompt has passed sets the rates for the whole request. The price schema calls that request-wide. - [6] One-hour writes are billed at twice the base input rate.
?? 0treats a missing split as none. Your singlecached_ratecannot say this. - [7] Dividing by one million turns a per-million rate into a per-token rate. Your toy divides by one thousand, and its rates are arbitrary units.
The Anthropic adapter fills these fields from message_start, adds totalTokens itself (its comment says Anthropic provides none), then calls calculateCost.
Read it yourself: open anthropic-messages.ts at message_start and write down which provider field lands in which Usage field. Then search for message_delta in the same function, about 130 lines later. That branch writes a field only when the event includes it. A missing input_tokens stored as 0 would wipe the count from message_start.
04.10
Real code: OpenCode shrinks the reply to fit
Your estimator returns a number and stops. It never asks whether that number plus the reply still fits in the window. OpenCode asks, in outputLimit, before it builds the request. limit comes from the model catalog: limit.context is the window, limit.output is the longest reply that model is allowed to write.
// Used when the catalog has no output limit for the model.
const OUTPUT_TOKEN_FALLBACK = 32_000
// No reply needs more, however much the model allows.
const OUTPUT_TOKEN_MAX = 256_000
// ... (a summary is capped at 32,000 tokens)
// Prompt text is estimated at about 4 characters per token, which can run low on dense text such as code.
const ESTIMATE_ERROR = 0.15 // [1]
// Never ask for less; only reachable with automatic compaction off, since it keeps the window from filling this far.
const OUTPUT_TOKEN_MIN = 1_024 // [2]
// ... (request types)
/** The default output limit: the catalog limit, fitted to the room the prompt leaves in the context window. */
export const outputLimit = (
limit: Model.Info["limit"],
kind: "primary" | "compaction",
inputTokens?: Input["inputTokens"],
) => {
const model = Math.min(limit.output > 0 ? limit.output : OUTPUT_TOKEN_FALLBACK, OUTPUT_TOKEN_MAX)
const requested = kind === "compaction" ? Math.min(model, SUMMARY_OUTPUT_MAX) : model // [3]
if (inputTokens === undefined || limit.context <= 0) return requested // [4]
const room = limit.context - inputTokens.measured - Math.ceil(inputTokens.estimated * (1 + ESTIMATE_ERROR)) // [5]
return Math.min(requested, Math.max(OUTPUT_TOKEN_MIN, room)) // [6]
}
Source: anomalyco/opencode, packages/core/src/session/model-request.ts (MIT)
What to notice
- [1] 0.15 is a pad. The comment above it says the 4-character guess can come out low on dense text such as code, so
estimatedis multiplied by 1.15. - [2] 1,024 is the floor applied to
roombefore the comparison withrequested. The comment says a full window is reachable when automatic compaction is off. - [3]
kindis"primary"for a normal turn and"compaction"for the summary call from Chapter 5. A summary is clamped to 32,000. Every reply is capped at 256,000, catalog or not. - [4] With no measurement, or
limit.contextof 0 or less, OpenCode skips the subtraction and returnsrequestedfrom [3]: still capped at 256,000, and at 32,000 for a summary. A catalog output of 1,000,000 becomes 256,000. - [5]
measuredsums the latest usable reply from this provider: non-cached input, cache read, cache write, visible output, and reasoning. An errored reply, or another provider's count, is skipped.estimatedis the messages after that reply. Text usesToken.estimate. Images useestimateMedia. System text and tool definitions stay out ofestimatedonce that reply exists, because they already sit in the input count. With a window of 200,000,measured150,000, andestimated10,000, room is200000 - 150000 - ceil(10000 * 1.15)= 38,500. (Those three figures are made up.) - [6] The budget is the smaller of
requestedand the room, and the room is raised to 1,024 first. A catalog limit already under 1,024 stays under it. Room 0 against a 64,000 catalog still asks for 1,024. Compaction is what cuts the history.
runStep passes estimatePrompt(loaded) as inputTokens on the primary call.
Read it yourself: open clampMaxTokensToContext in pi. It subtracts a flat 4,096 and, when nothing is left, asks for 1 output token. Then find inputTokens in OpenCode's runStep (llm.ts). estimatePrompt in compaction.ts is what splits it into measured and estimated.
04.11
Read the source yourself
Five stops, in the order the chapter used them. Each link is pinned to the commit these slides were read against.
- LiteLLM's model registry. Read
sample_specat the top of the file first, then theclaude-sonnet-4-5entry. Check which field is the window and which field the template calls legacy. - OpenCode's
Token.estimate, thenoutputLimit.outputLimitis where the guess changesmaxTokens. Compaction also usesToken.estimate, inoldestToDropincompaction.ts, to choose which older messages to drop, so the guess changes the prompt itself. - pi's
estimate.ts. ReadestimateMessageTokens, thenestimateContextTokensfurther down that file. The second function is where a real provider total replaces the guess. - pi's
calculateCost, then the first case inmodel-cost-tiers.test.ts. Base input is $0.10 per million tokens. The step starts once input is over 100,000 tokens. At 100,000 the rate is still $0.10 per million. At 150,000 fresh input tokens,calculateCostbills all 150,000 at $0.50 per million, which is $0.075. - Codex's
for_prompt, thennormalize_historyin the same file. Then readget_incremental_itemsand list the cases that send the full input anyway.
Try this on the toy. Change estimate_tokens so it rounds up the way Codex does: return max(1, (len(text) + CHARS_PER_TOKEN - 1) // CHARS_PER_TOKEN). The printout does not call estimate_tokens. Compare estimate_tokens("abcde") and estimate_tokens("abcdef") before and after the edit: 5 characters go from 1 to 2, and 6 from 1 to 2. The session printout stays put until simulate_session uses that function.
04.12
Connecting to the bigger picture
Chapter 3 built the prompt; this chapter measures it and explains why it cannot grow forever. Chapter 5 attacks the problem from both sides: caching to tame the cost curve, and compaction to stay under the window ceiling. The estimate_tokens and CostModel you wrote here become the capstone's budget meter, the thing Chapter 12 wires into a real kill switch.
04.13
Key takeaways
- Text becomes tokens (about 4 characters each); the model reads all input tokens before producing any output token, so bloat taxes every later call.
- The context window is a hard ceiling on input plus output tokens for one inference call.
- Because every turn re-sends the full history, total tokens over a session grow quadratically with the number of turns. Doubling length roughly quadruples cost.
- Editing-heavy sessions fill the window fastest. The healthy habit is one focused task per session, with delegation for the rest.
- You can estimate tokens cheaply (4 chars each) and build budgets on top of that estimate, which is exactly what later chapters do.
Original source: the "Inside the Codex Agent Loop" deep-dive based on Michael Bolin's Unrolling the Codex agent loop.
0 of 13 slides completed
Your learning progress stays in this browser.
Self-check · 5 questions
01 / 05
What is the approximate rule of thumb for converting characters to tokens?