CH 04 · Tokens, Context Windows, and the Quadratic Problem · 01/13
Read as one pageTokens, Context Windows, and the Quadratic Problem
Every turn replays the full conversation, so tokens sent grow quadratically as a session lengthens, capped by the context window.
13 slides · 13 min read
In this chapter
- 01Real code: one model's limits and prices in LiteLLM
- 02The quadratic problem (the insight most people miss)
- 03Real code: Codex rebuilds the prompt from all of history
- 04Why it matters
- 05How it works: estimating before you spend
- 06Code MVP: a token and cost estimator
- 07Real code: three harnesses guess tokens from length
- 08Real code: pi prices the provider's real usage
- 09Real code: OpenCode shrinks the reply to fit
- 10Read the source yourself
- 11Connecting to the bigger picture
- 12Key takeaways
Before the model can do anything with your prompt, the prompt is turned into numbers. This step is called tokenization: the text is chopped into tokens (chunks roughly the size of a short word or word-piece), and each token is mapped to an integer that indexes the model's vocabulary. A rough rule of thumb the harnesses use is about 4 characters per token. The model then samples output tokens one at a time, and those get decoded back into text. The token-by-token streaming you see in the terminal is literally this sampling process exposed to you.
Two facts about tokens shape the entire economics of agents.
Fact one: the model must read every input token before it can write a single output token. A bloated AGENTS.md, a huge file you read, a verbose tool result: all of it adds latency and cost to every inference call for the rest of the session, not just the next one.
Fact two: there is a hard ceiling called the context window. This is the maximum number of tokens the model can use for one inference call, and it counts both input and output. An agent that makes hundreds of tool calls in one turn can run straight into that ceiling. Managing it is one of the harness's core jobs.
Finished this idea?
Your learning progress stays in this browser.