CH 04 · Tokens, Context Windows, and the Quadratic Problem · 08/13
Read as one pageReal code: three harnesses guess tokens from length
You just wrote estimate_tokens: characters divided by 4, rounded down, never less than 1. OpenCode (a TypeScript coding agent), Codex, and pi (a small toolkit; pi-ai is its provider library) guess the same way, with no vocabulary loaded. Watch the divisor, the rounding, and whatever is not plain text.
OpenCode's Token.estimate is the function the rest of its session code calls.
export * as Token from "./token.js"
const CHARS_PER_TOKEN = 4
export const estimate = (input: string) => Math.max(0, Math.round(input.length / CHARS_PER_TOKEN)) // [1]
Source: anomalyco/opencode, packages/core/src/util/token.ts (MIT)
Codex counts bytes and rounds up. saturating_add adds and, past the largest value the type can hold, stays there, so a huge length cannot wrap around to a small one.
const APPROX_BYTES_PER_TOKEN: usize = 4; // [2]
// ... (helpers that cut a long string down to a head and a tail)
pub fn approx_token_count(text: &str) -> usize {
let len = text.len(); // [3]
len.saturating_add(APPROX_BYTES_PER_TOKEN.saturating_sub(1)) / APPROX_BYTES_PER_TOKEN // [4]
}
Source: openai/codex, codex-rs/utils/string/src/truncate.rs (Apache-2.0)
pi divides by 3.5 and walks one Message, a single transcript entry, by role.
const CHARS_PER_TOKEN = 3.5; // [5]
const ESTIMATED_IMAGE_CHARS = 4800; // [6]
// ... (estimateTextTokens rounds up: Math.ceil(length / 3.5))
export function estimateMessageTokens(message: Message): number {
let chars = 0;
if (message.role === "system") {
return (
estimateTextTokens(getSystemMessageText(message)) +
estimateToolsTokens(message.toolsAdded) + // [7]
estimateToolsTokens(message.toolsRemoved)
);
}
if (message.role === "user") return estimateTextAndImageContentTokens(message.content);
if (message.role === "toolResult") return estimateTextAndImageContentTokens(message.content);
for (const block of message.content) {
if (block.type === "text") {
chars += block.text.length;
} else if (block.type === "thinking") {
chars += block.thinking.length; // [8]
} else {
chars += block.name.length + safeJsonStringify(block.arguments).length; // [9]
}
}
return Math.ceil(chars / CHARS_PER_TOKEN);
}
Source: earendil-works/pi, packages/ai/src/utils/estimate.ts (MIT)
What to notice
- [1]
Math.roundsends a half up: 6 characters become 2 tokens, and your//gives 1. One character rounds to 0. Yours returns 1. - [2] The unit is bytes. The next function,
approx_bytes_for_tokens(lines 92-94, past this range), multiplies a token budget by 4 so tool-output truncation knows how many bytes to keep. - [3]
text.len()is bytes.éis two bytes, so other languages come out higher than a character count. - [4] Adding 3 before dividing by 4 rounds up: 5 bytes become 2 tokens. Your
//turns 5 characters into 1. - [5] 100 characters is 25 tokens at your divisor of 4, and
Math.ceil(100 / 3.5)is 29. - [6] The skipped helper adds 4,800 characters per image (1,372 tokens) and never inspects the pixels. Your counter only sees
content. - [7]
toolsAddedis the full definitions that become available.toolsRemovedis only the names that go away. Both are JSON-encoded and divided by 3.5. A tool schema outsidecontentadds nothing to your sum. An item with nocontentstill counts as 1, becauseestimate_tokens("")ismax(1, 0). - [8]
thinkingis reasoning text stored apart from the visible answer, and it counts. - [9] A tool call counts its name plus its arguments. A
content-only sum misses both.
Read it yourself: open estimateContextTokens in the same file. When an earlier assistant reply carries a real Usage, pi keeps that number and estimates only the messages after it.
Finished this idea?
Your learning progress stays in this browser.