Skip to slide
043 / 170

CH 04 · Tokens, Context Windows, and the Quadratic Problem · 08/13

Read as one page

Real code: three harnesses guess tokens from length

You just wrote estimate_tokens: characters divided by 4, rounded down, never less than 1. OpenCode (a TypeScript coding agent), Codex, and pi (a small toolkit; pi-ai is its provider library) guess the same way, with no vocabulary loaded. Watch the divisor, the rounding, and whatever is not plain text.

OpenCode's Token.estimate is the function the rest of its session code calls.

export * as Token from "./token.js"

const CHARS_PER_TOKEN = 4

export const estimate = (input: string) => Math.max(0, Math.round(input.length / CHARS_PER_TOKEN)) // [1]

Source: anomalyco/opencode, packages/core/src/util/token.ts (MIT)

Codex counts bytes and rounds up. saturating_add adds and, past the largest value the type can hold, stays there, so a huge length cannot wrap around to a small one.

const APPROX_BYTES_PER_TOKEN: usize = 4;                                            // [2]

// ... (helpers that cut a long string down to a head and a tail)

pub fn approx_token_count(text: &str) -> usize {
    let len = text.len();                                                           // [3]
    len.saturating_add(APPROX_BYTES_PER_TOKEN.saturating_sub(1)) / APPROX_BYTES_PER_TOKEN // [4]
}

Source: openai/codex, codex-rs/utils/string/src/truncate.rs (Apache-2.0)

pi divides by 3.5 and walks one Message, a single transcript entry, by role.

const CHARS_PER_TOKEN = 3.5;                                         // [5]
const ESTIMATED_IMAGE_CHARS = 4800;                                  // [6]

// ... (estimateTextTokens rounds up: Math.ceil(length / 3.5))

export function estimateMessageTokens(message: Message): number {
  let chars = 0;

  if (message.role === "system") {
    return (
      estimateTextTokens(getSystemMessageText(message)) +
      estimateToolsTokens(message.toolsAdded) +                      // [7]
      estimateToolsTokens(message.toolsRemoved)
    );
  }
  if (message.role === "user") return estimateTextAndImageContentTokens(message.content);
  if (message.role === "toolResult") return estimateTextAndImageContentTokens(message.content);

  for (const block of message.content) {
    if (block.type === "text") {
      chars += block.text.length;
    } else if (block.type === "thinking") {
      chars += block.thinking.length;                               // [8]
    } else {
      chars += block.name.length + safeJsonStringify(block.arguments).length; // [9]
    }
  }
  return Math.ceil(chars / CHARS_PER_TOKEN);
}

Source: earendil-works/pi, packages/ai/src/utils/estimate.ts (MIT)

What to notice

  1. [1] Math.round sends a half up: 6 characters become 2 tokens, and your // gives 1. One character rounds to 0. Yours returns 1.
  2. [2] The unit is bytes. The next function, approx_bytes_for_tokens (lines 92-94, past this range), multiplies a token budget by 4 so tool-output truncation knows how many bytes to keep.
  3. [3] text.len() is bytes. é is two bytes, so other languages come out higher than a character count.
  4. [4] Adding 3 before dividing by 4 rounds up: 5 bytes become 2 tokens. Your // turns 5 characters into 1.
  5. [5] 100 characters is 25 tokens at your divisor of 4, and Math.ceil(100 / 3.5) is 29.
  6. [6] The skipped helper adds 4,800 characters per image (1,372 tokens) and never inspects the pixels. Your counter only sees content.
  7. [7] toolsAdded is the full definitions that become available. toolsRemoved is only the names that go away. Both are JSON-encoded and divided by 3.5. A tool schema outside content adds nothing to your sum. An item with no content still counts as 1, because estimate_tokens("") is max(1, 0).
  8. [8] thinking is reasoning text stored apart from the visible answer, and it counts.
  9. [9] A tool call counts its name plus its arguments. A content-only sum misses both.

Read it yourself: open estimateContextTokens in the same file. When an earlier assistant reply carries a real Usage, pi keeps that number and estimates only the messages after it.

Finished this idea?

Your learning progress stays in this browser.