Skip to slide
045 / 170

CH 04 · Tokens, Context Windows, and the Quadratic Problem · 10/13

Read as one page

Real code: OpenCode shrinks the reply to fit

Your estimator returns a number and stops. It never asks whether that number plus the reply still fits in the window. OpenCode asks, in outputLimit, before it builds the request. limit comes from the model catalog: limit.context is the window, limit.output is the longest reply that model is allowed to write.

// Used when the catalog has no output limit for the model.
const OUTPUT_TOKEN_FALLBACK = 32_000
// No reply needs more, however much the model allows.
const OUTPUT_TOKEN_MAX = 256_000
// ... (a summary is capped at 32,000 tokens)
// Prompt text is estimated at about 4 characters per token, which can run low on dense text such as code.
const ESTIMATE_ERROR = 0.15                                              // [1]
// Never ask for less; only reachable with automatic compaction off, since it keeps the window from filling this far.
const OUTPUT_TOKEN_MIN = 1_024                                           // [2]

// ... (request types)

/** The default output limit: the catalog limit, fitted to the room the prompt leaves in the context window. */
export const outputLimit = (
  limit: Model.Info["limit"],
  kind: "primary" | "compaction",
  inputTokens?: Input["inputTokens"],
) => {
  const model = Math.min(limit.output > 0 ? limit.output : OUTPUT_TOKEN_FALLBACK, OUTPUT_TOKEN_MAX)
  const requested = kind === "compaction" ? Math.min(model, SUMMARY_OUTPUT_MAX) : model // [3]
  if (inputTokens === undefined || limit.context <= 0) return requested   // [4]
  const room = limit.context - inputTokens.measured - Math.ceil(inputTokens.estimated * (1 + ESTIMATE_ERROR)) // [5]
  return Math.min(requested, Math.max(OUTPUT_TOKEN_MIN, room))            // [6]
}

Source: anomalyco/opencode, packages/core/src/session/model-request.ts (MIT)

What to notice

  1. [1] 0.15 is a pad. The comment above it says the 4-character guess can come out low on dense text such as code, so estimated is multiplied by 1.15.
  2. [2] 1,024 is the floor applied to room before the comparison with requested. The comment says a full window is reachable when automatic compaction is off.
  3. [3] kind is "primary" for a normal turn and "compaction" for the summary call from Chapter 5. A summary is clamped to 32,000. Every reply is capped at 256,000, catalog or not.
  4. [4] With no measurement, or limit.context of 0 or less, OpenCode skips the subtraction and returns requested from [3]: still capped at 256,000, and at 32,000 for a summary. A catalog output of 1,000,000 becomes 256,000.
  5. [5] measured sums the latest usable reply from this provider: non-cached input, cache read, cache write, visible output, and reasoning. An errored reply, or another provider's count, is skipped. estimated is the messages after that reply. Text uses Token.estimate. Images use estimateMedia. System text and tool definitions stay out of estimated once that reply exists, because they already sit in the input count. With a window of 200,000, measured 150,000, and estimated 10,000, room is 200000 - 150000 - ceil(10000 * 1.15) = 38,500. (Those three figures are made up.)
  6. [6] The budget is the smaller of requested and the room, and the room is raised to 1,024 first. A catalog limit already under 1,024 stays under it. Room 0 against a 64,000 catalog still asks for 1,024. Compaction is what cuts the history.

runStep passes estimatePrompt(loaded) as inputTokens on the primary call.

Read it yourself: open clampMaxTokensToContext in pi. It subtracts a flat 4,096 and, when nothing is left, asks for 1 output token. Then find inputTokens in OpenCode's runStep (llm.ts). estimatePrompt in compaction.ts is what splits it into measured and estimated.

Finished this idea?

Your learning progress stays in this browser.