CH 04 · Tokens, Context Windows, and the Quadratic Problem · 10/13
Read as one pageReal code: OpenCode shrinks the reply to fit
Your estimator returns a number and stops. It never asks whether that number plus the reply still fits in the window. OpenCode asks, in outputLimit, before it builds the request. limit comes from the model catalog: limit.context is the window, limit.output is the longest reply that model is allowed to write.
// Used when the catalog has no output limit for the model.
const OUTPUT_TOKEN_FALLBACK = 32_000
// No reply needs more, however much the model allows.
const OUTPUT_TOKEN_MAX = 256_000
// ... (a summary is capped at 32,000 tokens)
// Prompt text is estimated at about 4 characters per token, which can run low on dense text such as code.
const ESTIMATE_ERROR = 0.15 // [1]
// Never ask for less; only reachable with automatic compaction off, since it keeps the window from filling this far.
const OUTPUT_TOKEN_MIN = 1_024 // [2]
// ... (request types)
/** The default output limit: the catalog limit, fitted to the room the prompt leaves in the context window. */
export const outputLimit = (
limit: Model.Info["limit"],
kind: "primary" | "compaction",
inputTokens?: Input["inputTokens"],
) => {
const model = Math.min(limit.output > 0 ? limit.output : OUTPUT_TOKEN_FALLBACK, OUTPUT_TOKEN_MAX)
const requested = kind === "compaction" ? Math.min(model, SUMMARY_OUTPUT_MAX) : model // [3]
if (inputTokens === undefined || limit.context <= 0) return requested // [4]
const room = limit.context - inputTokens.measured - Math.ceil(inputTokens.estimated * (1 + ESTIMATE_ERROR)) // [5]
return Math.min(requested, Math.max(OUTPUT_TOKEN_MIN, room)) // [6]
}
Source: anomalyco/opencode, packages/core/src/session/model-request.ts (MIT)
What to notice
- [1] 0.15 is a pad. The comment above it says the 4-character guess can come out low on dense text such as code, so
estimatedis multiplied by 1.15. - [2] 1,024 is the floor applied to
roombefore the comparison withrequested. The comment says a full window is reachable when automatic compaction is off. - [3]
kindis"primary"for a normal turn and"compaction"for the summary call from Chapter 5. A summary is clamped to 32,000. Every reply is capped at 256,000, catalog or not. - [4] With no measurement, or
limit.contextof 0 or less, OpenCode skips the subtraction and returnsrequestedfrom [3]: still capped at 256,000, and at 32,000 for a summary. A catalog output of 1,000,000 becomes 256,000. - [5]
measuredsums the latest usable reply from this provider: non-cached input, cache read, cache write, visible output, and reasoning. An errored reply, or another provider's count, is skipped.estimatedis the messages after that reply. Text usesToken.estimate. Images useestimateMedia. System text and tool definitions stay out ofestimatedonce that reply exists, because they already sit in the input count. With a window of 200,000,measured150,000, andestimated10,000, room is200000 - 150000 - ceil(10000 * 1.15)= 38,500. (Those three figures are made up.) - [6] The budget is the smaller of
requestedand the room, and the room is raised to 1,024 first. A catalog limit already under 1,024 stays under it. Room 0 against a 64,000 catalog still asks for 1,024. Compaction is what cuts the history.
runStep passes estimatePrompt(loaded) as inputTokens on the primary call.
Read it yourself: open clampMaxTokensToContext in pi. It subtracts a flat 4,096 and, when nothing is left, asks for 1 output token. Then find inputTokens in OpenCode's runStep (llm.ts). estimatePrompt in compaction.ts is what splits it into measured and estimated.
Finished this idea?
Your learning progress stays in this browser.