Skip to slide
044 / 170

CH 04 · Tokens, Context Windows, and the Quadratic Problem · 09/13

Read as one page

Real code: pi prices the provider's real usage

CostModel.turn_cost multiplies your own guess by arbitrary rates per thousand tokens. pi waits until the provider answers, then prices the counts the provider reports. Those counts sit on a Usage value attached to the assistant message.

export interface Usage {
  input: number;                                                          // [1]
  output: number;
  cacheRead: number;
  cacheWrite: number;
  /** Subset of `cacheWrite` written with 1h retention. Only Anthropic reports this split. */
  cacheWrite1h?: number;                                                  // [2]
  // ... (reasoning tokens are a subset of output, so they are already counted)
  reasoning?: number;
  totalTokens: number;
  cost: {                                                                 // [3]
    input: number;
    output: number;
    cacheRead: number;
    cacheWrite: number;
    total: number;
  };
}

Source: earendil-works/pi, packages/ai/src/types.ts (MIT)

calculateCost fills usage.cost from the model's price table. The rates are US dollars per million tokens, the same kind of numbers as the LiteLLM entry earlier, written per million instead of per token.

export function calculateCost(model: AnyModel, usage: Usage): Usage["cost"] {
  const inputTokens = usage.input + usage.cacheRead + usage.cacheWrite;  // [4]
  let rates: ModelCostRates = model.cost;
  let matchedThreshold = -1;
  for (const tier of model.cost.tiers ?? []) {                            // [5]
    if (inputTokens > tier.inputTokensAbove && tier.inputTokensAbove > matchedThreshold) {
      rates = tier;
      matchedThreshold = tier.inputTokensAbove;
    }
  }

  // Anthropic charges 2x base input for 1h cache writes.
  const longWrite = usage.cacheWrite1h ?? 0;                              // [6]
  const shortWrite = usage.cacheWrite - longWrite;
  usage.cost.input = (rates.input / 1000000) * usage.input;               // [7]
  usage.cost.output = (rates.output / 1000000) * usage.output;
  usage.cost.cacheRead = (rates.cacheRead / 1000000) * usage.cacheRead;
  usage.cost.cacheWrite = (rates.cacheWrite * shortWrite + rates.input * 2 * longWrite) / 1000000;
  usage.cost.total = usage.cost.input + usage.cost.output + usage.cost.cacheRead + usage.cost.cacheWrite;
  return usage.cost;
}

Source: earendil-works/pi, packages/ai/src/models.ts (MIT)

What to notice

  1. [1] input is fresh tokens, output is tokens written, cacheRead is a cache hit, cacheWrite is a cache fill. Your turn_cost has no write price, so a cache fill looks free.
  2. [2] cacheWrite1h is the part of cacheWrite stored for one hour. Anthropic reports that split. reasoning is already inside output, so leave it out of the sum.
  3. [3] cost is written here, in dollars. totalTokens is the provider's own total when it sends one.
  4. [4] The tier check adds all three input kinds. A cached token still counts toward how big the prompt was.
  5. [5] inputTokensAbove is where a higher price starts. The highest step the prompt has passed sets the rates for the whole request. The price schema calls that request-wide.
  6. [6] One-hour writes are billed at twice the base input rate. ?? 0 treats a missing split as none. Your single cached_rate cannot say this.
  7. [7] Dividing by one million turns a per-million rate into a per-token rate. Your toy divides by one thousand, and its rates are arbitrary units.

The Anthropic adapter fills these fields from message_start, adds totalTokens itself (its comment says Anthropic provides none), then calls calculateCost.

Read it yourself: open anthropic-messages.ts at message_start and write down which provider field lands in which Usage field. Then search for message_delta in the same function, about 130 lines later. That branch writes a field only when the event includes it. A missing input_tokens stored as 0 would wipe the count from message_start.

Finished this idea?

Your learning progress stays in this browser.