Skip to slide
037 / 170

CH 04 · Tokens, Context Windows, and the Quadratic Problem · 02/13

Read as one page

Real code: one model's limits and prices in LiteLLM

The toy you build later in this chapter hard-codes two things a real harness has to look up: how big the window is, and what each token costs. LiteLLM, a Python library that puts one interface in front of many model providers, keeps both for every model in a single JSON file. Here is its entry for Claude Sonnet 4.5. Watch which numbers describe the window and which describe money.

"claude-sonnet-4-5": {
    "cache_creation_input_token_cost": 3.75e-06,
    // ...
    "cache_read_input_token_cost": 3e-07,   // [1]
    "input_cost_per_token": 3e-06,          // [2]
    // ... (batch prices, deprecation date)
    "litellm_provider": "anthropic",
    "max_input_tokens": 200000,             // [3]
    "max_output_tokens": 64000,             // [4]
    "max_tokens": 64000,                    // [5]
    "mode": "chat",
    "output_cost_per_token": 1.5e-05,
    // ... (batch price, web search prices, feature flags)
    "supports_prompt_caching": true,
    // ...
    "prompt_cache_min_tokens": 1024,
    "source": "https://platform.claude.com/docs/en/about-claude/pricing"
},

Source: BerriAI/litellm, model_prices_and_context_window.json (MIT)

What to notice

  1. [1] Every price is in dollars per single token, so the numbers look tiny. Multiply by a million to get the usual form: 3e-07 is $0.30 per million tokens read from the prompt cache (Chapter 5 covers the cache itself).
  2. [2] 3e-06 is $3 per million fresh input tokens. A cached token costs one tenth of that, the same ratio as the toy's cached_rate = 0.1. Further down, output_cost_per_token is $15 per million, five times the input price, which is why the toy makes output_rate bigger than input_rate.
  3. [3] max_input_tokens is 200,000, the number LiteLLM stores as this model's input cap. It is not, by itself, Fact two's combined ceiling.
  4. [4] max_output_tokens is a separate 64,000. The file does not say whether those 64,000 come out of the 200,000. A later slide shows OpenCode fitting the reply into the room the prompt leaves.
  5. [5] The template marks max_tokens LEGACY: it copies max_output_tokens when the provider has one, and otherwise copies max_input_tokens. On this row it copies the 64,000 output cap, so reading it as the window misses the 200,000 input cap.

Read it yourself: open the same file at the sample_spec entry, the template at the top that documents every field, and read the three lines about max_input_tokens, max_output_tokens and max_tokens. Then open generic_cost_per_token in litellm/litellm_core_utils/llm_cost_calc/utils.py. It looks up this entry with get_model_info and returns two numbers. Find out what they are, and where cache-hit tokens get priced.

Finished this idea?

Your learning progress stays in this browser.