Skip to slide
163 / 221

CH 14 · Cost, Latency, and Model Tiering · 02/14

Read as one page

Where cost comes from

LLM cost is driven by tokens: input (everything you send) and output (everything the model generates), usually priced separately with output more expensive. For an agent, the multipliers are:

  • Loop iterations. Each tool round-trip is another model call that re-sends accumulated context. A 5-iteration turn can cost several times a single call.
  • Context size. Every token of context is paid for on every call. Bloated prompts and dumped tool results are paid for repeatedly across the loop.
  • Model choice. Top-tier models can cost 10–50× a small model per token. Using a flagship model for a trivial task is the most common waste.
  • Reasoning tokens. Thinking/reasoning generates extra (billed) tokens. Valuable when it improves a hard answer; pure waste on a bulk job.

Finished this idea?

Your learning progress stays in this browser.