Skip to slide
162 / 221

CH 14 · Cost, Latency, and Model Tiering · 01/14

Read as one page

Cost, Latency, and Model Tiering

Cost and latency compound because an agent loop makes several model calls per turn. Covers model tiering as the main lever.

14 slides · 5 min read

Contour map for Cost, Latency, and Model Tiering

In this chapter

  1. 01Where cost comes from
  2. 02Where latency comes from
  3. 03The master lever: model tiering
  4. 04Turn reasoning on and off per call
  5. 05Shrink and curate context
  6. 06Reduce loop iterations
  7. 07Parallelize independent work
  8. 08Cache aggressively
  9. 09Stream to mask latency
  10. 10Right-size output
  11. 11Measure before optimizing
  12. 12Metering and limits
  13. 13The cost/latency playbook

Agents can be expensive and slow if built naively, and both problems compound: an agent loop makes multiple model calls per turn, each potentially large. Cost and latency are not afterthoughts to optimize later; they're design constraints that shape your architecture. This chapter covers where the costs and delays come from and the levers that control them.

Finished this idea?

Your learning progress stays in this browser.