CH 14 · Cost, Latency, and Model Tiering · 01/14
Read as one pageCost, Latency, and Model Tiering
Cost and latency compound because an agent loop makes several model calls per turn. Covers model tiering as the main lever.
14 slides · 5 min read
In this chapter
- 01Where cost comes from
- 02Where latency comes from
- 03The master lever: model tiering
- 04Turn reasoning on and off per call
- 05Shrink and curate context
- 06Reduce loop iterations
- 07Parallelize independent work
- 08Cache aggressively
- 09Stream to mask latency
- 10Right-size output
- 11Measure before optimizing
- 12Metering and limits
- 13The cost/latency playbook
Agents can be expensive and slow if built naively, and both problems compound: an agent loop makes multiple model calls per turn, each potentially large. Cost and latency are not afterthoughts to optimize later; they're design constraints that shape your architecture. This chapter covers where the costs and delays come from and the levers that control them.
Finished this idea?
Your learning progress stays in this browser.