CH 14 · Cost, Latency, and Model Tiering · 07/14
Read as one pageReduce loop iterations
Fewer iterations means fewer calls means less cost and latency:
- Batching tools. "Fetch these N documents" in one call avoids N serial iterations. "Extract all columns for this document in one call" instead of one call per cell.
- Good tool design and descriptions so the model picks the right tool first time instead of fumbling through several.
- Cheap discovery tools (list/find) so the model targets what it needs instead of reading everything and iterating.
Finished this idea?
Your learning progress stays in this browser.