Skip to slide
168 / 221

CH 14 · Cost, Latency, and Model Tiering · 07/14

Read as one page

Reduce loop iterations

Fewer iterations means fewer calls means less cost and latency:

  • Batching tools. "Fetch these N documents" in one call avoids N serial iterations. "Extract all columns for this document in one call" instead of one call per cell.
  • Good tool design and descriptions so the model picks the right tool first time instead of fumbling through several.
  • Cheap discovery tools (list/find) so the model targets what it needs instead of reading everything and iterating.

Finished this idea?

Your learning progress stays in this browser.