Skip to slide
035 / 221

CH 03 · Tool Design · 08/18

Read as one page

Batching

When the model commonly needs N of something, give it a tool that does N at once rather than forcing N separate calls. "Fetch these documents" (a list) beats N× "fetch this document." Batching cuts round-trips, which cuts latency and loop iterations. Cap the batch size in the schema to prevent abuse.

Finished this idea?

Your learning progress stays in this browser.