Skip to content
Agent YAP

What is it?

KV cache (key-value cache)

A performance optimization inside the model's serving system. As the model processes tokens, it stores intermediate results (the "keys" and "values") so it does not have to recompute them for tokens it has already seen. Reusing this cache makes responses cheaper and faster. The catch: the cache only helps if the start of the context stays stable. If you keep changing earlier parts of the context (for example, by swapping tool definitions around each turn), you "break the cache" and lose the savings. This is a major reason agents try to keep their context stable.

From the AI agent glossary

Where KV cache is explained

  1. The Four Moves: Reduce, Offload, Isolate, RetrieveContext Engineering · Move 3: Isolate context
  2. Prompt Caching and Context ManagementBuilding Coding Agents and Harnesses · Prompt caching: paying once for the stable part
  3. How Context Goes Wrong: Rot, Pollution, and ConfusionContext Engineering · Context pollution: clutter that distracts

Every term, A to Z