Skip to content
Agent YAP

What is it?

Inference

Inference is the act of actually using a trained model to produce an output, as opposed to training it. Every time you send a prompt to a model and get a response, that is one inference.

Inference cost is hugely important in practice because it is paid every single time anyone uses the model, across millions of users. This is why a smaller or more efficient model (Chapters 2, 4, and 5) is so valuable: it lowers the cost of every interaction.


From the Foundational Modelling glossary

Where Inference is explained

  1. How Memory Is Built and RetrievedAgentic Memory · How a production pipeline does write and manage: AgentCore
  2. The Agent LoopBuilding Coding Agents and Harnesses · Concept explanation
  3. Tokens, Context Windows, and the Quadratic ProblemBuilding Coding Agents and Harnesses · Concept explanation
  4. Model-Provider Abstraction and Multi-Model StrategyArchitecture and System Design · A model registry and validation
  5. Governing Evolving Memory (the SSGM Framework)Agentic Memory · The four design principles

Every term, A to Z