What is it?
Inference
Inference is the act of actually using a trained model to produce an output, as opposed to training it. Every time you send a prompt to a model and get a response, that is one inference.
Inference cost is hugely important in practice because it is paid every single time anyone uses the model, across millions of users. This is why a smaller or more efficient model (Chapters 2, 4, and 5) is so valuable: it lowers the cost of every interaction.
From the Foundational Modelling glossary
Where Inference is explained
- How Memory Is Built and RetrievedAgentic Memory · How a production pipeline does write and manage: AgentCore
- The Agent LoopBuilding Coding Agents and Harnesses · Concept explanation
- Tokens, Context Windows, and the Quadratic ProblemBuilding Coding Agents and Harnesses · Concept explanation
- Model-Provider Abstraction and Multi-Model StrategyArchitecture and System Design · A model registry and validation
- Governing Evolving Memory (the SSGM Framework)Agentic Memory · The four design principles