Chapter 01 · 14 min read · Updated
Anatomy of an AI Agent
An AI agent has six parts: the model, the loop, tools, context, memory, and the surrounding system. How each one behaves and how they interact in one turn.
Prefer one idea per screen? Read it as 12 slides →
In this chapter11
- 01.1The spectrum: prompt → chatbot → agent
- 01.2The six parts
- 01.3Real code: Hermes Agent assembles all six parts in one function
- 01.4Real code: Hermes keeps long-term memory in two small files
- 01.5How the parts interact in one turn
- 01.6Real code: OpenClaw checks the caller before any agent work
- 01.7Real code: one smolagents step, from memory to model and back
- 01.8A crucial reframing: the model is the easy part
- 01.9Real code: in pi, the model call is one line of a long file
- 01.10Two stances toward the model
- 01.11Read the source yourself
Before you can design an agent, you need a clear mental model of its parts. This chapter lays out that anatomy and the vocabulary the rest of the folder uses. Every term here recurs throughout, so it's worth getting precise.
01.1
The spectrum: prompt → chatbot → agent
It helps to place "agent" on a spectrum:
- A single prompt takes input and returns output once. No memory, no actions. (A summariser, a classifier.)
- A chatbot holds a conversation: it has history, but it still only talks. It can't do anything in the world.
- An agent is a model in a loop that can take actions through tools, observe their results, and decide its next step, repeating until a task is done. It can read a file, call an API, write a document, and chain these together.
The defining feature of an agent is the loop with tools. Everything else in this folder elaborates on the pieces that make that loop useful, safe, and reliable.
01.2
The six parts
A production agent has six conceptual parts. Hold all six in your head and most design questions become "which part does this belong to?"
1. The model
The LLM itself, the reasoning engine. You usually don't build this; you call it via a provider API. Key properties you'll design around: its context window (how much it can read at once), whether it supports tool/function calling, whether it can stream, and whether it exposes a reasoning/thinking mode. Treat the model as a powerful but stateless, occasionally-unreliable component you orchestrate, not as the system itself.
2. The loop (orchestrator)
The control structure that drives the model: call the model, see if it asked for tools, run them, feed results back, repeat until it produces a final answer. This is the agent's "main function." It owns iteration limits, streaming, and the bridge between the model's requests and your code's actions. (Chapter 2.)
3. The tools
The actions the agent can take, each described to the model by a schema (name, description, parameters). A tool might read a document, search a database, send an email, or generate a file. Tools are how the agent affects the world and gathers information beyond its training. The set of tools defines the agent's capabilities. (Chapter 3.)
4. The context
Everything assembled and sent to the model on a given call: the system prompt, the conversation history, the available tools, and any injected information (available documents, retrieved snippets, prior-turn summaries). Context is constructed fresh for each call. This is the most underappreciated part and the one that most determines quality. (Chapter 5.)
5. The memory / state
What persists across calls and across sessions: conversation history, generated artifacts, the status of long-running work, user preferences. The model is stateless between API calls; memory is how continuity is created. It usually lives in a database and is selectively loaded into context. (Chapters 5, 9.)
6. The surrounding system
Everything that makes the agent a real product rather than a demo: authentication and authorization, storage, streaming transport, secrets management, rate limiting, observability, error handling, and the data model. This is the bulk of the engineering, and it's where agents succeed or fail in production. (Chapters 8–18.)
01.3
Real code: Hermes Agent assembles all six parts in one function
Hermes Agent, a large Python agent from Nous Research, builds an AIAgent by sending its constructor arguments to one function, init_agent. AIAgent.__init__ forwards those arguments except the deprecated tool_delay, which it ignores after a warning. Read init_agent top to bottom and you can point at the line that installs each part: the iteration budget the loop will consult, the model client, the tool list, the session store, memory, and the context engine. The loop itself is in another file, and several of these helpers are not named after the part.
def init_agent(
agent, base_url: str | None = None, api_key: str | None = None, provider: str | None = None, api_mode: str | None = None,
# ... (37 lines: the rest of the keyword parameters, from UI callbacks to chat identity)
):
# ... (16 lines: safe-worker overrides; copy plain settings onto the agent)
agent.iteration_budget = iteration_budget or IterationBudget(max_iterations) # [1]
# ... (28 lines: provider name, capabilities, credential pool)
_resolve_api_mode(agent, api_mode, provider_name, base_url) # [2]
_finalize_routing(agent, api_mode, credential_pool)
# ... (22 lines: UI callbacks, request overrides, prompt-cache settings)
_setup_logging(agent)
_set_defaults(agent, _STREAM_STATE)
_build_client(agent, api_key, base_url, fallback_model) # [3]
_init_fallback_chain(agent, fallback_model)
_load_tools(agent, enabled_toolsets, disabled_toolsets) # [4]
_init_session_state( # [5]
agent, session_id, session_db, parent_session_id, reasoning_config, max_tokens,
checkpoints_enabled, checkpoint_max_snapshots, checkpoint_max_total_size_mb, checkpoint_max_file_size_mb,
)
# ... (9 lines: read the config file once; apply display settings)
_init_memory(agent, _agent_cfg, skip_memory, platform, memory_manager=memory_manager) # [6]
# ... (5 lines: parse compression settings; work out the model's context length)
_build_context_engine(agent, _agent_cfg, cs, _custom_providers, _effective_context_length, session_db) # [7]
Source: NousResearch/hermes-agent, agent/agent_init.py (MIT)
What to notice
- [1] The loop's limit.
IterationBudgetis a thread-safe counter of loop iterations, not of raw model calls. A turn whose only tool wasexecute_code(the model writes a program that calls tools) is refunded, and so is an iteration that gets restarted. On a normal iteration,begin_iterationcallsconsume()before the model request. If that returnsFalse, the turn stops with reasonbudget_exhausted. If_budget_grace_callis set, that iteration skipsconsume()and clears the flag, so the model gets one more call after the budget is spent. The parameter default formax_iterationsissys.maxsize. The terminal UI passes a configured cap instead, 500 when the config omits one. - [2] The model, part one.
_resolve_api_modewalks an ordered list of rules, first match wins, and storesapi_mode, the request format this provider speaks, such aschat_completionsoranthropic_messages._finalize_routing, the next call, can still change that choice, including upgradingchat_completionstocodex_responses(OpenAI's Responses API) when the caller did not set a format. - [3] The model, part two.
_build_clientcreates the client for that format._init_fallback_chainstores an ordered list of backup models, tried when the primary one is exhausted. - [4] The tools.
_load_toolsasks the tool registry for definitions, filtered by enabled and disabled toolsets (named groups of tools), and keeps them onagent.tools. - [5] The surrounding system. A session ID, an optional SQLite session store, and file checkpoints. Logging was set up a few lines earlier, before the model client exists.
- [6] The memory. File-based memory, plus an external memory plugin if one is configured.
- [7] The context. The context engine compacts the conversation once input crosses a configured fraction of the window. The default fraction is 50 percent, and a window under 512,000 tokens is raised to at least 75 percent.
Read it yourself: in run_agent.py, name the argument AIAgent.__init__ drops. In _init_memory, if memory fails to load, does the agent still start? In turn_iteration_prep.py, which iteration skips consume()?
01.4
Real code: Hermes keeps long-term memory in two small files
The memory part persists across sessions and is selectively loaded into context. Hermes's built-in version, MemoryStore, is the object _init_memory creates in the previous slide, and it is deliberately small: two Markdown files, a size budget for each, and a frozen copy for the system prompt. Watch the difference between what is on disk and what the model sees.
class MemoryStore:
"""Bounded curated memory with file persistence; one instance per AIAgent.
``_system_prompt_snapshot`` is frozen at load time (prefix-cache stable);
``memory_entries`` / ``user_entries`` are live state persisted to disk."""
# ... (7 lines: a cap on failed memory writes per turn)
def __init__(self, memory_char_limit: int = 2200, user_char_limit: int = 1375, *, # [1]
memory_enabled: bool = True, user_profile_enabled: bool = True):
self.memory_entries: list[str] = [] # [2]
self.user_entries: list[str] = []
self.memory_char_limit, self.user_char_limit = memory_char_limit, user_char_limit
self.memory_enabled, self.user_profile_enabled = memory_enabled, user_profile_enabled
self._system_prompt_snapshot: dict[str, str] = {"memory": "", "user": ""} # [3]
# ... (28 lines: the per-turn failure counter)
def load_from_disk(self):
# ... (16 lines: docstring; a helper that blocks entries matching a threat pattern)
for target in ("memory", "user"):
path = self._path_for(target) # [4]
# ... (3 lines: make sure the memories folder exists)
# Deduplicate (order-preserving, first occurrence wins).
entries = list(dict.fromkeys(self._read_file(path)))
self._set_entries(target, entries)
# ... (7 lines: warn, never truncate, when a file is over its limit)
self._system_prompt_snapshot[target] = self._render_block(target, [_sanitize(e, path.name) for e in entries]) # [5]
Source: NousResearch/hermes-agent, tools/memory_tool_store.py (MIT)
What to notice
- [1] The budgets are in characters, not tokens: 2,200 for the agent's own notes and 1,375 for the user profile by default. A character count does not depend on any model's tokenizer, so the cap means the same thing after you switch models. When a new entry would go over,
addrefuses it and tells the model to remove or shorten stale entries in the same batch as the new one. - [2] The live state is two plain lists of strings, one remembered fact per entry. On disk they become
MEMORY.mdandUSER.md, entries separated by a§line. - [3]
_system_prompt_snapshotis a second copy, already rendered as text.format_for_system_prompthands the prompt builder this snapshot, never the live lists. - [4]
_path_forpoints into amemoriesfolder inside Hermes's home directory. - [5] The snapshot is rendered at load from those entries, after
_sanitizesubstitutes a[BLOCKED: ...]note for any entry that matches a prompt-injection or data-exfiltration pattern. The live lists and the file keep the original text, so you can still see and delete it.
Why freeze it? Providers cache the start of a prompt: when a request begins with exactly the same bytes as an earlier one, that part is cheaper and faster. A fact saved mid-session would break that cache on every later call. Writes go to disk at once and leave the snapshot alone. A saved fact reaches the prompt the next time the snapshot is rebuilt: at startup, after compression, on a new session or a session switch, and on undo. Pasting the whole live file into every call instead grows without limit, breaks the cache on every save, and carries one poisoned entry into every future session.
Read it yourself: in tools/memory_tool_store.py, read _mutate and add. What does the store do when MEMORY.md exists but cannot be read, and why is "treat it as empty" the dangerous choice? Then search agent/system_prompt.py for load_from_disk and read invalidate_system_prompt: when does a saved fact finally reach the prompt?
01.5
How the parts interact in one turn
Walk through a single user request to see the parts cooperate:
- The request arrives. The surrounding system authenticates the user and authorizes what they can touch.
- The orchestrator assembles context: it loads relevant memory (history, available artifacts), builds the system prompt, and gathers the tools available on this surface.
- The orchestrator calls the model with that context.
- The model streams reasoning and text, and may request tools. The orchestrator runs each tool (these may touch the surrounding system: storage, external APIs), and feeds results back into the loop.
- The model produces a final answer. The orchestrator streams it to the user and writes the outcome into memory.
Every chapter in this folder is, in effect, a deep look at one of these steps.
01.6
Real code: OpenClaw checks the caller before any agent work
Step 1 of the turn belongs to the surrounding system: decide who is calling, before any model or prompt is involved. OpenClaw runs on your own machine. Its Control UI, CLI, and TUI (a terminal interface) connect to a local server it calls the Gateway, and this function is that server's shared token or password check. Chat channels such as Telegram and Slack are a different door: the Gateway connects out to them, and those senders do not present this secret.
async function authorizeSharedSecretAuth(params: {
method: "token" | "password";
configuredSecret?: string;
providedSecret?: string;
limiter?: AuthRateLimiter;
ip?: string;
rateLimitScope: string;
deferRateLimitFailure?: boolean;
resetOnSuccess?: boolean;
}): Promise<GatewayAuthResult> {
// ... (9 lines: fail when the gateway has no usable secret configured)
if (!params.providedSecret) {
// Missing credentials do not consume the wrong-credential rate limit.
return { ok: false, reason: `${params.method}_missing` }; // [1]
}
if (!safeEqualSecret(params.providedSecret, params.configuredSecret)) { // [2]
if (!params.deferRateLimitFailure) {
await params.limiter?.recordFailureAndDelay(params.ip, params.rateLimitScope); // [3]
}
return { ok: false, reason: `${params.method}_mismatch` };
}
if (params.resetOnSuccess !== false) {
params.limiter?.reset(params.ip, params.rateLimitScope); // [4]
}
return { ok: true, method: params.method }; // [5]
}
Source: openclaw/openclaw, src/gateway/auth.ts (MIT)
What to notice
- [1] A request with no credential is refused but not counted as a failed guess. A client that simply forgot its token should not lock itself out.
- [2]
safeEqualSecretcompares using Node'stimingSafeEqual, after padding both values to the same length. A plain===can return sooner when the first characters differ, and an attacker who times many responses can recover a secret one character at a time. The padding also keeps the secret's length from leaking. - [3] A wrong secret is recorded against the caller's address and a scope (a named counter, so token failures do not share a budget with device pairing). With the defaults in
auth-rate-limit.ts, 10 failures inside 60 seconds lock that address out for 5 minutes. Loopback callers (connections from this same machine to itself, such as127.0.0.1) are not locked out by default. Each of their failures waits longer instead, from 250 milliseconds, doubling up to 5 seconds. A client on the same machine that arrives through a LAN address is not exempt.deferRateLimitFailurepostpones the record for a caller that still has other credentials to try. - [4] When
resetOnSuccessis notfalse(the default), a correct secret clears that scope's failure history for that address. - [5] Every outcome is a returned object, not a thrown error. Failures include a
reasonsuch astoken_missingortoken_mismatch. Success is{ ok: true, method }, with noreason.
The check for an address that is already locked out runs just before this function, in authorizeGatewayConnectCore. The naive version, if (token !== config.token) return 401, leaks timing, lets anyone guess forever, and tells you nothing when a real user gets refused.
Read it yourself: in src/gateway/auth.ts, read authorizeGatewayConnectCore and write down the order of checks a request passes before its token is compared. Then open src/gateway/auth-rate-limit.ts and read the comment on recordFailureAndDelay: why does it record failures only after checking credentials?
01.7
Real code: one smolagents step, from memory to model and back
smolagents is Hugging Face's compact agent library. _step_stream, on its ToolCallingAgent, rebuilds the context, calls the model, runs the tools, and yields whether this step produced the final answer. The caller, _run_stream, appends the step to memory and streams the final answer to the user. The method yields a stream of events: token deltas (ChatMessageStreamDelta), each requested call (ToolCall), each result (ToolOutput), and one final ActionOutput that says whether the step finished the task. ChatMessage is the model's reply.
def _step_stream(
self, memory_step: ActionStep
) -> Generator[ChatMessageStreamDelta | ToolCall | ToolOutput | ActionOutput]:
# ... (5 lines: docstring)
memory_messages = self.write_memory_to_messages() # [1]
input_messages = memory_messages.copy()
# Add new step in logs
memory_step.model_input_messages = input_messages # [2]
try:
# ... (16 lines: if streaming is on, call generate_stream and show tokens live)
else:
chat_message: ChatMessage = self.model.generate( # [3]
input_messages,
stop_sequences=["Observation:", "Calling tools:"],
tools_to_call_from=self.tools_and_managed_agents,
)
# ... (6 lines: log the reply)
# Record model output
memory_step.model_output_message = chat_message # [4]
memory_step.model_output = chat_message.content
memory_step.token_usage = chat_message.token_usage
except Exception as e:
raise AgentGenerationError(f"Error while generating output:\n{e}", self.logger) from e
# ... (9 lines: parse tool calls out of the reply)
final_answer, got_final_answer = None, False
for output in self.process_tool_calls(chat_message, memory_step): # [5]
yield output
# ... (18 lines: accept at most one final_answer call per step)
yield ActionOutput( # [6]
output=final_answer,
is_final_answer=got_final_answer,
)
Source: huggingface/smolagents, src/smolagents/agents.py (Apache-2.0)
What to notice
- [1] Context, built fresh.
write_memory_to_messagesstarts from the system prompt and asks every saved step to render itself as chat messages. The agent never keeps one growingmessageslist; it rebuilds the list from its records on every call. - [2]
memory_stepis anActionStep, a small record (a Python dataclass) for this one step. The first thing stored in it is exactly what the model is about to see, so any step can be inspected later. - [3] The model call.
tools_to_call_fromsends the tool definitions, plus managed agents: other agents this one can call as if they were tools. The stop sequences ask a model that supports them to halt if it starts writingObservation:, the label reserved for real tool results. A model without that parameter gets the text cut out of the reply afterward. - [4] The reply and its token usage go onto the same record.
- [5] The tools.
process_tool_callsruns each requested tool (in parallel threads when there are several), passes each call and result up as it happens, then writes the calls and their observations (tool results as text) ontomemory_step. - [6] The step reports whether it produced the final answer. A malformed tool call raises
AgentParsingError._run_streamstores that onaction_step.errorand continues, so the next call shows the model the error.AgentGenerationError, the exception in this excerpt, means the model call itself failed. This file treats that as a bug in the agent, re-raises it, and does not seterror. The step is still appended, because thefinallyblock runs, and the run stops.
Why records instead of a message list? The same memory can be rendered more than one way: when the agent revises its plan, it calls write_memory_to_messages(summary_mode=True), which leaves out the system prompt and earlier plans. A plain append-only list cannot do that.
Read it yourself: open src/smolagents/memory.py and read ActionStep.to_messages. Which field becomes which message role, and what does a stored error turn into? Then find self.memory.steps.append(action_step) in _run_stream.
01.8
A crucial reframing: the model is the easy part
Newcomers assume the model is the system and the rest is glue. The opposite is closer to the truth. The provider gives you a smart model for a few cents a call. Your design work is everything around it:
- Context engineering (deciding what the model sees) is where most quality lives.
- Tool design (what actions exist and how they're described) is where most capability and reliability live.
- The surrounding system (auth, storage, streaming, state, observability) is where most of the engineering effort lives.
If you remember one thing from this chapter, make it this: you are not building a model; you are building the system that makes a model useful, safe, and reliable. The chapters ahead are a tour of that system, part by part.
01.9
Real code: in pi, the model call is one line of a long file
pi's packages/ai is a library that talks to many LLM providers behind one interface. This is its Anthropic adapter, a file of about 1,700 lines. The line that actually calls the model is marked [5]. Everything around it is the system this section describes: building the request, retrying, cancelling, and recovering when the provider rejects a request.
const requestOptions = {
...(options?.signal ? { signal: options.signal } : {}),
...(options?.timeoutMs !== undefined ? { timeout: options.timeoutMs } : {}),
maxRetries: 0, // [1]
};
let disablePreferredStrictTools = false;
let response: Response;
for (;;) {
options?.signal?.throwIfAborted();
const built = buildParams(model, normalizedContext, isOAuth, options, disablePreferredStrictTools); // [2]
if (built.providerThinkingLevel) output.providerThinkingLevel = built.providerThinkingLevel;
let params = built.params;
const nextParams = await options?.onPayload?.(params, model); // [3]
if (nextParams !== undefined) {
params = { ...(nextParams as MessageCreateParamsStreaming), stream: true };
}
try {
response = await retryProviderRequest( // [4]
() => client.beta.messages.create(params, requestOptions).asResponse(), // [5]
{
maxRetries: options?.maxRetries,
maxRetryDelayMs: options?.maxRetryDelayMs,
signal: options?.signal,
},
);
break;
} catch (error) {
// Only downgrade a rejected request, never an already-started stream.
// ... (15 lines: rethrow unless it is a 400 saying the strict tool schemas are too large)
disablePreferredStrictTools = true; // [6]
// ... (6 lines: record a diagnostic on the reply)
}
}
Source: earendil-works/pi, packages/ai/src/api/anthropic-messages.ts (MIT)
What to notice
- [1] The Anthropic SDK (the official client library) retries failed requests by default.
maxRetries: 0turns that off so retries happen in exactly one place, [4], instead of stacking. - [2]
buildParamsconverts pi's provider-neutral conversation,normalizedContext, into Anthropic's request shape: messages, tool definitions, system prompt, cache markers, thinking settings. - [3]
onPayloadis a hook that lets the caller inspect or replace the exact request body before it is sent, which is handy for logging or a last-minute override. - [4]
retryProviderRequestretries only errors it classifies as retryable, waits between attempts, and stops at once if the user cancelled.signalis anAbortSignal, JavaScript's standard way to cancel async work. Unless the caller asks for retries, it makes one attempt. - [5] The model call itself.
MessageCreateParamsStreamingis the Anthropic SDK's type for a streaming request. - [6] Strict tool schemas ask the provider to force tool arguments to match their schema exactly, and Anthropic rejects a request whose strict schemas are too many or too complex. If Anthropic returns that 400 and the request still has preferred-strict tools, pi sets the flag and loops, rebuilding so tools marked
strict: "prefer"are no longer strict. Other strict tools stay strict. Every other error is rethrown.
A rethrown error does not escape as a crash. The function's outer catch pushes an error event. stopReason is "aborted" if the caller cancelled, and "error" otherwise. pi's agent loop ends the turn on that message instead of throwing. A bare await client.messages.create(params) works in a demo and, in production, stacks a second retry policy, cannot be cancelled, and lets one provider quirk take the turn down.
Read it yourself: open packages/ai/src/api/openai-completions.ts and compare: is the same maxRetries: 0 plus retryProviderRequest wrapper there? Then read the StreamFunction type in packages/ai/src/types.ts. Chat adapters share that one signature, which is what makes the model a component you can swap. Image and classifier adapters in this folder use other types.
01.10
Two stances toward the model
Finally, two stances worth naming because they shape design:
- The model as a smart but unreliable collaborator. It will occasionally produce malformed output, hallucinate, or call a tool wrong. Design defensively: validate its outputs, give every tool call a defined result, and degrade gracefully. (This stance drives Chapter 13.)
- The model as a component you can swap. Today's best model won't be next quarter's. Decouple from any specific provider so you can change models without rewriting your agent. (This stance drives Chapter 4.)
Hold both stances and you'll make architecture decisions that age well.
01.11
Read the source yourself
The slides above quoted about a hundred lines from four codebases. Here is a short reading tour, in order, that takes each part further than a slide can.
- smolagents,
src/smolagents/memory.py. The whole memory model of a working agent in about 200 lines:AgentMemoryand its step records (TaskStep,ActionStep,PlanningStep,SystemPromptStep). Look for which steps return no messages in summary mode. The comment that says why a plan is dropped is inagents.py, just above thesummary_mode=Truecall. - Hermes Agent,
agent/turn_iteration_prep.py. The budget check in front of each iteration of the main tool loop. A turn incodex_app_servermode, which hands the whole turn to a subprocess, returns before that loop. Look for_budget_grace_call, then find where it is set inagent/turn_truncation.pyand work out why a fallback model needs one call past the budget. - pi,
packages/ai/src/api/. Provider adapters plus a few shared helpers. Openopenai-responses.tsandbedrock-converse-stream.tsnext to the Anthropic file and findexport const streamin each. List what differs (request shape, event parsing, whetherretryProviderRequestis used at all) and what stays identical (theStreamFunctionsignature). - OpenClaw,
src/gateway/auth-rate-limit.ts. The limiter behind the auth slide. It tracks at most 10,000 entries by default, one per scope plus address. Look for what it evicts when that table is full, and why it never evicts an active lockout.
Exercise: clone smolagents, build a ToolCallingAgent with any model you can access, and give it a task that needs two tool calls. After the run, print agent.write_memory_to_messages() and label every message with the ActionStep field it came from. Then pick one of the four repos and write a six-row table, one row each for the model, the loop, the tools, the context, memory, and the surrounding system. In each row, name the file and the function that implements that part.
0 of 12 slides completed
Your learning progress stays in this browser.
Self-check · 10 questions
01 / 10
What is the defining feature that distinguishes an agent from a chatbot?