Chapter 04 · 14 min read · Updated
Model-Provider Abstraction and Multi-Model Strategy
Decoupling an agent from one model provider takes an adapter pattern and a validated registry. Covers tiering models by task and cost.
Prefer one idea per screen? Read it as 15 slides →
In this chapter14
- 04.1Why decouple
- 04.2What makes providers hard to abstract
- 04.3Real code: pi rewrites a turn from another model
- 04.4The adapter pattern
- 04.5Real code: one interface every provider implements
- 04.6Keep the adapter thin and domain-free
- 04.7A model registry and validation
- 04.8Real code: LiteLLM reads the provider from the name
- 04.9Model tiering: the multi-model strategy
- 04.10Real code: a small model writes the chat title
- 04.11Reasoning/thinking as a per-call decision
- 04.12Real code: Anthropic maps one reasoning level
- 04.13What you gain
- 04.14Read the source yourself
The model you build on today will not be the best model in six months, and the best model for one task is rarely the best for another. Both facts point to the same architectural decision: decouple your agent from any specific model provider. This chapter explains why, how to do it cleanly, and how to route work intelligently across model tiers.
04.1
Why decouple
Three forces make provider abstraction worth the effort even if you launch with a single model:
- Models change constantly. New versions ship monthly; prices fall; capabilities shift. If switching models means rewriting your agent, you're stuck on yesterday's model.
- Different tasks want different models. A complex drafting task wants a top-tier model; generating a chat title wants the cheapest one. Bulk extraction across a thousand documents wants a fast mid-tier model. You can't serve all of these well with one hard-wired choice.
- Risk and economics. A second provider is insurance against an outage, a price hike, or a policy change at any one vendor. And letting users bring their own provider key (Chapter 12) requires supporting whichever provider they chose.
04.2
What makes providers hard to abstract
Providers disagree at the wire level about nearly everything:
- Tool-schema format: each wants its own shape, and some reject edge cases (an empty parameter object) that others accept.
- Message format: content blocks vs. parts vs. input items; some require you to replay the model's prior turn verbatim, sometimes including opaque signatures that must be echoed exactly.
- Streaming events: different event vocabularies for text, reasoning, and tool calls.
- Reasoning/thinking: different switches and different ways the reasoning is surfaced.
- History chaining: some resend full history; some use a server-side "previous response id."
If any of this leaks into your agent loop, every provider-specific quirk becomes load-bearing and a fourth provider means a rewrite.
04.3
Real code: pi rewrites a turn from another model
Those disagreements sit in the history, so a turn from another provider, api, or model id cannot be sent on unchanged. pi repairs that in transformMessages (packages/ai). The Anthropic, Google, OpenAI, Bedrock, and Mistral adapters each call it before building a request. An AssistantMessage is one assistant turn: provider, api, model id, and content blocks. The api is the wire protocol, such as anthropic-messages. model is the model about to be called.
if (msg.role === "assistant") {
const assistantMsg = msg as AssistantMessage;
const isSameModel = // [1]
assistantMsg.provider === model.provider &&
assistantMsg.api === model.api &&
assistantMsg.model === model.id;
const transformedContent = assistantMsg.content.flatMap((block) => {
if (block.type === "thinking") {
// ... (3 lines: why thinking from another model is dropped)
if (!isSameModel) return []; // [2]
// Keep redacted and signed blocks even if the thinking text is empty
// (OpenAI encrypted reasoning); they are needed for replay.
if (block.redacted || block.thinkingSignature) return block; // [3]
// ... (2 lines: drop unsigned thinking whose text is empty)
}
// ... (7 lines: text from another model keeps the words and drops textSignature)
if (block.type === "toolCall") {
const toolCall = block as ToolCall;
let normalizedToolCall: ToolCall = toolCall;
if (!isSameModel && toolCall.thoughtSignature) { // [4]
normalizedToolCall = { ...toolCall };
delete (normalizedToolCall as { thoughtSignature?: string }).thoughtSignature;
}
if (!isSameModel && normalizeToolCallId) { // [5]
const normalizedId = normalizeToolCallId(toolCall.id, model, assistantMsg);
if (normalizedId !== toolCall.id) {
toolCallIdMap.set(toolCall.id, normalizedId);
normalizedToolCall = { ...normalizedToolCall, id: normalizedId };
}
}
return normalizedToolCall;
Source: earendil-works/pi, packages/ai/src/api/transform-messages.ts (MIT)
What to notice
- [1]
isSameModelis true only when provider, api, and model id all match. Checking the id alone would replay a turn from a different provider, or from a different api on the same provider. - [2] A thinking block is the model's reasoning, stored apart from the reply. When the provider, api, or model id differs,
transformMessagesremoves that block. The cut comment says the signature is valid only for the model that produced it, and that readable thinking sent as ordinary text would be treated as part of the reply. - [3] On the same model, a block is kept when it is
redactedor carries athinkingSignature, even when the thinking text is empty.thinkingSignatureis opaque replay data, andredactedmeans safety filters removed the text and left an encrypted payload in that signature for the next request. The comment on the kept line names OpenAI encrypted reasoning. The cut lines drop empty unsigned thinking, and they droptextSignaturefrom another model's text. - [4]
thoughtSignaturesits on the tool call. TheToolCalltype calls it Google-specific: an opaque signature for reusing thought context. When the model changed,transformMessagescopies the call and deletes the field, because leaving the signature in place would attach one model's thought context to another model's request. - [5]
normalizeToolCallIdruns only when the model changed, so a same-model conversation keeps its ids. The comment abovetransformMessagessays OpenAI Responses ids run to 450 characters or more and can contain|, while Anthropic requires^[a-zA-Z0-9_-]+$and at most 64 characters. Anthropic's function replaces each character outside letters, digits,_, and-with_, then slices to 64. The new id goes intotoolCallIdMap, and a later tool result rewrites itstoolCallIdfrom that map.
Read it yourself: Read closePendingToolCalls. It inserts an error tool result, text No result provided, when a call has no result. Then apply normalizeToolCallId on paper to | plus 80 a characters. You should get _ plus 63 a characters.
04.4
The adapter pattern
The clean solution is a provider adapter layer: define one neutral interface that the rest of your system speaks, and write a thin adapter per provider that translates to and from the native API. Concretely:
- Neutral types. Define your own minimal vocabulary: a message (
{role, content}), a tool schema (pick one canonical format, most teams use the widely-documented function-calling shape), a normalized tool call ({id, name, input}), a normalized tool result ({id, content}), and a set of streaming callbacks (onText,onReasoning,onToolCall). - A dispatcher. One function,
streamChatWithTools(params), that inspects the requested model id, picks the provider, and calls the right adapter. The rest of your code calls only this. - One adapter per provider. Each implements the same contract: convert neutral input to native, run the streaming tool-call loop, convert native streaming back to your neutral callbacks, and return the accumulated result. The tool-call replay loop (append model turn, append results, repeat) lives inside the adapter, so the provider-specific replay rules are encapsulated.
- Schema converters. Small functions that turn your canonical tool schema into each provider's dialect, handling the edge cases (omit empty parameter objects, ensure arrays have item types, etc.).
The payoff: your agent loop calls exactly one function and never branches on provider. Adding a provider is one new adapter file plus registering its model ids. Switching a user from one model to another is a single field change.
04.5
Real code: one interface every provider implements
The adapter pattern is a contract the rest of the program speaks, plus one translator per provider. The Vercel AI SDK writes that contract as the type LanguageModelV4, in the package @ai-sdk/provider. AnthropicLanguageModel, in @ai-sdk/anthropic, is declared implements LanguageModelV4 and sets specificationVersion to 'v4'. The OpenAI and Google language-model classes do the same. Application code talks to the SDK. The SDK calls doGenerate or doStream.
export type LanguageModelV4 = {
/**
* The language model must specify which language model interface version it implements.
*/
readonly specificationVersion: 'v4'; // [1]
// ... (3 lines: doc comment, this string is the provider id)
readonly provider: string;
// ... (3 lines: doc comment, this string is the provider's model id)
readonly modelId: string;
// ... (15 lines: supportedUrls, file URLs the model can read without the SDK downloading them)
// ... (6 lines: doc comment, the "do" prefix so application code does not call this directly)
doGenerate( // [2]
options: LanguageModelV4CallOptions,
): PromiseLike<LanguageModelV4GenerateResult>;
// ... (8 lines: the same naming note, and a note that the result is a stream of output parts)
doStream( // [3]
options: LanguageModelV4CallOptions,
): PromiseLike<LanguageModelV4StreamResult>;
};
Source: vercel/ai, packages/provider/src/language-model/v4/language-model-v4.ts (Apache-2.0)
What to notice
- [1]
specificationVersionis the interface generation this object implements.resolveLanguageModel, inpackages/ai/src/model/resolve-model.ts, accepts'v4','v3', and'v2'. Any other value, including a missing one, throwsUnsupportedModelVersionError. A v2 or v3 object is passed throughasLanguageModelV4before the rest of the call uses it. - [2]
provideris the provider id,anthropic.messagesby default, andmodelIdis that provider's model id. The API key travels in the request headers.doGenerateis the one-shot call, and its argument,LanguageModelV4CallOptions, holds the shared bag: prompt, tools, temperature, stop sequences, and areasoninglevel. The call options have no field for a document or a citation. A citation can still come back on atext-endstream part, as part of the response.PromiseLikemeans the caller can await the result. The removed comment says thedoprefix exists so application code does not call the method by accident. - [3]
doStreamreturns a stream of parts under one set of names. Reply text is atext-delta, reasoning is areasoning-delta, and a tool call being assembled is atool-input-delta. Anthropic'sdoStreamreceives the API eventthinking_deltaand enqueues areasoning-deltawhosedeltaisvalue.delta.thinking. The OpenAI responses language model and the Google language model enqueuereasoning-deltatoo. Switching on each provider's own event names would put the wire format back in the agent loop.
Read it yourself: Find resolvedModel.doStream in stream-language-model-call.ts. The call spreads callSettings (the rest of that function's arguments; the comment documents reasoning as one of them) and passes prompt, tools, and providerOptions. Then read providerOptions in language-model-v4-call-options.ts. A setting the shared fields cannot express goes there, and stays inside the provider.
04.6
Keep the adapter thin and domain-free
A disciplined adapter knows nothing about your domain. It doesn't know what a "document" or a "citation" is. Its only job is: neutral input → native call → neutral streaming out → accumulated text. All the domain logic (which tools ran, what they touched, how to format the answer) is reconstructed above the adapter, from the callbacks and tool results. This separation keeps adapters small and interchangeable, and keeps your domain logic in one place rather than smeared across three provider files.
04.7
A model registry and validation
Centralise the list of allowed model ids in one place: a registry. This does double duty:
- Provider inference. A simple id-prefix rule (or an explicit map) tells the dispatcher which provider owns a model.
- Validation. The model id often arrives from the client (the user picked it). Validate it against the registry's allow-list before passing it to a provider, and fall back to a safe default for unknown ids. Never hand an unvalidated, client-supplied model id straight to an API.
04.8
Real code: LiteLLM reads the provider from the name
A registry names the provider that owns a model id, and it refuses an id it cannot place. LiteLLM does both in get_llm_provider, which completion() calls before the request. The first excerpt is the shortcut for a provider/model string. The second excerpt runs only when custom_llm_provider is still empty, after the name sets and the startswith prefix rules.
# check if llm provider part of model name
if (
model.split("/", 1)[0] in litellm.provider_list # [1]
and model.split("/", 1)[0] not in litellm.model_list_set
and len(model.split("/"))
> 1 # handle edge case where user passes in `litellm --model mistral` https://github.com/BerriAI/litellm/issues/1351
):
return _get_openai_compatible_provider_info( # [2]
model=model,
api_base=api_base,
api_key=api_key,
dynamic_api_key=dynamic_api_key,
litellm_params=litellm_params,
)
Source: BerriAI/litellm, litellm/litellm_core_utils/get_llm_provider_logic.py (MIT)
Between the excerpts, a bare name is tested against sets filled at startup from the cost map. A known id resolves there, with no prefix on the string.
# Last resort for an otherwise-unknown model: a declarative
# fallback-generalization routing rule (e.g. routes future claude-* to anthropic).
# Exact provider matches above always win; this only runs on a miss.
if not custom_llm_provider:
custom_llm_provider = match_routing_generalization(model) # [3]
if not custom_llm_provider:
# ... (8 lines: a debug print, then error_str, which tells the caller to pass provider/model)
raise litellm.exceptions.BadRequestError( # [4]
message=error_str,
model=model,
response=None,
llm_provider="",
)
Source: BerriAI/litellm, litellm/litellm_core_utils/get_llm_provider_logic.py (MIT)
What to notice
- [1]
provider_listholds the names from theLlmProvidersenum, andsplit("/", 1)cuts at the first slash only. The text before that slash has to be one of those names, and it has to be absent frommodel_list_set. That set isset(model_list), a union of many per-provider name sets. Mistral and Fireworks are registered in other sets and are not in this union. The id also has to contain a slash, so a baremistraldoes not take this shortcut. A prefix that is both a provider name and a name inmodel_list_setskips this helper, and the followingelifstill splits on the slash and returns, so the name sets never see that id. - [2]
_get_openai_compatible_provider_infosplits the same way. The prefix becomes the provider, and the remainder becomes the model id. - [3]
match_routing_generalizationreturns the provider of the first routing rule whose pattern matches. This line runs only whilecustom_llm_provideris still empty, so a name-set match or astartswithprefix above it wins. - [4] If the rule returns nothing, the function raises
BadRequestErrorand does not substitute a model. A registry that quietly fell back to a default provider would send the request somewhere the caller never chose.
Read it yourself: Trace claude-haiku-4-5 and not-a-real-model through get_llm_provider. The first hits anthropic_models. The second reaches the BadRequestError. Then read anthropic-claude-ids, the rule that still sends an unmapped Claude-shaped id to Anthropic.
04.9
Model tiering: the multi-model strategy
Once you can call any model trivially, route work to the right model. A common and effective scheme is three tiers:
- Top tier: the most capable (and expensive) models, for the primary interactive task where quality matters most: complex reasoning, drafting, the main chat.
- Mid tier: fast, capable-enough models for high-volume structured work: bulk extraction, classification, where you run many calls and need throughput and lower cost.
- Low tier: the cheapest, fastest models for trivial background tasks: generating a title, a quick reformat, a yes/no check.
Map tasks to tiers deliberately. Interactive chat → top tier with reasoning on. Bulk extraction → mid tier with reasoning off. Title generation → low tier. This single discipline can cut costs dramatically while improving perceived performance, because cheap tasks finish faster.
A nice refinement: when users bring their own keys, route background tasks to whichever provider they actually have a key for, picking that provider's cheapest model. The tiering becomes "cheapest available model of an available provider."
04.10
Real code: a small model writes the chat title
Title generation is the low-tier job from the previous section: short, cheap, and a poor use of the model that is drafting the reply. OpenCode keeps a hidden agent whose id is title for that call. selectTitle picks the model. The session's own model is the fallback. The function is written with Effect, a library where a function* pauses at yield* until the operation returns a value. Read yield* as "run this, then continue with the result."
const selectTitle = Effect.fn("SessionContext.selectTitle")(function* (session: SessionSchema.Info) {
const agent = yield* agents.get(Agent.ID.make("title"))
if (!agent) return
const primary = yield* resolveModel(session).pipe(Effect.orElseSucceed(() => undefined)) // [1]
const info = yield* Effect.gen(function* () {
if (agent.model) return yield* model.get(agent.model.providerID, agent.model.id) // [2]
if (!primary) return
return yield* model.small(primary.ref.providerID) // [3]
})
const variant =
agent.model?.variant ?? MINIMAL_REASONING_VARIANTS.find((id) => info?.variants.some((item) => item.id === id)) // [4]
const preferred =
info &&
(yield* resolveModel({
...session,
model: Model.Ref.make({
providerID: info.providerID,
id: info.id,
...(variant ? { variant } : {}),
}),
}).pipe(Effect.orElseSucceed(() => undefined)))
const selected = preferred ?? primary // [5]
if (!selected) return
return { agent, primary, selected }
})
Source: anomalyco/opencode, packages/core/src/session/context.ts (MIT)
What to notice
- [1]
primaryis the model already chosen for the session.Effect.orElseSucceed(() => undefined)turns a failure into "no model", so a title problem does not fail the session. If the title agent is missing, the previous line returns and no title is generated. - [2] When that agent has its own
model,model.getloads that provider and id. The configured choice wins over the search below. - [3] Otherwise
model.small(packages/core/src/model.ts) stays on the primary model's provider. It looks throughavailable, the enabled models, and keeps those that areactive, with text in and text out. It then returns the first present family amonggpt-luna,gemini-flash-lite,gemini-flash, andclaude-haiku. If the provider has none of them, the title stays on the primary model. One hard-coded small id would fail for a user who does not have that provider. - [4]
MINIMAL_REASONING_VARIANTS, further down this file, isnone, thenminimal, thenlow. The agent's own variant wins when set. Otherwise this line takes the first of the three listed on the model. The title asks for the smallest reasoning setting available. - [5]
selectedis that model when the secondresolveModelsucceeds, and the primary model otherwise.Model.Refis the pointer to a provider, a model id, and an optional variant, andresolveModelturns that pointer into the model the title call will use.SessionTitle.generatetriesselected, thenprimaryif the attempt returned nothing and the refs differ. The title agent is hidden, and its registration pushes a deny-all permission. This call still has no tools for a simpler reason:request.titleis invoked without a tool list, andprepareuses an empty definition list. The text is truncated to at most 100 characters.
Read it yourself: Read small. Among enabled, active, text-in and text-out models, a provider that has families gemini-flash-lite and claude-haiku, and does not have gpt-luna, yields the flash-lite model, because that family is earlier in the list. In agent.ts, confirm the title agent is hidden and a deny-all permission is pushed. title.ts calls request.title with no tool list, and title.ts is the fallback from selected to primary.
04.11
Reasoning/thinking as a per-call decision
Modern models can expose a reasoning/thinking mode. Treat it as a per-call toggle, not a global setting:
- On for interactive surfaces where the user benefits from seeing the agent think, or where the task is genuinely hard.
- Off for bulk and one-shot jobs where the reasoning stream just burns tokens and time.
Your neutral interface should carry an enableThinking-style flag, and each adapter translates it to that provider's mechanism (and, when off, explicitly zeroes the thinking budget where the provider allows, to actually save the tokens).
04.12
Real code: Anthropic maps one reasoning level
One portable reasoning level goes in, and a provider-specific request shape comes out. On LanguageModelV4CallOptions, reasoning is provider-default, none, minimal, low, medium, high, xhigh, or max. resolveAnthropicReasoningConfig maps that word onto the object Anthropic will accept, and it clamps a level the model cannot take. doGenerate and doStream both reach the function through getArgs and prepareRequest. prepareRequest calls it only when the caller picked a level other than provider-default, and only when effort is not already set on the provider options.
function resolveAnthropicReasoningConfig({
// ... (17 lines: the reasoning level, the model id, and the capability flags)
}): Pick<AnthropicLanguageModelOptions, 'thinking' | 'effort'> | undefined {
if (!isCustomReasoning(reasoning)) { // [1]
return undefined;
}
if (reasoning === 'none') { // [2]
// `between_tools` is the lowest thinking setting on models that support
// it: no upfront thinking, only short progress notes between tool calls.
if (supportsBetweenToolsThinking) {
return { thinking: { type: 'between_tools' } };
}
// ... (13 lines: a model that rejects a disabled switch gets effort 'low' and a warning)
return { thinking: { type: 'disabled' } };
}
if (supportsAdaptiveThinking) { // [3]
// ... (12 lines: map minimal to low, and xhigh to xhigh or max)
return {
thinking: { type: 'adaptive', display: 'summarized' },
effort,
};
}
const budgetTokens = mapReasoningToProviderBudget({ // [4]
// ... (4 lines: the level, the model's output cap, and the warning list)
});
if (budgetTokens == null) {
return undefined;
}
return { thinking: { type: 'enabled', budgetTokens } };
}
Source: vercel/ai, packages/anthropic/src/anthropic-language-model.ts (Apache-2.0)
What to notice
- [1]
isCustomReasoning, inpackages/provider-utils, is false whenreasoningis missing or'provider-default'. The function then returnsundefined, so the shared flag adds no thinking field of its own. Which of the later returns runs is decided by capability flags fromgetModelCapabilities(modelId). - [2]
'none'becomes one of three request shapes. When the model can leave short notes between tool calls, the return is{ thinking: { type: 'between_tools' } }, which means no thinking up front. The cut lines returneffort: 'low'plus a warning when a disabled switch would be rejected, and{ thinking: { type: 'disabled' } }when the model accepts that switch. Thedisabledvalue is still sent. The comment besidesendThinkingsays some models default thinking on, so omitting the field would leave thinking on and spend the output budget. - [3] When the model uses adaptive thinking, the result is
{ type: 'adaptive', display: 'summarized' }plus an effort word frommapReasoningToProviderEffort. The helper maps a requested level onto a word that model accepts, and it warns when the word it sends differs from the one the caller asked for. - [4] Every other model gets a token budget.
mapReasoningToProviderBudgettakes a share of the model's output cap, then clamps that share to a floor of 1024 and to the cap itself. The return is{ thinking: { type: 'enabled', budgetTokens } }.
Read it yourself: In getModelCapabilities, list the flags that change the 'none' return. Then read the comment beside sendThinking and note why a disabled switch is still written into the request.
04.13
What you gain
With a provider abstraction and a tiering strategy in place:
- You can adopt a new model the day it ships by adding an id.
- You can A/B two models by changing a parameter.
- You can serve cost-sensitive bulk jobs and quality-sensitive interactive jobs from the same codebase.
- You can offer BYOK across providers.
- You're insulated from any single vendor's outage or price change.
This layer is among the highest-leverage abstractions in any agent. Build it early; retrofitting it after provider quirks have spread through your code is far more painful than building it up front.
04.14
Read the source yourself
Five stops, in the order this chapter used them. Each link is pinned to the commit the excerpts came from.
- Replay. Start at
transformMessagesin earendil-works/pi. After the excerpt, readclosePendingToolCalls. Then applynormalizeToolCallIdon paper to|plus 80acharacters. The result is_plus 63acharacters. - The contract. Read
LanguageModelV4, then list thetypestrings for text, reasoning, and tool input inlanguage-model-v4-stream-part.ts. In the Anthropic model, find thethinking_deltacase that enqueues areasoning-delta. - The registry. Trace
claude-haiku-4-5,azure/chatgpt-v-2, andnot-a-real-modelthroughget_llm_provider. Then read the ruleanthropic-claude-ids. - The small model. Read
selectTitle, thenMINIMAL_REASONING_VARIANTSjust below it, thenModel.small, then the fallback intitle.ts. - The flag. Read
getModelCapabilitiesforclaude-sonnet-5-5, thenresolveAnthropicReasoningConfig. Write down the object for'none'on that id, and the object for'high'on an id containingclaude-haiku-4-5.
The exercise: take claude-haiku-4-5 and the description's example claude-newfamily-5. The first is an exact cost-map row whose litellm_provider is anthropic, so the name-set branch wins and the regex never runs. The second is not in the cost map, and the anthropic-claude-ids pattern matches it, so the last-resort rule returns anthropic. not-a-real-model matches neither and raises BadRequestError.
0 of 15 slides completed
Your learning progress stays in this browser.
Self-check · 10 questions
01 / 10
The adapter layer's job is to translate between: