Skip to content

Chapter 04 · 14 min read · Updated

Model-Provider Abstraction and Multi-Model Strategy

Decoupling an agent from one model provider takes an adapter pattern and a validated registry. Covers tiering models by task and cost.

Prefer one idea per screen? Read it as 15 slides →

The model you build on today will not be the best model in six months, and the best model for one task is rarely the best for another. Both facts point to the same architectural decision: decouple your agent from any specific model provider. This chapter explains why, how to do it cleanly, and how to route work intelligently across model tiers.

04.1

Why decouple

Three forces make provider abstraction worth the effort even if you launch with a single model:

  1. Models change constantly. New versions ship monthly; prices fall; capabilities shift. If switching models means rewriting your agent, you're stuck on yesterday's model.
  2. Different tasks want different models. A complex drafting task wants a top-tier model; generating a chat title wants the cheapest one. Bulk extraction across a thousand documents wants a fast mid-tier model. You can't serve all of these well with one hard-wired choice.
  3. Risk and economics. A second provider is insurance against an outage, a price hike, or a policy change at any one vendor. And letting users bring their own provider key (Chapter 12) requires supporting whichever provider they chose.

04.2

What makes providers hard to abstract

Providers disagree at the wire level about nearly everything:

  • Tool-schema format: each wants its own shape, and some reject edge cases (an empty parameter object) that others accept.
  • Message format: content blocks vs. parts vs. input items; some require you to replay the model's prior turn verbatim, sometimes including opaque signatures that must be echoed exactly.
  • Streaming events: different event vocabularies for text, reasoning, and tool calls.
  • Reasoning/thinking: different switches and different ways the reasoning is surfaced.
  • History chaining: some resend full history; some use a server-side "previous response id."

If any of this leaks into your agent loop, every provider-specific quirk becomes load-bearing and a fourth provider means a rewrite.

04.3

Real code: pi rewrites a turn from another model

Those disagreements sit in the history, so a turn from another provider, api, or model id cannot be sent on unchanged. pi repairs that in transformMessages (packages/ai). The Anthropic, Google, OpenAI, Bedrock, and Mistral adapters each call it before building a request. An AssistantMessage is one assistant turn: provider, api, model id, and content blocks. The api is the wire protocol, such as anthropic-messages. model is the model about to be called.

if (msg.role === "assistant") {
  const assistantMsg = msg as AssistantMessage;
  const isSameModel =  // [1]
    assistantMsg.provider === model.provider &&
    assistantMsg.api === model.api &&
    assistantMsg.model === model.id;

  const transformedContent = assistantMsg.content.flatMap((block) => {
    if (block.type === "thinking") {
      // ... (3 lines: why thinking from another model is dropped)
      if (!isSameModel) return [];  // [2]
      // Keep redacted and signed blocks even if the thinking text is empty
      // (OpenAI encrypted reasoning); they are needed for replay.
      if (block.redacted || block.thinkingSignature) return block;  // [3]
      // ... (2 lines: drop unsigned thinking whose text is empty)
    }
    // ... (7 lines: text from another model keeps the words and drops textSignature)
    if (block.type === "toolCall") {
      const toolCall = block as ToolCall;
      let normalizedToolCall: ToolCall = toolCall;

      if (!isSameModel && toolCall.thoughtSignature) {  // [4]
        normalizedToolCall = { ...toolCall };
        delete (normalizedToolCall as { thoughtSignature?: string }).thoughtSignature;
      }

      if (!isSameModel && normalizeToolCallId) {  // [5]
        const normalizedId = normalizeToolCallId(toolCall.id, model, assistantMsg);
        if (normalizedId !== toolCall.id) {
          toolCallIdMap.set(toolCall.id, normalizedId);
          normalizedToolCall = { ...normalizedToolCall, id: normalizedId };
        }
      }

      return normalizedToolCall;

Source: earendil-works/pi, packages/ai/src/api/transform-messages.ts (MIT)

What to notice

  1. [1] isSameModel is true only when provider, api, and model id all match. Checking the id alone would replay a turn from a different provider, or from a different api on the same provider.
  2. [2] A thinking block is the model's reasoning, stored apart from the reply. When the provider, api, or model id differs, transformMessages removes that block. The cut comment says the signature is valid only for the model that produced it, and that readable thinking sent as ordinary text would be treated as part of the reply.
  3. [3] On the same model, a block is kept when it is redacted or carries a thinkingSignature, even when the thinking text is empty. thinkingSignature is opaque replay data, and redacted means safety filters removed the text and left an encrypted payload in that signature for the next request. The comment on the kept line names OpenAI encrypted reasoning. The cut lines drop empty unsigned thinking, and they drop textSignature from another model's text.
  4. [4] thoughtSignature sits on the tool call. The ToolCall type calls it Google-specific: an opaque signature for reusing thought context. When the model changed, transformMessages copies the call and deletes the field, because leaving the signature in place would attach one model's thought context to another model's request.
  5. [5] normalizeToolCallId runs only when the model changed, so a same-model conversation keeps its ids. The comment above transformMessages says OpenAI Responses ids run to 450 characters or more and can contain |, while Anthropic requires ^[a-zA-Z0-9_-]+$ and at most 64 characters. Anthropic's function replaces each character outside letters, digits, _, and - with _, then slices to 64. The new id goes into toolCallIdMap, and a later tool result rewrites its toolCallId from that map.

Read it yourself: Read closePendingToolCalls. It inserts an error tool result, text No result provided, when a call has no result. Then apply normalizeToolCallId on paper to | plus 80 a characters. You should get _ plus 63 a characters.

04.4

The adapter pattern

The clean solution is a provider adapter layer: define one neutral interface that the rest of your system speaks, and write a thin adapter per provider that translates to and from the native API. Concretely:

  1. Neutral types. Define your own minimal vocabulary: a message ({role, content}), a tool schema (pick one canonical format, most teams use the widely-documented function-calling shape), a normalized tool call ({id, name, input}), a normalized tool result ({id, content}), and a set of streaming callbacks (onText, onReasoning, onToolCall).
  2. A dispatcher. One function, streamChatWithTools(params), that inspects the requested model id, picks the provider, and calls the right adapter. The rest of your code calls only this.
  3. One adapter per provider. Each implements the same contract: convert neutral input to native, run the streaming tool-call loop, convert native streaming back to your neutral callbacks, and return the accumulated result. The tool-call replay loop (append model turn, append results, repeat) lives inside the adapter, so the provider-specific replay rules are encapsulated.
  4. Schema converters. Small functions that turn your canonical tool schema into each provider's dialect, handling the edge cases (omit empty parameter objects, ensure arrays have item types, etc.).

The payoff: your agent loop calls exactly one function and never branches on provider. Adding a provider is one new adapter file plus registering its model ids. Switching a user from one model to another is a single field change.

04.5

Real code: one interface every provider implements

The adapter pattern is a contract the rest of the program speaks, plus one translator per provider. The Vercel AI SDK writes that contract as the type LanguageModelV4, in the package @ai-sdk/provider. AnthropicLanguageModel, in @ai-sdk/anthropic, is declared implements LanguageModelV4 and sets specificationVersion to 'v4'. The OpenAI and Google language-model classes do the same. Application code talks to the SDK. The SDK calls doGenerate or doStream.

export type LanguageModelV4 = {
  /**
   * The language model must specify which language model interface version it implements.
   */
  readonly specificationVersion: 'v4';  // [1]
  // ... (3 lines: doc comment, this string is the provider id)
  readonly provider: string;
  // ... (3 lines: doc comment, this string is the provider's model id)
  readonly modelId: string;
  // ... (15 lines: supportedUrls, file URLs the model can read without the SDK downloading them)
  // ... (6 lines: doc comment, the "do" prefix so application code does not call this directly)
  doGenerate(  // [2]
    options: LanguageModelV4CallOptions,
  ): PromiseLike<LanguageModelV4GenerateResult>;
  // ... (8 lines: the same naming note, and a note that the result is a stream of output parts)
  doStream(  // [3]
    options: LanguageModelV4CallOptions,
  ): PromiseLike<LanguageModelV4StreamResult>;
};

Source: vercel/ai, packages/provider/src/language-model/v4/language-model-v4.ts (Apache-2.0)

What to notice

  1. [1] specificationVersion is the interface generation this object implements. resolveLanguageModel, in packages/ai/src/model/resolve-model.ts, accepts 'v4', 'v3', and 'v2'. Any other value, including a missing one, throws UnsupportedModelVersionError. A v2 or v3 object is passed through asLanguageModelV4 before the rest of the call uses it.
  2. [2] provider is the provider id, anthropic.messages by default, and modelId is that provider's model id. The API key travels in the request headers. doGenerate is the one-shot call, and its argument, LanguageModelV4CallOptions, holds the shared bag: prompt, tools, temperature, stop sequences, and a reasoning level. The call options have no field for a document or a citation. A citation can still come back on a text-end stream part, as part of the response. PromiseLike means the caller can await the result. The removed comment says the do prefix exists so application code does not call the method by accident.
  3. [3] doStream returns a stream of parts under one set of names. Reply text is a text-delta, reasoning is a reasoning-delta, and a tool call being assembled is a tool-input-delta. Anthropic's doStream receives the API event thinking_delta and enqueues a reasoning-delta whose delta is value.delta.thinking. The OpenAI responses language model and the Google language model enqueue reasoning-delta too. Switching on each provider's own event names would put the wire format back in the agent loop.

Read it yourself: Find resolvedModel.doStream in stream-language-model-call.ts. The call spreads callSettings (the rest of that function's arguments; the comment documents reasoning as one of them) and passes prompt, tools, and providerOptions. Then read providerOptions in language-model-v4-call-options.ts. A setting the shared fields cannot express goes there, and stays inside the provider.

04.6

Keep the adapter thin and domain-free

A disciplined adapter knows nothing about your domain. It doesn't know what a "document" or a "citation" is. Its only job is: neutral input → native call → neutral streaming out → accumulated text. All the domain logic (which tools ran, what they touched, how to format the answer) is reconstructed above the adapter, from the callbacks and tool results. This separation keeps adapters small and interchangeable, and keeps your domain logic in one place rather than smeared across three provider files.

04.7

A model registry and validation

Centralise the list of allowed model ids in one place: a registry. This does double duty:

  • Provider inference. A simple id-prefix rule (or an explicit map) tells the dispatcher which provider owns a model.
  • Validation. The model id often arrives from the client (the user picked it). Validate it against the registry's allow-list before passing it to a provider, and fall back to a safe default for unknown ids. Never hand an unvalidated, client-supplied model id straight to an API.

04.8

Real code: LiteLLM reads the provider from the name

A registry names the provider that owns a model id, and it refuses an id it cannot place. LiteLLM does both in get_llm_provider, which completion() calls before the request. The first excerpt is the shortcut for a provider/model string. The second excerpt runs only when custom_llm_provider is still empty, after the name sets and the startswith prefix rules.

# check if llm provider part of model name

if (
    model.split("/", 1)[0] in litellm.provider_list  # [1]
    and model.split("/", 1)[0] not in litellm.model_list_set
    and len(model.split("/"))
    > 1  # handle edge case where user passes in `litellm --model mistral` https://github.com/BerriAI/litellm/issues/1351
):
    return _get_openai_compatible_provider_info(  # [2]
        model=model,
        api_base=api_base,
        api_key=api_key,
        dynamic_api_key=dynamic_api_key,
        litellm_params=litellm_params,
    )

Source: BerriAI/litellm, litellm/litellm_core_utils/get_llm_provider_logic.py (MIT)

Between the excerpts, a bare name is tested against sets filled at startup from the cost map. A known id resolves there, with no prefix on the string.

# Last resort for an otherwise-unknown model: a declarative
# fallback-generalization routing rule (e.g. routes future claude-* to anthropic).
# Exact provider matches above always win; this only runs on a miss.
if not custom_llm_provider:
    custom_llm_provider = match_routing_generalization(model)  # [3]

if not custom_llm_provider:
    # ... (8 lines: a debug print, then error_str, which tells the caller to pass provider/model)
    raise litellm.exceptions.BadRequestError(  # [4]
        message=error_str,
        model=model,
        response=None,
        llm_provider="",
    )

Source: BerriAI/litellm, litellm/litellm_core_utils/get_llm_provider_logic.py (MIT)

What to notice

  1. [1] provider_list holds the names from the LlmProviders enum, and split("/", 1) cuts at the first slash only. The text before that slash has to be one of those names, and it has to be absent from model_list_set. That set is set(model_list), a union of many per-provider name sets. Mistral and Fireworks are registered in other sets and are not in this union. The id also has to contain a slash, so a bare mistral does not take this shortcut. A prefix that is both a provider name and a name in model_list_set skips this helper, and the following elif still splits on the slash and returns, so the name sets never see that id.
  2. [2] _get_openai_compatible_provider_info splits the same way. The prefix becomes the provider, and the remainder becomes the model id.
  3. [3] match_routing_generalization returns the provider of the first routing rule whose pattern matches. This line runs only while custom_llm_provider is still empty, so a name-set match or a startswith prefix above it wins.
  4. [4] If the rule returns nothing, the function raises BadRequestError and does not substitute a model. A registry that quietly fell back to a default provider would send the request somewhere the caller never chose.

Read it yourself: Trace claude-haiku-4-5 and not-a-real-model through get_llm_provider. The first hits anthropic_models. The second reaches the BadRequestError. Then read anthropic-claude-ids, the rule that still sends an unmapped Claude-shaped id to Anthropic.

04.9

Model tiering: the multi-model strategy

Once you can call any model trivially, route work to the right model. A common and effective scheme is three tiers:

  • Top tier: the most capable (and expensive) models, for the primary interactive task where quality matters most: complex reasoning, drafting, the main chat.
  • Mid tier: fast, capable-enough models for high-volume structured work: bulk extraction, classification, where you run many calls and need throughput and lower cost.
  • Low tier: the cheapest, fastest models for trivial background tasks: generating a title, a quick reformat, a yes/no check.

Map tasks to tiers deliberately. Interactive chat → top tier with reasoning on. Bulk extraction → mid tier with reasoning off. Title generation → low tier. This single discipline can cut costs dramatically while improving perceived performance, because cheap tasks finish faster.

A nice refinement: when users bring their own keys, route background tasks to whichever provider they actually have a key for, picking that provider's cheapest model. The tiering becomes "cheapest available model of an available provider."

04.10

Real code: a small model writes the chat title

Title generation is the low-tier job from the previous section: short, cheap, and a poor use of the model that is drafting the reply. OpenCode keeps a hidden agent whose id is title for that call. selectTitle picks the model. The session's own model is the fallback. The function is written with Effect, a library where a function* pauses at yield* until the operation returns a value. Read yield* as "run this, then continue with the result."

const selectTitle = Effect.fn("SessionContext.selectTitle")(function* (session: SessionSchema.Info) {
  const agent = yield* agents.get(Agent.ID.make("title"))
  if (!agent) return
  const primary = yield* resolveModel(session).pipe(Effect.orElseSucceed(() => undefined))  // [1]
  const info = yield* Effect.gen(function* () {
    if (agent.model) return yield* model.get(agent.model.providerID, agent.model.id)  // [2]
    if (!primary) return
    return yield* model.small(primary.ref.providerID)  // [3]
  })
  const variant =
    agent.model?.variant ?? MINIMAL_REASONING_VARIANTS.find((id) => info?.variants.some((item) => item.id === id))  // [4]
  const preferred =
    info &&
    (yield* resolveModel({
      ...session,
      model: Model.Ref.make({
        providerID: info.providerID,
        id: info.id,
        ...(variant ? { variant } : {}),
      }),
    }).pipe(Effect.orElseSucceed(() => undefined)))
  const selected = preferred ?? primary  // [5]
  if (!selected) return
  return { agent, primary, selected }
})

Source: anomalyco/opencode, packages/core/src/session/context.ts (MIT)

What to notice

  1. [1] primary is the model already chosen for the session. Effect.orElseSucceed(() => undefined) turns a failure into "no model", so a title problem does not fail the session. If the title agent is missing, the previous line returns and no title is generated.
  2. [2] When that agent has its own model, model.get loads that provider and id. The configured choice wins over the search below.
  3. [3] Otherwise model.small (packages/core/src/model.ts) stays on the primary model's provider. It looks through available, the enabled models, and keeps those that are active, with text in and text out. It then returns the first present family among gpt-luna, gemini-flash-lite, gemini-flash, and claude-haiku. If the provider has none of them, the title stays on the primary model. One hard-coded small id would fail for a user who does not have that provider.
  4. [4] MINIMAL_REASONING_VARIANTS, further down this file, is none, then minimal, then low. The agent's own variant wins when set. Otherwise this line takes the first of the three listed on the model. The title asks for the smallest reasoning setting available.
  5. [5] selected is that model when the second resolveModel succeeds, and the primary model otherwise. Model.Ref is the pointer to a provider, a model id, and an optional variant, and resolveModel turns that pointer into the model the title call will use. SessionTitle.generate tries selected, then primary if the attempt returned nothing and the refs differ. The title agent is hidden, and its registration pushes a deny-all permission. This call still has no tools for a simpler reason: request.title is invoked without a tool list, and prepare uses an empty definition list. The text is truncated to at most 100 characters.

Read it yourself: Read small. Among enabled, active, text-in and text-out models, a provider that has families gemini-flash-lite and claude-haiku, and does not have gpt-luna, yields the flash-lite model, because that family is earlier in the list. In agent.ts, confirm the title agent is hidden and a deny-all permission is pushed. title.ts calls request.title with no tool list, and title.ts is the fallback from selected to primary.

04.11

Reasoning/thinking as a per-call decision

Modern models can expose a reasoning/thinking mode. Treat it as a per-call toggle, not a global setting:

  • On for interactive surfaces where the user benefits from seeing the agent think, or where the task is genuinely hard.
  • Off for bulk and one-shot jobs where the reasoning stream just burns tokens and time.

Your neutral interface should carry an enableThinking-style flag, and each adapter translates it to that provider's mechanism (and, when off, explicitly zeroes the thinking budget where the provider allows, to actually save the tokens).

04.12

Real code: Anthropic maps one reasoning level

One portable reasoning level goes in, and a provider-specific request shape comes out. On LanguageModelV4CallOptions, reasoning is provider-default, none, minimal, low, medium, high, xhigh, or max. resolveAnthropicReasoningConfig maps that word onto the object Anthropic will accept, and it clamps a level the model cannot take. doGenerate and doStream both reach the function through getArgs and prepareRequest. prepareRequest calls it only when the caller picked a level other than provider-default, and only when effort is not already set on the provider options.

function resolveAnthropicReasoningConfig({
  // ... (17 lines: the reasoning level, the model id, and the capability flags)
}): Pick<AnthropicLanguageModelOptions, 'thinking' | 'effort'> | undefined {
  if (!isCustomReasoning(reasoning)) {  // [1]
    return undefined;
  }

  if (reasoning === 'none') {  // [2]
    // `between_tools` is the lowest thinking setting on models that support
    // it: no upfront thinking, only short progress notes between tool calls.
    if (supportsBetweenToolsThinking) {
      return { thinking: { type: 'between_tools' } };
    }
    // ... (13 lines: a model that rejects a disabled switch gets effort 'low' and a warning)
    return { thinking: { type: 'disabled' } };
  }

  if (supportsAdaptiveThinking) {  // [3]
    // ... (12 lines: map minimal to low, and xhigh to xhigh or max)
    return {
      thinking: { type: 'adaptive', display: 'summarized' },
      effort,
    };
  }

  const budgetTokens = mapReasoningToProviderBudget({  // [4]
    // ... (4 lines: the level, the model's output cap, and the warning list)
  });
  if (budgetTokens == null) {
    return undefined;
  }
  return { thinking: { type: 'enabled', budgetTokens } };
}

Source: vercel/ai, packages/anthropic/src/anthropic-language-model.ts (Apache-2.0)

What to notice

  1. [1] isCustomReasoning, in packages/provider-utils, is false when reasoning is missing or 'provider-default'. The function then returns undefined, so the shared flag adds no thinking field of its own. Which of the later returns runs is decided by capability flags from getModelCapabilities(modelId).
  2. [2] 'none' becomes one of three request shapes. When the model can leave short notes between tool calls, the return is { thinking: { type: 'between_tools' } }, which means no thinking up front. The cut lines return effort: 'low' plus a warning when a disabled switch would be rejected, and { thinking: { type: 'disabled' } } when the model accepts that switch. The disabled value is still sent. The comment beside sendThinking says some models default thinking on, so omitting the field would leave thinking on and spend the output budget.
  3. [3] When the model uses adaptive thinking, the result is { type: 'adaptive', display: 'summarized' } plus an effort word from mapReasoningToProviderEffort. The helper maps a requested level onto a word that model accepts, and it warns when the word it sends differs from the one the caller asked for.
  4. [4] Every other model gets a token budget. mapReasoningToProviderBudget takes a share of the model's output cap, then clamps that share to a floor of 1024 and to the cap itself. The return is { thinking: { type: 'enabled', budgetTokens } }.

Read it yourself: In getModelCapabilities, list the flags that change the 'none' return. Then read the comment beside sendThinking and note why a disabled switch is still written into the request.

04.13

What you gain

With a provider abstraction and a tiering strategy in place:

  • You can adopt a new model the day it ships by adding an id.
  • You can A/B two models by changing a parameter.
  • You can serve cost-sensitive bulk jobs and quality-sensitive interactive jobs from the same codebase.
  • You can offer BYOK across providers.
  • You're insulated from any single vendor's outage or price change.

This layer is among the highest-leverage abstractions in any agent. Build it early; retrofitting it after provider quirks have spread through your code is far more painful than building it up front.

04.14

Read the source yourself

Five stops, in the order this chapter used them. Each link is pinned to the commit the excerpts came from.

  1. Replay. Start at transformMessages in earendil-works/pi. After the excerpt, read closePendingToolCalls. Then apply normalizeToolCallId on paper to | plus 80 a characters. The result is _ plus 63 a characters.
  2. The contract. Read LanguageModelV4, then list the type strings for text, reasoning, and tool input in language-model-v4-stream-part.ts. In the Anthropic model, find the thinking_delta case that enqueues a reasoning-delta.
  3. The registry. Trace claude-haiku-4-5, azure/chatgpt-v-2, and not-a-real-model through get_llm_provider. Then read the rule anthropic-claude-ids.
  4. The small model. Read selectTitle, then MINIMAL_REASONING_VARIANTS just below it, then Model.small, then the fallback in title.ts.
  5. The flag. Read getModelCapabilities for claude-sonnet-5-5, then resolveAnthropicReasoningConfig. Write down the object for 'none' on that id, and the object for 'high' on an id containing claude-haiku-4-5.

The exercise: take claude-haiku-4-5 and the description's example claude-newfamily-5. The first is an exact cost-map row whose litellm_provider is anthropic, so the name-set branch wins and the regex never runs. The second is not in the cost map, and the anthropic-claude-ids pattern matches it, so the last-resort rule returns anthropic. not-a-real-model matches neither and raises BadRequestError.

0 of 15 slides completed

Your learning progress stays in this browser.

Self-check · 10 questions

01 / 10

The adapter layer's job is to translate between:

Related chapters