Skip to main content

LLM Providers

AgentOS abstracts every LLM behind a single IProvider interface. Thirteen providers are wired in directly: eleven reached with an API key or a base URL, and two local CLI bridges that ride an existing Claude Max or Google account subscription. Three of the thirteen are gateways: OpenRouter and Requesty route to models from many vendors, and LiteLLM is a proxy you host. Every provider speaks the same streaming protocol, supports the same tool-call shape (with the documented exceptions below), and reports each call's token usage in the same shape; Prices and Cost Reporting says which providers also report the call's cost. The fallback chain is auto-built from whichever keys are set in the environment and is overridable per agent.

Table of Contents​

  1. Overview
  2. Provider Matrix
  3. Quick Start
  4. Auto-Detection Order
  5. Provider Configuration
  6. Fallback Behavior
  7. Prices and Cost Reporting
  8. Provider Details
  9. Programmatic Configuration
  10. Adding a Custom Provider
  11. Provider Capabilities Detail
  12. Related Documentation

Overview​

AgentOS abstracts LLM access behind a unified IProvider interface. You configure providers via environment variables, and AgentOS handles model selection, streaming, tool calling, retries, and fallback routing.

Key features:

  • 13 providers supported out of the box (11 by API key or base URL, 2 CLI-based)
  • CLI providers: Use your Claude Max or Google account subscription via local CLI — no API key needed
  • Auto-detection: Set an API key or install a CLI and the provider is available
  • Fallback: Automatic retry with alternate providers on failure (fallbackProviders)
  • Cost-aware caps: agency() checks controls.maxCostUSD and the other run limits after each run; agent() applies controls.maxTotalTokens and controls.maxDurationMs to each model call; a budget on generateText(), streamText(), generateObject(), embedText() or agent() checks each provider call against a spend cap before it is sent; a custom router sends requests to cheaper models
  • Streaming: All providers support streaming with a unified async iterator
  • Tool calling: Unified function/tool calling across providers that support it

Provider Matrix​

ProviderEnv VarDefault ModelStreamingTool CallingVisionEmbeddingDefault model's price (input / output, USD per 1M tokens)
OpenAIOPENAI_API_KEYgpt-4oYesYesYesYes$2.50 / $10.00
AnthropicANTHROPIC_API_KEYclaude-sonnet-4-6YesYesYesNo$3.00 / $15.00
GeminiGEMINI_API_KEYgemini-2.5-flashYesYesYesYes$0.30 / $2.50
GroqGROQ_API_KEYllama-3.3-70b-versatileYesYesNoNoEnterprise contracts only
TogetherTOGETHER_API_KEYmeta-llama/Llama-3.3-70B-Instruct-TurboYesYesNoNo$1.04 / $1.04
MistralMISTRAL_API_KEYmistral-large-latestYesYesNoYes$0.50 / $1.50
xAIXAI_API_KEYgrok-2YesYesYesNoNot on xAI's model list
OpenRouterOPENROUTER_API_KEYopenai/gpt-4oYesYesYes*Yes*$2.50 / $10.00
RequestyREQUESTY_API_KEYopenai/gpt-4oYesYesYes*Yes*$2.50 / $10.00
LiteLLM(none read; pass apiKey)(none; pass model)YesYes*Yes*Yes*Your proxy's upstream prices
OllamaOLLAMA_BASE_URLllama3.2YesPartialModel-dep.YesFree
Claude Code CLI(PATH detection)claude-sonnet-4-6YesYesYesNoFree*
Gemini CLI(PATH detection)gemini-3.5-flashYesPartial**YesNoFree*

Prices are the vendors' list prices, read on 2026-10-12; Prices and Cost Reporting links each source.

*CLI providers use your existing subscription — $0 per token. **Gemini CLI tool calling uses XML prompt-based parsing (less reliable than native API tool calling).

Gemini CLI ToS Warning: Google's Gemini CLI ToS may prohibit third-party subprocess invocation with OAuth auth. Use gemini with API key for production. See CLI Providers for details.

*OpenRouter, Requesty and LiteLLM capabilities depend on the underlying model selected.

Quick Start​

Option 1: Environment Variable (Simplest)​

Set one API key and start using AgentOS:

export OPENAI_API_KEY=sk-...
import { agent } from '@framers/agentos';

const myAgent = agent({});  // Auto-detects from env (OpenAI here)
const result = await myAgent.generate('Hello, world!');
console.log(result.text);

Option 2: Programmatic​

import { agent } from '@framers/agentos';

const myAgent = agent({
  provider: 'anthropic',
  model: 'claude-sonnet-4-5-20250929',
});

The agent() factory is synchronous — it does not return a Promise. The first network call happens on generate() / stream() / session().send().

Auto-Detection Order​

When neither provider nor model is set, AgentOS checks for API keys in this order and uses the first one found:

  1. OPENROUTER_API_KEY → OpenRouter
  2. OPENAI_API_KEY → OpenAI
  3. ANTHROPIC_API_KEY → Anthropic
  4. GEMINI_API_KEY → Google Gemini
  5. GROQ_API_KEY → Groq
  6. TOGETHER_API_KEY → Together AI
  7. MISTRAL_API_KEY → Mistral
  8. XAI_API_KEY → xAI
  9. REQUESTY_API_KEY → Requesty
  10. which claude → Claude Code CLI (PATH detection — no API key, uses Max subscription)
  11. which gemini → Gemini CLI (PATH detection — no API key, uses Google account)
  12. OLLAMA_BASE_URL → Ollama

An OLLAMA_BASE_URL that is empty or whitespace alone is unset, as the call reads it, so detection passes over Ollama.

LiteLLM is never auto-detected: name it with provider: 'litellm' (LiteLLM).

You can override auto-detection in four ways, highest priority first:

  1. Inline — agent({ provider: '...', apiKey: '...' }) on a single call.
  2. Module-level default — setDefaultProvider({ provider, apiKey }) once at boot. Every subsequent call inherits it; inline opts still win when supplied. Useful when credentials live in a secrets manager rather than .env.
  3. Reorder the auto-detect chain — setProviderPriority(['anthropic', 'openai', ...]) to change which env-var keys are preferred when multiple are set, without forcing a single provider. Empty array disables auto-detect entirely.
  4. CLI flag — for the Wunderland CLI, pass --provider <name>.
import { setDefaultProvider, generateText, agent } from '@framers/agentos';

setDefaultProvider({
  provider: 'openai',
  apiKey: process.env.MY_OWN_KEY,
  // optional: model: 'gpt-4o-mini', baseUrl: '...'
});

// No env vars, no inline opts — just works:
const { text } = await generateText({ prompt: 'hello' });
const bot = agent({ instructions: '...' });

// Inline still wins:
generateText({ apiKey: 'sk-tenant-scoped', prompt: 'isolated call' });

Provider Configuration​

Each provider is configured via environment variables. You can set them in your shell or .env file:

# .env

# Primary provider
OPENAI_API_KEY=sk-...

# Fallback provider
OPENROUTER_API_KEY=sk-or-...

# Local provider (no API key needed)
OLLAMA_BASE_URL=http://localhost:11434

For a provider that rotates keys, the key variable (or the apiKey you pass) may hold a comma-separated list: a key pool. The LLM providers move a request to the next usable key when one is revoked, unpaid or throttled, probe a failed key before it returns, and persist the keys' standing when given a store; AGENTOS_KEY_POOL_STRATEGY_ANTHROPIC=ordered uses Anthropic's keys in the order given (credited keys first, the paid key last), and the same variable with another provider's name (_OPENAI, _GEMINI, _OPENROUTER, _REQUESTY) orders that provider's; AGENTOS_KEY_POOL_STRATEGY=ordered orders every provider's keys, so a list kept for load spreading on another provider becomes ordered too. API Key Rotation has the strategies, the failure kinds, and the providers that need a single key.

Per-Agent Override​

Individual agents pick their provider/model directly in the agent({ ... }) config:

import { agent } from '@framers/agentos';

const writer = agent({
  provider: 'anthropic',
  model: 'claude-sonnet-4-5-20250929',
  apiKey: process.env.ANTHROPIC_API_KEY, // optional override
});

Fallback Behavior​

AgentOS supports automatic fallback when a provider request fails on a retryable error: HTTP 401, 402, 403, 429, 500, 502, 503, 504 or 529, a network failure or request timeout, a primary provider that cannot initialize (for example a revoked key that its model listing rejects), a request larger than the model's context window, or a content-policy refusal (isRetryableError()). Fallback is on by default with an auto-built chain — to disable it, pass an empty array.

The chain serves generateText() and streamText(), and agent() and agency() through them. The GMIs of the full runtime (processRequest()) call their provider without a fallback chain; a GMI built with a completion gateway moves to the next hop when a hop cannot start or an attempt fails with a retryable error before any output (Model calls through a completion gateway). An agent created with runtime: 'gmi' builds that gateway from its own fallbackProviders (unset, the same auto-built chain), policyTier, router and onFallback, so its sessions fall back as a gateway does: before a step's first output, and not after it (GMIs from agent()).

A failover never repeats work the caller already received or that had side effects. A stream that has delivered text or tool activity is not restarted on a fallback provider; it ends with an error part, so the consumer never receives a partial answer followed by a second one. A generateText call that already ran native tool rounds continues on the fallback provider from those rounds (the fallback sees each tool call and its result) instead of starting over and running the tools again. A call whose prompt-emulated tools (toolMode: 'prompt') ran throws the error instead of failing over.

Each leg receives the caller's customModelParams minus the fields only another vendor accepts: OpenRouter's routing controls (provider, models, route, transforms) reach only OpenRouter, and Gemini's request fields (thinkingConfig, topK, safetySettings and the rest of GEMINI_ONLY_PARAM_KEYS) reach only Gemini. The structured-output fields follow the same rule: OpenAI's response_format does not reach Anthropic or Gemini, and Anthropic's output_config does not reach OpenAI or Gemini, so an OpenAI call that fails over to Claude (or the reverse) sends the leg only fields its API defines; the leg's own structured output is built for it by its provider.

With planning on, the planning call carries the caller's messages, so it takes OpenRouter's routing controls and plugins from customModelParams: a provider.data_collection: 'deny' or a context-compression plugin set for the turn holds for it too. The other keys, such as max_tokens or response_format, go to the turn's requests alone.

Primary provider (e.g., Anthropic)
  ↓ fails with a retryable error
OpenAI gpt-5.6-sol               (if OPENAI_API_KEY is set)
  ↓ fails
OpenRouter openai/gpt-5.6-sol    (if OPENROUTER_API_KEY is set)
  ↓ fails
Gemini gemini-3.1-pro-preview    (if GEMINI_API_KEY is set)
  ↓ fails
Error returned to caller

The auto-built chain (buildFallbackChain()) takes these legs in this order, each when its key is set, and leaves out the primary's own provider: OpenAI gpt-5.6-sol, Anthropic claude-sonnet-5-5 (effort low, 1024 tokens of output headroom), OpenRouter openai/gpt-5.6-sol, and Gemini gemini-3.1-pro-preview (effort low, 1024 tokens of output headroom). Every leg runs with prompt caching off (cache: false). Ollama is not in the auto-built chain; list it in fallbackProviders to use it. For a call whose policy tier is mature or private-adult, the chain starts with the policy catalog's fallback ladder for that tier when OPENROUTER_API_KEY is set (buildPolicyAwareFallbackChain()).

Configuring Fallback​

import { agent } from '@framers/agentos';

const myAgent = agent({
  provider: 'anthropic',
  model: 'claude-sonnet-4-5-20250929',
  // Ordered fallback chain — each entry can override the model.
  fallbackProviders: [
    { provider: 'openrouter', model: 'anthropic/claude-sonnet-4-5-20250929' },
    { provider: 'ollama',     model: 'llama3.2' },
  ],
  onFallback: (err, next) => {
    console.warn(`Falling back to ${next}: ${err.message}`);
  },
});

// Disable fallback entirely:
const strict = agent({ provider: 'anthropic', fallbackProviders: [] });

OpenRouter as Universal Fallback​

Setting OPENROUTER_API_KEY automatically enables it as a fallback for any primary provider in the auto-built chain. OpenRouter routes to 200+ models across all major providers.

# Primary: Anthropic. Fallback: OpenRouter (automatic)
export ANTHROPIC_API_KEY=sk-ant-...
export OPENROUTER_API_KEY=sk-or-...

Prices and Cost Reporting​

Every provider reports each call's token usage. Whether the result also carries the call's cost, usage.costUSD, depends on the provider:

Providerusage.costUSD
OpenAI, Anthropic, GeminiPriced from the provider's own table of the vendor's list prices (openaiPricing.ts, anthropicCatalog.ts, GeminiProvider.ts). A model the table has no row for reports no cost.
OpenRouter, RequestyThe cost the gateway returns with the response
Ollama, Claude Code CLI, Gemini CLI0
Groq, Together, Mistral, xAI, LiteLLM, MiniMaxThese providers send their calls through the OpenAI provider (MiniMax extends it), which prices a call from OpenAI's table (openaiPricing.ts) by the model id the response names, exact or with a trailing -YYYY-MM-DD removed; an id the table has no row for reports no cost. The table lists only OpenAI's models, so a call to the default model of Groq, Together, Mistral, xAI or MiniMax reports no cost, and a LiteLLM call whose response names an OpenAI model, such as gpt-4o-mini, is priced at OpenAI's list price. A spend budget charges such a call at the row a host registered for the provider with registerModelPricing(), where it registered one, instead of the cost reported.

Prices for a host: openAIModelPricing(modelId) and anthropicModelPricing(modelId), exported from @framers/agentos/safety/runtime and from the package root with the types OpenAIModelPrice and AnthropicModelPrice, give a spend guard a call's list price in USD per 1K tokens before the call is made, without loading a provider: { input, output, cachedInput? } for OpenAI, cachedInput being the cached-input rate OpenAI's pricing page lists for the model (absent where it lists none, for the pro models, or no longer lists the model; such a model's cache reads are billed at input), and { input, output, cacheRead, cacheWrite, cacheWriteOneHour, longContext? } for Claude, cacheWrite and cacheWriteOneHour being the 5-minute and 1-hour prompt-cache writes at 1.25x and 2x the input rate. modelPricing(providerId, modelId) reads both tables, and a table a host registers for any other provider with registerModelPricing(), in one shape in USD per million tokens with every tier a request can reach; estimateMaxCallCostUSD() bounds a call's cost with it (Pricing a call before it is sent). anthropicModelPricing resolves an exact catalog id, a dated snapshot of a bare id (claude-sonnet-5-5-20261001) and the bare alias of a dated row (claude-opus-4-5), and answers undefined for any other id (claude-opus-4, claude-sonnet-4, claude), never a prefix guess; a retired snapshot (claude-opus-4-20250514) prices by its exact id alone. longContext is present on a model priced by prompt length (Claude Haiku 5.5: { fromTokens: 100000, input, output, cacheRead, cacheWrite, cacheWriteOneHour }) and names the prompt size above which its long input, output and cacheRead rates apply; estimateAnthropicCostUSD bills a response whose prompt, cached tokens included, is over that size at the same long rates.

The list prices of the default text models, in USD per 1M tokens, as each vendor's page gave them on 2026-10-12. Each vendor's other models cost more or less; the linked pages list them.

ProviderDefault text modelInputOutputSource
Ollamallama3.200Runs on your machine
Claude Code CLIclaude-sonnet-4-600Uses your existing subscription
Gemini CLIgemini-3.5-flash00Uses your existing subscription
Geminigemini-2.5-flash$0.30 ($1.00 for audio)$2.50, thinking includedGemini API pricing
Mistralmistral-large-latest (Mistral Large 3)$0.50$1.50Mistral Large 3
Togethermeta-llama/Llama-3.3-70B-Instruct-Turbo$1.04$1.04Together AI pricing
OpenAIgpt-4o$2.50$10.00OpenAI pricing
OpenRouteropenai/gpt-4o$2.50$10.00OpenRouter: GPT-4o
Requestyopenai/gpt-4o$2.50$10.00Requesty model list
Anthropicclaude-sonnet-4-6$3.00$15.00Anthropic pricing
Groqllama-3.3-70b-versatileOn requestOn requestGroq models: enterprise contracts only (Groq)
xAIgrok-2Not listedNot listedxAI models (xAI)
LiteLLMnone; pass modelYour proxy's upstreamYour proxy's upstream

Cost-Aware Caps​

Run limits live on controls, and agency() is the surface that checks all of them:

import { agency } from '@framers/agentos';

const team = agency({
  provider: 'anthropic',
  agents: {
    researcher: { instructions: 'Find the facts.' },
    writer: { instructions: 'Write the summary.' },
  },
  controls: {
    maxCostUSD: 0.05,           // total cost of a run
    maxTotalTokens: 50_000,     // prompt + completion tokens of a run
    maxDurationMs: 30_000,      // wall-clock time of a run
    maxAgentCalls: 10,          // agent invocations in a run
    onLimitReached: 'error',    // 'stop' | 'warn' | 'error'
  },
  on: {
    limitReached: (e) => console.warn(`${e.metric}: ${e.value} > ${e.limit}`),
  },
});

agency() compares a run against these limits when the run ends, and compares the agency's running token and cost totals against them before each new run starts. A breach calls on.limitReached; with onLimitReached: 'error' it throws an AgencyConfigError instead. 'stop' and 'warn' both report through the callback, and no limit interrupts a run that is in progress.

agent() reads two of the fields, per model call: controls.maxTotalTokens becomes the call's completion-token cap (maxTokens) when the agent sets no maxTokens, and controls.maxDurationMs becomes the request timeout. It does not read maxCostUSD, maxAgentCalls or onLimitReached. To cap an agent's spend, give it a budget (agent({ budget: { maxCostUSD: 0.25 } })): each provider call of its generate(), stream() and sessions is checked against what is left before it is sent, and by default a call whose estimated cost would pass the cap is refused with CostCapExceededError. The budget holds its spending in a CostGuard (Cost Optimization).

For cheap-first routing across multiple models, attach a custom IModelRouter via agent({ router }) — the router decides which provider/model to call per request. See Cost Optimization for the full guide.

Provider Details​

OpenAI​

export OPENAI_API_KEY=sk-...
ModelContextVisionTool CallingNotes
gpt-6-astra1.05MYesYesMost capable GPT ($10/$50 per MTok)
gpt-6-sol1.05MYesYesGPT-6 at $2/$10 per MTok
gpt-6-luna1.05MYesYesCheapest GPT-6 ($0.10/$0.50 per MTok)
gpt-5.6-sol1.05MYesYesGPT-5.6 flagship ($4/$20 per MTok)
gpt-4o128KYesYesAccepts temperature
gpt-4o-mini128KYesYesFast, cheap
gpt-image-2.5-flaren/an/an/aImage generation only: the generateImage() default
gpt-image-2.5-sunburstn/an/an/aImage editing: the editImage(), variateImage() and transferStyle() default

The GPT-5, GPT-6 and o-series models reject temperature and top_p and require max_completion_tokens; the provider handles both. Streamed and non-streamed calls follow one routing rule. The provider sends a call to the Responses API when it goes to a Responses-only model (the -pro models, the codex models, gpt-5.6-cyber and the deep-research models), when it is a GPT-6 call carrying function tools, or when it is a GPT-5 call carrying function tools and an effort; everything else goes to Chat Completions. OpenAI serves GPT-6 tool calls only through Responses (Chat Completions accepts them from Sol and Luna only with reasoning_effort: "none"). On the Responses path, responseFormat travels as text.format, images in user turns and tool results travel as input_image, and a streamed call reports its usage on the final chunk. A result marked failed throws RESPONSES_FAILED, streamed or not, whatever text it holds, a non-streamed one with the usage it reported in details.usage; a non-streamed one whose error code is retryable (server_error, rate_limit_exceeded, vector_store_timeout) comes with a 200 and is sent again while the call has attempts left, as a 5xx is, and the provider reports that attempt as the success its status was; the usage it reported, priced as its own request, is added to the result the call returns, or to the error it ends with in details.usage (an answer with no output items among them, whose INVALID_RESPONSE carries its body with usage in the provider's form), so two requests each below the long-context threshold are never priced as one above it. One marked failed with rate_limit_exceeded rests its pooled key as a 429 does, and the retry goes out with the key the pool draws next: a usable one, or, when every key rests, the one whose rest ends soonest (Key rotation). A message part Responses cannot carry, such as audio, fails a call that has no Chat Completions route with RESPONSES_UNMAPPABLE_CONTENT. OpenAI retires o1, o1-pro, o3-mini and o4-mini on 2026-10-23, and o3 and o3-pro on 2026-12-11.

usage.costUSD prices the input as the prompt-caching guide does: cached_tokens at the model's cached-input rate (gpt-6-sol $0.20 against $2 input per MTok), and on GPT-5.6 and later cache_write_tokens at 1.25x input for the 30-minute lifetime and 2.5x for the 12-hour one, by the Responses API's cache_write_tokens_details split (a usage without the split is priced as 30-minute writes, the lifetime a request with no cache setting writes with); the rest of prompt_tokens at the input rate. Over 272,000 prompt tokens on a model with long-context pricing, every input rate doubles. The 12-hour count reaches the usage as cacheCreationExtendedInputTokens.

OAuth support: Use your ChatGPT subscription instead of an API key via the device code flow. See OAuth Auth for details.

Anthropic​

export ANTHROPIC_API_KEY=sk-ant-...
ModelContextVisionTool CallingNotes
claude-opus-5-51MYesYesRecommended starting model ($4/$20 per MTok)
claude-fable-5-11MYesYesMost capable ($10/$50 per MTok)
claude-sonnet-5-51MYesYesCurrent Sonnet ($2/$10 per MTok)
claude-sonnet-51MYesYesNear-Opus coding and agentic work ($2/$10 per MTok)
claude-opus-51MYesYesPrevious Opus ($5/$25 per MTok)
claude-sonnet-4-61MYesYesBest value
claude-haiku-5-51MYesYesCurrent Haiku ($0.10/$0.50 per MTok up to a 100K-token prompt)
claude-haiku-4-5-20251001200KYesYesPrevious Haiku ($1/$5 per MTok)
claude-opus-4-5-20251101200KYesYesLegacy Opus, 64K output ($5/$25 per MTok)

Reasoning-default models (claude-opus-5-5, claude-fable-5-1, claude-sonnet-5-5, claude-opus-5, claude-fable-5, claude-sonnet-5, claude-opus-4-8, claude-opus-4-7) reject temperature and top_p with HTTP 400; the provider drops both automatically for these models and sends adaptive thinking when a thinking budget is requested. claude-haiku-5-5 rejects any non-default temperature, top_p or top_k too, so the provider drops temperature and top_p for it as well; it thinks adaptively by default, so a thinking budget sends no thinking block there. The same fields passed through customModelParams are dropped too (top_k as well on Claude Haiku 5.5, whose migration guide says to remove it), with a one-time warning. A customModelParams.model replaces the model on the wire, and the provider shapes the request (sampling, thinking, effort, max_tokens and caching) for that model rather than the one given. Claude Opus 5.5, Claude Sonnet 5.5, Claude Mythos 5.1 and the Fable models also reject a forced tool_choice, so the provider sends auto for them; a schema request on these models takes the native JSON output format (output_config.format, see Tool Calling Support below) when the schema fits the format's limits, and the provider's own schema block in the system prompt when it does not. The provider sends a forced tool_choice to every other Claude model, so a schema the format cannot carry goes to it as a forced tool call (Claude Haiku 5.5 among them, probed on 2026-10-19). Anthropic retired claude-opus-4-20250514 and claude-sonnet-4-20250514 on 2026-06-15 (the API answers 404 for both; the catalog keeps them as deprecated); a schema request on those two ids takes the forced tool.

Opus 5 and later, Sonnet 5 and later, Haiku 5.5, Fable and Mythos think by default. Pass thinking: false to turn thinking off: the provider sends { type: 'between_tools' } on claude-sonnet-5-5 and { type: 'disabled' } on claude-opus-5, claude-sonnet-5 and claude-haiku-5-5, and lowers effort to high on Sonnet 5.5, Opus 5 and Haiku 5.5, which turn thinking off only at that level or below; an effort passed through customModelParams.output_config is lowered the same way, and the rest of that output_config is sent as given. claude-opus-5-5, the Fable models and Mythos always think; thinking: false sends nothing there and logs a warning once. Older models think only when asked, so thinking: false leaves the field out. effort follows each model's ladder: claude-opus-4-5 takes low, medium and high, claude-opus-4-6 and claude-sonnet-4-6 add max, and later models (claude-haiku-5-5 among them) take all five levels; a level a model does not take is sent as high. claude-haiku-4-5 takes no effort, so the provider leaves it out there. generateObject and streamObject take the same thinking and effort options and forward them unchanged; a structured call on a model that thinks by default needs thinking: false or room in maxTokens for the thinking.

usage.costUSD prices cache reads at 0.1x the input price, except Claude Opus 5.5 and Claude Sonnet 5.5 (0.05x) and Claude Fable 5.1 (0.025x), and cache writes at 1.25x for the 5-minute TTL and 2x for the 1-hour TTL, read from the response's cache_creation split; a response without the split is priced as 5-minute writes, the TTL a cache_control without ttl writes with. The 1-hour count reaches the call's usage as cacheCreationExtendedInputTokens (cacheCreationExtendedTokens on a generateText() or streamText() result), so a spend budget that prices the tokens itself (a model the catalog lacks, priced at a host's row) prices the two TTLs apart.

A call on the anthropic provider is sent to ANTHROPIC_BASE_URL when it is set (https://api.anthropic.com form, without /v1, as Anthropic's own SDKs read it), after the call's own baseUrl and the global default's, as OPENAI_BASE_URL routes the openai provider. A value with surrounding whitespace is read trimmed, trailing slashes are dropped before /v1/messages is appended, and an empty or whitespace-only value is unset.

When Claude declines a request (stop_reason: "refusal"), the provider throws an AnthropicProviderError with code content_filter, whether the refusal came before any output or partway through it. The error carries stop_details and any partial text in details, and no tool call from the refused turn runs. generateText and streamText treat it like other content-policy errors: the fallback chain fires, and with fallbackProviders: [] the call fails. A streamText refusal that arrives after text has streamed ends the stream with an error part and finish reason error instead of falling back, because the refused text has already reached the consumer. Refusals do not count toward the provider-health breaker.

Google Gemini​

export GEMINI_API_KEY=AIza...

The key travels in the x-goog-api-key header, never in the request URL, so proxy and access logs do not record it. A comma-separated list of keys is a key pool: a key that answers HTTP 429, or the 400 that names an invalid key, is taken out and the request moves to the next usable key at once; a streaming request marks the key and the next request draws another.

ModelContextVisionTool CallingNotes
gemini-3.1-pro-preview1MYesYesMost capable ($2/$12 per MTok, $4/$18 once the prompt passes 200K tokens)
gemini-3.8-flash1MYesYesNewest Flash ($0.75/$3.75 per MTok through 2026-12-31, then $1.50/$7.50)
gemini-3.5-flash-lite1MYesYesLow-cost Gemini 3
gemini-2.5-pro1MYesYesPrevious-generation Pro
gemini-2.5-flash1MYesYesFast, large context
gemini-2.5-flash-lite1MYesYesCheapest ($0.10/$0.40 per MTok)

Google has retired gemini-2.0-flash, gemini-2.0-flash-lite and gemini-1.5-pro, and requests for them return HTTP 404. A request for gemini-3.1-pro-preview that returns 404 is retried once on gemini-pro-latest, within the call's maxAttempts: the alias gets the attempts the pinned request left, and with maxAttempts: 1 the 404 ends the call. On gemini-3.1-pro-preview, effort sets the thinking level (low, medium or high; xhigh and max send high), and other models keep the API default. A thinkingConfig object passed through customModelParams takes precedence: thinkingBudget on Gemini 2.5, thinkingLevel on Gemini 3. With thinkingConfig.includeThoughts, the model's thought summaries come back on the provider's message as reasoningText (streamed as reasoningTextDelta chunks), apart from the answer; no provider sends them back. The shared thinking option is Anthropic's and Gemini ignores it. Gemini 3 requires its thought signature back on every tool-call turn; the provider captures and replays it, and sends Google's placeholder when a turn has none, such as a call made by another provider in a fallback chain.

Images go to Gemini in two ways. A data URL is sent inline, read as WHATWG Fetch reads it, and a request carries at most 100 MB of inline data counted in base64 (Google's file limits). An https URL is sent as fileData, and Gemini fetches it itself, up to 15 MB a file (file input methods); the provider fetches nothing. An http URL, a malformed data URL and too much inline data are refused before the request, with HTTP status 400, so they count as the caller's error and not against Gemini's health. A prompt Gemini blocks (promptFeedback.blockReason) and an answer its filters stop with no content (SAFETY, RECITATION, BLOCKLIST, PROHIBITED_CONTENT, SPII and the IMAGE_ reasons) throw a content_filter error that names the reason.

Groq​

export GROQ_API_KEY=gsk_...

provider: 'groq' without a model uses llama-3.3-70b-versatile. Groq shut that model and llama-3.1-8b-instant down for its free and developer plans on 2026-08-16 and serves them only under enterprise contracts, and it shut mixtral-8x7b-32768 down on 2025-03-20 (Groq deprecations, read 2026-10-12). On other plans, name a model from Groq's model list, such as openai/gpt-oss-120b, which Groq recommends in place of Llama 3.3 70B ($0.15 input and $0.60 output per 1M tokens on 2026-10-12):

import { generateText } from '@framers/agentos';

const { text } = await generateText({
  provider: 'groq',
  model: 'openai/gpt-oss-120b',
  prompt: 'Hello',
});

Groq runs its models on its own LPU hardware; its model list gives each model's speed in tokens per second.

Together AI​

export TOGETHER_API_KEY=...
ModelContextVisionTool CallingNotes
meta-llama/Llama-3.3-70B-Instruct-Turbo128KNoYesDefault
openai/gpt-oss-120b128KNoYesAlways reasons
zai-org/GLM-5.3-Flash1MNoYesThinking on by default

Together serverless offers no embedding models.

Mistral AI​

export MISTRAL_API_KEY=...
ModelContextVisionTool CallingNotes
mistral-large-latest256KNoYesMistral Large 3 (mistral-large-2512); the default
codestral-latest128KNoYesCodestral (codestral-2508), for code
mistral-small-latest256KNoYesMistral Small 4 (mistral-small-2603)

Each alias's model and context window are from its page in Mistral's model list, read 2026-10-12.

xAI (Grok)​

export XAI_API_KEY=xai-...

provider: 'xai' without a model uses grok-2. xAI's model list names neither grok-2 nor grok-2-mini on 2026-10-12, so name a model from that list.

OpenRouter​

export OPENROUTER_API_KEY=sk-or-...

OpenRouter is a multi-provider proxy that routes to 200+ models. Specify the model using the provider/model format:

import { agent } from '@framers/agentos';

const myAgent = agent({
  provider: 'openrouter',
  model: 'anthropic/claude-sonnet-4-5-20250929',
});

Popular OpenRouter models:

  • openai/gpt-4o
  • anthropic/claude-sonnet-4-5-20250929
  • google/gemini-2.5-flash
  • meta-llama/llama-3.3-70b-instruct

Requesty​

export REQUESTY_API_KEY=...

Requesty is an OpenAI-compatible gateway at https://router.requesty.ai/v1 (REQUESTY_BASE_URL overrides it). Name models as vendor/model:

import { agent } from '@framers/agentos';

const myAgent = agent({
  provider: 'requesty',
  model: 'anthropic/claude-sonnet-4-5-20250929',
});

provider: 'requesty' without a model uses openai/gpt-4o.

LiteLLM​

LiteLLM is a proxy you host; it exposes the models you configure on it through an OpenAI-compatible API. AgentOS reaches it with provider: 'litellm' and three explicit fields, because it reads no environment variable for this provider and has no default model for it:

import { generateText } from '@framers/agentos';

const { text } = await generateText({
  provider: 'litellm',
  model: 'anthropic/claude-sonnet-4-6',     // a model name your proxy serves
  apiKey: process.env.LITELLM_API_KEY,      // the proxy's master or virtual key
  baseUrl: 'http://localhost:4000/v1',      // the default when omitted
  prompt: 'Hello',
});

provider: 'litellm' without model throws Unknown provider "litellm", and without apiKey it throws No API key for litellm. The provider class has a default model of its own, gpt-4o-mini, used only when the class is driven directly without a model id; generateText(), streamText() and agent() never reach it. LiteLLM is not part of auto-detection or of the auto-built fallback chain; add it to fallbackProviders with its model, apiKey and baseUrl to use it as a fallback.

Ollama​

export OLLAMA_BASE_URL=http://localhost:11434

Run any open model locally. No API key, no cost, full privacy.

# Pull models manually
ollama pull llama3.2
ollama pull codellama
ollama pull dolphin-mixtral
ModelParametersContextTool CallingNotes
llama3.23B/8B128KPartialGeneral-purpose
codellama7B/13B/34B16KNoCode-optimized
dolphin-mixtral8x7B32KNoUncensored
mistral7B32KPartialFast
phi33.8B128KNoSmall, fast

Programmatic Configuration​

Provider + Model + Auth​

The agent factory accepts provider, model, apiKey, and baseUrl directly. There is no separate LLMProviderConfig type — these fields live on AgentOptions (and on BaseAgentConfig, so every sub-agent in an agency() roster takes the same fields).

import { agent } from '@framers/agentos';

const myAgent = agent({
  provider: 'anthropic',
  model: 'claude-sonnet-4-5-20250929',
  apiKey: process.env.ANTHROPIC_API_KEY,        // optional override
  baseUrl: undefined,                           // optional custom base URL
});

Per-Call Overrides​

generate() and stream() accept the same provider/model fields as a per-call override on top of the agent's base config — useful for sending one specific question through a different provider:

const result = await myAgent.generate(
  'Run this complex analysis as a one-off.',
  {
    provider: 'openai',
    model: 'gpt-4o',
  },
);

Operation id across retries​

OpenAIProvider, AnthropicProvider and OpenRouterProvider can send each call's operation id in a header you name. A server in front of the provider, such as a proxy that meters spend, reads it to tell a retry of a call from a new call.

# Every request from these providers carries x-operation-id: <uuid>
export AGENTOS_OPERATION_ID_HEADER=x-operation-id
import { AnthropicProvider } from '@framers/agentos/core/llm/providers/implementations/AnthropicProvider';

const anthropic = new AnthropicProvider();
await anthropic.initialize({
  apiKey: process.env.ANTHROPIC_API_KEY!,
  baseURL: 'https://proxy.example.com',
  operationIdHeader: 'x-operation-id', // wins over AGENTOS_OPERATION_ID_HEADER
});
  • The provider makes the id, a UUID, once before a call's first attempt, unless the call names its own in the operationId option. Every retry of that call sends the same value: a retry after a 429, a 5xx or a network failure, an Anthropic stream that ended before its last event, and a hop to another key of the pool.
  • The next call gets a new id. So does a request sent again with a changed body: the request generateObject sends again with a raised token budget after a truncated answer, and OpenRouter's schema request sent again without its schema, which keeps a call's own operationId when it names one.
  • On a SpendBudget with a spend meter, the generation helpers and a launched team's seats pass each provider call the operation id its reservation is under, so the header carries the id the meter holds the call by (Cost Optimization). embedText() sends its requests itself and puts that id on each of them, in the header AGENTOS_OPERATION_ID_HEADER names.
  • Streamed calls, embeddings (OpenAI, OpenRouter), the key probe's requests and OpenRouter's health check carry the header too; each probe request has an id of its own. An embeddings call that names an operationId sends it in place of the id the provider makes.
  • GeminiProvider sends the header on embeddings only, and only when the call names an operationId, which it sends on every batch and retry, under its operationIdHeader or else the name AGENTOS_OPERATION_ID_HEADER gives. Its completions send none.
  • operationIdHeader in the provider's config names the header; when it is unset the provider reads AGENTOS_OPERATION_ID_HEADER at each call. A blank value counts as unset. With neither, no header is sent.
  • On OpenAI, the id replaces a customHeaders entry of the same name in any case (X-Operation-Id for x-operation-id), so each request carries one value. The other custom headers are sent as before.
  • A name that is not an HTTP header name, or one the provider sets itself (Authorization, x-api-key, x-goog-api-key, Content-Type, anthropic-version, anthropic-beta, OpenAI-Organization, HTTP-Referer, X-Title, User-Agent), is refused with a RangeError before anything is sent: by the provider's initialize(), by each call, and by probeApiKey() and so probeProviderKey(). A key probe answers for the key; a refused name is the host's error, so the probe rejects rather than answering unknown.

Adding a Custom Provider​

AIModelProviderManager builds providers from a fixed list of ids, skips an id it does not know, and has no method that registers another IProvider class. There are three ways to reach a model the list does not cover:

  1. An OpenAI-compatible endpoint. Point the openai provider at it with baseUrl (or OPENAI_BASE_URL):

    import { generateText } from '@framers/agentos';

    const { text } = await generateText({
      provider: 'openai',
      baseUrl: 'https://llm.internal.example.com/v1',
      apiKey: process.env.INTERNAL_LLM_KEY,
      model: 'my-model',
      prompt: 'Hello',
    });

    The endpoint must answer GET /models: the provider lists the models when it initializes, and a listing that fails fails the call. A baseUrl that is empty or whitespace alone is unset: OPENAI_BASE_URL applies, and with neither the request goes to https://api.openai.com/v1; no request is built on a blank base.

  2. A gateway. OpenRouter, Requesty and a self-hosted LiteLLM proxy each put many vendors behind one provider id.

  3. A new provider in AgentOS. Implement IProvider (providerId, isInitialized, initialize(), generateCompletion(), generateCompletionStream(), generateEmbeddings(), listAvailableModels(), getModelInfo(), checkHealth(), shutdown()) next to the classes under src/core/llm/providers/implementations/, add its id to the manager and to provider-defaults.ts, and open a pull request. Adding an LLM Provider lists what a provider pull request must include.

The interface is importable for typing as import type { IProvider } from '@framers/agentos/core/llm/providers/IProvider'.

Provider Capabilities Detail​

Tool Calling Support​

The Structured Output column is the provider-side enforcement that generateObject() and a session's responseSchema request. generateObject() also writes the schema into the system prompt on every provider. A session's responseSchema, on either agent runtime, writes it there only when the provider-side format carries no schema: on a provider whose entry is "None", on an Anthropic model whose schema neither the JSON output format nor a forced tool carries (a record schema on a model that refuses a forced tool choice), and in JSON-object mode. Each fallback provider is checked for its own format, and so is the provider a call is sent to: a session send that agent({ router }) moves to another provider or model is sent the format of the provider and model it goes to, and the schema in its system prompt when that format carries none, so a send moved from Anthropic to an OpenRouter model goes out with OpenRouter's json_schema or JSON-object format rather than an Anthropic payload OpenRouter does not forward. A call on Anthropic with no Anthropic key or base URL goes to OpenRouter, whether the caller names Anthropic, a router picks it or a fallback leg names it, and is sent OpenRouter's format there, with the key and base URL a call that names OpenRouter is sent with (a setDefaultProvider() default that names OpenRouter, else OPENROUTER_API_KEY and OPENROUTER_BASE_URL). A fallback entry whose model id carries another provider's prefix ({ provider: 'anthropic', model: 'openrouter:anthropic/claude-sonnet-4.5' }) is sent the format of the provider the prefix names, and onFallback, the llm.fallback_fired line (its fallbackProvider and fallbackModel) and the leg's hop in result.fallback.hops name that provider and the model id after the prefix (openrouter, anthropic/claude-sonnet-4.5). An Anthropic entry the credential layer sends to OpenRouter ({ provider: 'anthropic', model: 'claude-sonnet-4-5' } with no Anthropic key or base URL and OPENROUTER_API_KEY set) is named the same way, openrouter and anthropic/claude-sonnet-4-5, in both generateText() and streamText(): in onFallback, the llm.fallback_fired, llm.fallback_succeeded and llm.fallback_failed_after_output lines and the hops; its breaker read is OpenRouter's, its capability check judges anthropic/claude-sonnet-4-5, and an llm.fallback_skipped line names the provider and model (fallbackProvider, fallbackModel) the entry would have been sent as. An excluded model (routerParams.excludedModelIds, a bare id or a provider:model id) skips every entry that names it as written, as the model resolves (the provider's default for an entry with no model, the id after an anthropic: or anthropic/ prefix) or as it is sent (anthropic/<model> on OpenRouter), in generateText(), streamText() and the completion gateway. When a leg's own walk serves on a later entry, the outer llm.fallback_succeeded and llm.fallback_failed_after_output lines name the provider that served, in both walkers. The completion gateway's onHopSkipped report names the entry as written (provider, model) and the provider and model it would have been sent to (sentAsProvider, sentAsModel), the names its onFallback and onHopFailure reports use. A send that goes to the agent's own provider and model keeps the agent's format. streamObject() sends no provider-side format on any provider: its schema travels in the system prompt only. All three validate the reply against the caller's Zod schema, and send() throws an ObjectGenerationError when the reply is not matching JSON.

On Anthropic, anthropicSchemaRoute(modelId, schema) (model-strict-tool-use.ts) picks one of three routes for the model and the schema. The native JSON output format (output_config.format with type: json_schema) goes to every model that has it (Opus 4.5 and later, Sonnet 4.5 and later, Haiku 4.5 and later, Fable, Mythos) when the schema fits the vendor's limits: an object root with properties, additionalProperties: false on every object (stamped on the wire copy), no $ref, oneOf, not or record, at most 16 union-typed and 24 optional properties. minLength, maxLength and pattern go out as written; minItems above 1, maxItems, numeric bounds and a format outside the vendor's list are stripped from the wire copy, and the caller's Zod schema re-validates the answer with every constraint. The model answers JSON in its text block; the format rides with adaptive thinking and beside a direct caller's tools, with no beta header, and the vendor's union-typed and optional property limits count a direct caller's strict: true tools with the format's schema. A forced tool call whose input is the schema, sent without strict and with tool_choice forced on it, carries a schema the format cannot carry (a record, say) on every model that accepts a forced tool choice, Claude Sonnet 4.6 (the provider's default text model), Opus 4.5 to 4.8, Opus 5, Sonnet 4.5, Sonnet 5, Haiku 4.5 and Haiku 5.5 among them. It carries every schema on Claude Opus 4 and Claude Sonnet 4 (the May 2025 ids, retired on 2026-06-15), which have no JSON output format. The call's input is the answer: it becomes the message's JSON text and leaves tool_calls, so generateText()'s tool loop never runs it as a tool. On a streamed call the text is the input as the model wrote it (the streamed input deltas joined), so a key the model repeated reaches generateObject()'s parse, which refuses it and asks again. A single-shot call (streamCompletions: false) receives the input already parsed by the API, which has kept only a repeated key's last value, and its text is that input written out with JSON.stringify. A call to the tool a request forces by name (a caller's toolChoice: { type: 'function', function: { name } }, as StructuredOutputManager's function_calling strategy sends) keeps its arguments as the stream delivered them, for the same reason; every other call's arguments are its parsed input written out. A passthrough customModelParams.tools array replaces the schema tool, so it must hold a tool of the schema's name for the forced choice to go out; without one the schema rides the prompt. A passthrough tool_choice keeps precedence too: one under which the model need not call the schema tool (auto, none, another tool's name) sends the schema in the prompt, and any keeps the forced tool, since under any the model calls a tool and writes no text: a call to the schema tool is the answer, and a call to another tool stays a tool call. The format and prompt routes send no schema tool, so a passthrough tool_choice they leave with no tool on the wire is dropped. When neither route carries the schema (such a schema on Claude Opus 5.5, Sonnet 5.5, Fable or Mythos 5.1, which refuse a forced tool choice), and when a request on the forced-tool route turns thinking on, which a forced tool choice cannot ride beside, the provider puts the schema into the system prompt itself as one block after the caller's ("Answer with JSON only, matching this schema: ...") and warns once per model and reason. The schema tool is never sent clamped to auto, and thinking moves only the forced-tool route: a request on the format route keeps the format with thinking on. A request that enables citations leaves the format, which the API refuses beside citations, for the forced tool, or for the prompt on a model that refuses a forced choice. Every completion and stream chunk of a schema request names the route it took (schemaRoute), and one whose schema rode the prompt says schemaInPayload: false, so a completion gateway lifts the answer from where the request put it. A cut-off answer (max_tokens) keeps finishReason: 'length', even when its JSON parses. The API sends none of the input of a forced call the limit stops (the single-shot response gives it {}, the stream one empty input delta; probed on 2026-10-19 on Claude Sonnet 4.6 and Haiku 5.5), so the provider answers that call with no text, never a {} that would pass for an answer under a schema whose fields are all optional, and answers a forced call cut mid-JSON with its raw input as the text, in one request. generateObject() raises its output budget and asks again when the answer does not parse or validate, and returns one that does with finishReason: 'length'; a completion gateway lifts nothing from a cut-off, and a session's send() on the GMI runtime rejects naming the cut-off, whatever the JSON. Two cautions from the vendor's page: a schema property that asks for the model's reasoning may draw a refusal, and the capitalization of enum and const values is not guaranteed, so generateObject() repairs an enum or literal value that differs from the schema's only by case once before it spends a retry.

The Tool Choice column is what the provider does with the toolChoice option ('auto', 'none', 'required', or { type: 'function', function: { name } }).

ProviderStructured OutputTool Choice
OpenAIjson_schema with strict: true when the schema fits OpenAI's strict rules, otherwise JSON-object modeSent as tool_choice
AnthropicThe native JSON output format (output_config.format) on every model that has it and whose schema fits its limits; a non-strict forced tool call whose input is the schema for a schema the format cannot carry on a model that accepts a forced choice, and for every schema on Claude Opus 4 and Sonnet 4; the provider's own schema block in the system prompt when neither carries the schema, and on the forced-tool route when the request turns thinking on, a passthrough tools drops the schema tool, or a passthrough tool_choice of auto, none or another tool's name lets the model skip it (the format route keeps the format with thinking on)'required' is sent as any and a named function as tool; 'none' is sent as auto. A model that rejects a forced choice gets auto
GeminiresponseSchema with JSON outputNot sent; the API default applies
OpenRouterStrict json_schema when the schema fits, otherwise JSON-object mode. When OpenRouter finds no endpoint that serves the strict schema (HTTP 404, "No endpoints found"), the request is sent once more in JSON-object mode with the schema written into its system prompt, unless the system prompt states a schema alreadySent as tool_choice; the routed model decides
RequestyNoneSent as tool_choice; the routed model decides
Groq, Together, Mistral, xAI, LiteLLMNoneSent as tool_choice; the vendor decides which values it accepts
OllamaNoneNot sent
Claude Code CLI, Gemini CLINoneWritten into the prompt as an instruction; 'none' leaves the tool schemas out

Embedding Support​

ProviderModelsDimensionsBatch Size
OpenAItext-embedding-3-small, text-embedding-3-large256–30722048
Geminigemini-embedding-001, gemini-embedding-23072100
Mistralmistral-embed1024512
Ollamanomic-embed-text, mxbai-embed-large768–1024512

embedText() posts an OpenAI, OpenRouter, Mistral or Requesty call to that provider's own /embeddings endpoint, or to the base URL the caller set, so a key never goes to another provider's host. It refuses Anthropic, Groq, Together, xAI, MiniMax, the image providers and the CLIs before their key is read, and a provider id agentos does not know needs baseUrl. embedText() lists each provider's endpoint and the documentation it comes from.

Prompt caching​

Provider-side prompt caching (Anthropic explicit breakpoints, OpenAI and Gemini automatic, OpenRouter forwarding) is on by default with per-call overrides — see Prompt Caching.