Skip to main content

High-Level API

Everything is one import. Pick the function that fits your task:

import {
  generateText, streamText,        // Text generation
  generateObject, streamObject,    // Structured output (Zod validated)
  generateImage, transferStyle,    // Image generation & style transfer
  removeBackground,                // Subject cut-out on a transparent PNG
  generateVideo, analyzeVideo,     // Video generation & analysis
  generateMusic, generateSFX,      // Audio generation
  performOCR,                      // Vision / OCR
  embedText,                       // Embeddings
  agent,                           // Multi-turn agent sessions
  agency,                          // Multi-agent teams
  probeProviderKey,                // Whether one API key works
} from '@framers/agentos';

Quick Reference​

FunctionWhat it doesExample
generateText()One-shot text generationawait generateText({ provider: 'openai', prompt: '...' })
streamText()Stream text in real-timefor await (const d of streamText({...}).textStream) {}
generateObject()Extract structured JSON (Zod)await generateObject({ schema: z.object({...}), prompt: '...' })
generateImage()Generate images (with character consistency)await generateImage({ provider: 'openai', prompt: '...' })
transferStyle()Restyle an image after a reference image, sending the provider bothawait transferStyle({ image: src, styleReference: ref, prompt: '...' })
removeBackground()Cut the subject out of an image onto a transparent PNG (fal BiRefNet)await removeBackground({ image: 'https://example.com/hero.png' })
generateVideo()Generate video from text/imageawait generateVideo({ prompt: '...' })
generateMusic()Generate musicawait generateMusic({ prompt: '...' })
performOCR()Extract text from imagesawait performOCR({ imagePath: './doc.png' })
embedText()Generate embeddingsawait embedText({ input: ['hello'] })
agent()Multi-turn sessions with history, tools and hooksconst a = agent({ provider: 'openai' })
gmi()agent({ runtime: 'gmi' }): sessions served by GMIs, with cognitive memory, sentiment tracking and metaprompts per profileconst a = gmi({ provider: 'openai', cognition: 'full' })
souledAgent()Soul-file agent whose long-term memory is its memory/ wikiawait souledAgent({ provider: 'anthropic', soul: '~/.agentos/agents/aria' })
agency()Multi-agent teamsconst team = agency({ agents: {...}, strategy: 'parallel' })
probeProviderKey()Whether one API key is live, out of credit, rejected or throttled, by its provider's billing probeawait probeProviderKey({ provider: 'anthropic', apiKey })

All functions accept provider as a top-level key.

Provider Resolution​

Calling Styles​

Two styles for specifying provider and model:

// 1. Provider-first — AgentOS picks the best default model for the task
await generateText({ provider: 'openai', prompt: '...' });

// 2. Provider + explicit model — full control
await generateText({ provider: 'anthropic', model: 'claude-sonnet-4-5-20250929', prompt: '...' });

Provider Defaults​

When you supply provider without an explicit model, AgentOS resolves the default model for the requested task automatically:

ProviderTypeText defaultImage defaultEmbedding defaultEnv var
openaiCloudgpt-4ogpt-image-2.5-flaretext-embedding-3-smallOPENAI_API_KEY
anthropicCloudclaude-sonnet-4-6——ANTHROPIC_API_KEY (ANTHROPIC_BASE_URL for another endpoint)
geminiCloudgemini-2.5-flash——GEMINI_API_KEY
openrouterCloudopenai/gpt-4o——OPENROUTER_API_KEY
claude-code-cliLocalclaude-sonnet-4-6——which claude
gemini-cliLocalgemini-3.5-flash——which gemini
stabilityCloud—stable-diffusion-xl-1024-v1-0—STABILITY_API_KEY
replicateCloud—black-forest-labs/flux-1.1-pro—REPLICATE_API_TOKEN
groqCloudllama-3.3-70b-versatile——GROQ_API_KEY
togetherCloudmeta-llama/Llama-3.3-70B-Instruct-Turbo——TOGETHER_API_KEY
mistralCloudmistral-large-latest——MISTRAL_API_KEY
xaiCloudgrok-2——XAI_API_KEY
requestyCloudopenai/gpt-4o——REQUESTY_API_KEY
bflCloud—flux-pro-1.1—BFL_API_KEY
falCloud—fal-ai/flux/dev—FAL_API_KEY
minimaxCloudMiniMax-M3.1-Flashimage-01—MINIMAX_API_KEY
ollamaLocalllama3.2—nomic-embed-textOLLAMA_BASE_URL
stable-diffusion-localLocal—v1-5-pruned-emaonly—STABLE_DIFFUSION_LOCAL_BASE_URL

When neither provider nor model is given, AgentOS checks configured runtimes in order (OPENROUTER_API_KEY → OPENAI_API_KEY → ANTHROPIC_API_KEY → GEMINI_API_KEY → GROQ_API_KEY → TOGETHER_API_KEY → MISTRAL_API_KEY → XAI_API_KEY → REQUESTY_API_KEY → MINIMAX_API_KEY → which claude → which gemini → OLLAMA_BASE_URL). An empty or whitespace-only OLLAMA_BASE_URL or STABLE_DIFFUSION_LOCAL_BASE_URL is unset, as the call reads it, so detection goes on to the next runtime. Or call setDefaultProvider({ provider, apiKey }) once at boot to skip env vars entirely; every subsequent function inherits that default while still letting inline apiKey win when supplied.

MiniMax text requests default to MiniMax-M3.1-Flash; the current catalog also includes MiniMax-M3. Both accept image input and tools. Explicit model IDs, including saved legacy or custom IDs, remain unchanged. Set MINIMAX_API_KEY and optionally MINIMAX_BASE_URL (https://api.minimax.io/v1 by default, or https://api.minimax.cn/v1 for China, the root MiniMax's China documentation names; the older China root https://api.minimaxi.com/v1 is accepted too). For example:

await generateText({ provider: 'minimax', prompt: 'Explain this briefly.', effort: 'high' });
await generateText({ model: 'minimax:MiniMax-M3', prompt: 'Explain this briefly.' });

Flash always uses adaptive thinking. Its allowed effort values are low, medium, high, xhigh, and max; omitting effort leaves the server default in place. thinking: false, service_tier, and reasoning_split: false are rejected for Flash. Tool continuations preserve separate reasoning content, and image URL/data-URL parts are forwarded through the Chat Completions API. See the MiniMax OpenAI API reference. MiniMax image generation uses image-01, also when setDefaultProvider() names a MiniMax text model such as MiniMax-M3.1-Flash: on the global root and on either China root, an image call drops the chat model the default names and takes the provider's image model. A root counts as one in any letter case of its scheme and host, as fetch lowercases them. An empty or whitespace-only baseUrl on the call or the default is unset, so MINIMAX_BASE_URL applies as it does for a call with no baseUrl: a MiniMax root there takes image-01, a custom endpoint there keeps the default's model, and with no variable the call goes to the global root and takes image-01 there too. MiniMax publishes no embedding model, so the minimax provider and embedText() refuse an embedding request before sending it; embed with another provider.

embedText() embeds on openai, openrouter, mistral, requesty, gemini and ollama, each at its own endpoint or at the baseUrl you set, and sends a key to no other host. It refuses a provider with no embeddings (anthropic, groq, together, xai, minimax, the image providers and the CLIs) before reading its key, and needs baseUrl for a provider id agentos does not know (embedText()).

Inline API Keys​

Every function accepts apiKey and baseUrl as top-level parameters, overriding the corresponding environment variable for that call:

// Pass apiKey directly — useful for multi-tenant apps, tests, or dynamic config
await generateText({
  provider: 'openai',
  apiKey: 'sk-my-specific-key',     // overrides OPENAI_API_KEY
  prompt: 'Hello world',
});

// Works on agent() and agency() too
const bot = agent({
  provider: 'anthropic',
  apiKey: process.env.CUSTOMER_KEY,  // per-customer key
  instructions: 'You are a helpful assistant.',
});

A baseUrl is read with surrounding whitespace removed, and one that is an empty or whitespace-only string is unset: the call reads the global default's URL, then the provider's base-URL variable, as it does with no baseUrl, and with none it goes to the provider's own endpoint. The same reading applies to the default's baseUrl and to the variable (OPENAI_BASE_URL= leaves OpenAI's own endpoint), so no request is built on a blank base, whose relative URL (/chat/completions) Node's fetch rejects and a browser's would resolve against the page's own origin, with the key. A blank baseUrl names no endpoint, so it is not refused where a base URL meant for one provider would be: beside a model whose provider: prefix names another provider than provider, and on an image call that names no provider, the call runs as it does with no baseUrl. A key there is still refused.

Local Providers​

Local providers don't require API keys — just a baseUrl (or the corresponding env var):

// Ollama — runs any GGUF model locally
await generateText({
  provider: 'ollama',
  model: 'llama3.2',
  prompt: 'Explain quantum entanglement simply.',
  baseUrl: 'http://localhost:11434', // or set OLLAMA_BASE_URL
});

// Anthropic fallback: if ANTHROPIC_API_KEY is unset but OPENROUTER_API_KEY is set,
// AgentOS automatically routes anthropic requests through OpenRouter.

generateText()​

import { generateText } from '@framers/agentos';

// Provider-first: AgentOS picks the default model for the provider.
const { text, usage } = await generateText({
  provider: 'openai',
  prompt: 'Summarize the TCP three-way handshake in 3 bullets.',
});

console.log(text);
console.log(usage.totalTokens);

generateText({ tools }) and streamText({ tools }) accept three forms:

  • A named high-level tool map
  • An ExternalToolRegistry (Record, Map, or iterable)
  • A prompt-only ToolDefinitionForLLM[]

External registries are exposed to the model and executed when called. Prompt-only ToolDefinitionForLLM[] are exposed to the model too, but if the model calls one without an executor attached, AgentOS returns an explicit tool error instead of silently no-oping.

In the native tool loop of either call, the model's next step receives a tool's output as JSON. A tool that succeeds with no output, or with one JSON leaves out (a function, a symbol, a toJSON that returns nothing), is sent as the empty string's JSON ("") and recorded as a success. A tool whose output cannot be written as JSON (a getter or a toJSON that throws, a cycle) fails with the error the write threw, in the native loops and the prompt-tool loop alike, and an agency seat's mask does not change that.

The same tools forms work on agent({ tools }) and agency({ tools }). When an agency-level tool set is combined with per-agent tools, AgentOS normalizes both sides first and then merges by tool name, with the per-agent tool winning on collisions.

Persist helper usage for later inspection:

import { generateText, getRecordedAgentOSUsage } from '@framers/agentos';

await generateText({
  provider: 'openai',
  prompt: 'Summarize QUIC in one sentence.',
  usageLedger: {
    enabled: true,
    sessionId: 'demo-session',
  },
});

const totals = await getRecordedAgentOSUsage({ enabled: true, sessionId: 'demo-session' });
console.log(totals.totalTokens);

streamText()​

import { streamText } from '@framers/agentos';

const result = streamText({
  provider: 'openai',
  prompt: 'Stream a short explanation of how TLS differs from TCP.',
});

for await (const delta of result.textStream) {
  process.stdout.write(delta);
}

console.log(await result.text);

Cancellation​

import { generateObject } from '@framers/agentos';
import { z } from 'zod';

try {
  const { object } = await generateObject({
    provider: 'openai',
    schema: z.object({ planet: z.string() }),
    prompt: 'Which planet in the solar system is the largest?',
    // Give up after eight seconds: the request is cancelled, and no retry or fallback runs.
    abortSignal: AbortSignal.timeout(8000),
  });
  console.log(object.planet);
} catch (error) {
  if (error instanceof DOMException && error.name === 'TimeoutError') {
    console.log('No answer within eight seconds.');
  } else {
    throw error;
  }
}

generateText(), streamText(), generateObject(), streamObject() and embedText() take an abortSignal, and so do an agent session's send(input, { abortSignal }) and stream(input, { abortSignal }). When the signal aborts:

  • The provider request in flight is cancelled. A non-streamed request to OpenAI, Anthropic, Gemini, OpenRouter, Requesty or Ollama is aborted at once, whether it waits for its answer or its headers, and so is a stream on any of them, a body read that has stalled included; the stream ends with its abort chunk. The OpenAI, OpenRouter, Requesty, Ollama and Gemini providers' generateEmbeddings() abort their request as well (on OpenRouter and Requesty the model-list read made first for a model not yet listed too, and an already-aborted call sends nothing), and embedText() aborts its request to the embeddings endpoint, Ollama or Gemini. Aborting ends the caller's wait, not always the vendor's work: OpenRouter stops processing and billing only for a stream on the providers it lists, and keeps processing and bills a non-streamed request in full (OpenRouter: stream cancellation). A provider that does not read the signal runs its request to the end, and the call still ends as below.
  • A provider waiting between retries stops waiting at once: the OpenAI, Anthropic, Gemini and OpenRouter providers end a 429's Retry-After (up to an hour) or the backoff after a 5xx, Anthropic's 529 or a network failure the moment the signal aborts, and send no further request.
  • No further step, retry or fallback hop starts, and generateObject() starts no further attempt. A tool round that generateText() or streamText() has started runs to its end first; the call then ends as below rather than returning that round as its result. The call is not counted against the provider's health.
  • The call rejects with the signal's reason: a DOMException named AbortError after controller.abort(), one named TimeoutError from AbortSignal.timeout(), or the value given to controller.abort(reason). streamText() and a session's stream() end with an error part that carries the reason (in an Error when the reason is not one), and streamObject()'s object rejects with the reason.
  • A call whose signal has already aborted sends no request.

A session stays open after an aborted send() or stream(), and its next call runs as usual. Without runtime: 'gmi', the aborted turn is added to no history. With it, the signal is joined with the session's own stop: the turn ends as one close() stops (the steps it finished stay in messages(), marked partial), a turn that had already failed when the signal aborted keeps its own error, and close() still stops a turn sent with a signal of its own. An agent's generate() and stream() pass an abortSignal in their options to generateText() and streamText(); with runtime: 'gmi' they take one too and end the same way (GMIs from agent()).

import { agent } from '@framers/agentos';

const session = agent({ provider: 'openai' }).session('support');
const turn = new AbortController();

const pending = session.send('Draft a reply to the last message.', { abortSignal: turn.signal });
turn.abort(); // the conversation moved on
await pending.catch((error) => console.log(error.name)); // 'AbortError'

Attempts​

import { generateText, type AttemptOutcome } from '@framers/agentos';

const outcomes: AttemptOutcome[] = [];
const { text } = await generateText({
  provider: 'openai',
  prompt: 'Name the largest moon of Saturn.',
  // One request: a 429, a 5xx, a timeout or a network failure throws, with no retry.
  maxAttempts: 1,
  onAttempt: (outcome) => outcomes.push(outcome),
});
console.log(text, outcomes); // ... [{ attempt: 1, status: 200, ok: true }]

generateText(), streamText(), generateObject(), streamObject() and embedText() take maxAttempts and onAttempt, which reach the provider as ModelCompletionOptions:

  • maxAttempts is the most requests one call sends, a first and its retries together; the provider's configured count holds when it is left out (maxRetries, 3, on the OpenAI, Anthropic and Gemini providers; 3 on OpenRouter). A count that is not a whole number of 1 or more is refused with a RangeError before anything is sent. The OpenAI provider counts each call's attempts with CountingAttemptBudget, the attempt authorizer the voice platform calls before each of its requests (Compound credit), so an attempt past the count is refused before it is sent. The Anthropic, Gemini and OpenRouter providers bound their retry loops by the same count, and none of the four waits a backoff after its last attempt, so maxAttempts: 1 on a judge or classifier call turns a 429, a 5xx or Anthropic's 529 into an error at once, for the fallback chain or the caller to handle. A key hop inside an attempt (Key rotation) is not counted. The Anthropic and Gemini streams send one request and retry nothing but key hops, and still refuse an invalid count before it is sent; OpenRouter's stream retries under the count. A request sent again on another route after a 404, a retired Gemini id on its alias or an OpenRouter request without its schema (when no endpoint takes json_schema), draws on the same count, the configured one when the call names none: it gets the attempts the first request left, and with none left the 404 ends the call. The other providers keep their configured retries.
  • onAttempt is reported by the OpenAI provider (and the providers that send through it); the others report nothing. It is called once for each request the call sent, with an AttemptOutcome: attempt, the request's number in the call from 1; status, the HTTP status it answered, or null when none came (a timeout, a network failure, or the caller's abort once the request had begun); and ok. A request refused before it is sent is not reported, and a callback that throws never ends the call. A generateObject() validation retry is a call of its own, so its attempts count from 1 again; embedText() sends each batch once and reports it as attempt 1.

A caller settles against the report: a request whose usage never arrived (a timeout, an error after it was sent, a cut stream) is one the caller knows was sent.

Prompt caching​

import { generateText } from '@framers/agentos';

// A judge: the same rubric on every call, a new candidate each time.
const { text } = await generateText({
  model: 'anthropic:claude-haiku-4-5-20251001',
  system: [{ text: rubric, cacheBreakpoint: true }],
  prompt: candidate,
  // Keep the rubric's cache marker; put no breakpoint on the candidate.
  cache: { tail: false },
});

generateText(), streamText(), generateObject(), agent({ cache }) and a fallback entry take cache, which reaches the provider as ModelCompletionOptions.cache. On Claude, directly and through OpenRouter's anthropic/* slugs, it decides the cache_control breakpoints a request carries:

  • Left out (or { ttl: '5m' }): the provider caches the request up to the newest message, so a conversation reads its history back from the cache on the next turn. A system block with cacheBreakpoint: true keeps its own marker, and the direct provider marks the newest message after it (OpenRouter sends a request that carries a caller marker as it is, a request-level cache_control in customModelParams included).
  • { ttl: '1h' }: the same markers with a one-hour TTL, for loops whose steps are more than five minutes apart.
  • { tail: false }: the markers the caller set go out as set, and the provider puts no breakpoint on the messages. For a call that sends the same long prefix every time and never the same tail (a judge, a classifier): the prefix is read back from the cache, and the tail, which no later call would read, is not written at the cache-write price (1.25x the input price at five minutes, 2x at one hour). With no cacheBreakpoint set, the end of the first system message is marked (the end of a customModelParams.system override when one replaces the system; on OpenRouter, the first system message of a customModelParams.messages override). It combines with ttl.
  • false: no marker at all, the caller's included (on messages, tool definitions and a request-level cache_control in customModelParams), for a call whose prompt is never sent again.

Prices, floors and the per-provider behavior: Prompt Caching.

agency().stream()​

streamText() is a single-call raw stream. agency().stream() separates raw live chunks from the finalized post-guardrail/post-HITL answer:

import { agency, type AgencyStreamResult } from '@framers/agentos';

const team = agency({
  provider: 'openai',
  strategy: 'sequential',
  agents: {
    researcher: { instructions: 'Collect the key facts.' },
    writer: { instructions: 'Turn the facts into a concise answer.' },
  },
  hitl: {
    approvals: { beforeReturn: true },
    handler: async () => ({
      approved: true,
      modifications: { output: 'Approved for delivery.' },
    }),
  },
});

const stream: AgencyStreamResult = team.stream('Summarize HTTP/3 rollout risks.');

for await (const chunk of stream.textStream) {
  process.stdout.write(chunk); // raw live output
}
process.stdout.write('\n');

for await (const approved of stream.finalTextStream) {
  console.log('Approved answer:', approved);
}

console.log('Agent calls:', await stream.agentCalls);
console.log('Final text:', await stream.text);

Use:

  • textStream for low-latency token UX
  • finalTextStream or text for the finalized approved answer
  • fullStream when you also need structured events like final-output

See Agency API and Streaming Semantics for the full contract.

QueryRouter​

Use QueryRouter when you want grounded answers over a local markdown corpus without booting the full AgentOS runtime.

import { QueryRouter } from '@framers/agentos';

const router = new QueryRouter({
  knowledgeCorpus: ['./docs', './packages/agentos/docs'],
  availableTools: ['web_search'],
});

await router.init();

console.log(router.getCorpusStats());

const result = await router.route('How does memory retrieval work?');
console.log(result.answer);
console.log(result.tiersUsed);
console.log(result.fallbacksUsed);

await router.close();

router.getCorpusStats() returns a QueryRouterCorpusStats snapshot that tells you what is actually live in the current host:

  • corpus size: configuredPathCount, chunkCount, topicCount, sourceCount
  • bundled platform knowledge: platformKnowledge.total plus per-category counts
  • retrieval path: vector+keyword-fallback or keyword-only
  • embedding health: embeddingStatus
  • runtime truth: graphRuntimeMode, rerankRuntimeMode, deepResearchRuntimeMode

Built-in status meanings:

  • embeddingStatus: 'active' means vector embeddings initialized successfully
  • embeddingStatus: 'disabled-no-key' means init stayed keyword-only because no embedding credential was available
  • embeddingStatus: 'failed-init' means vector init was attempted but fell back to keyword-only mode after an error
  • graphRuntimeMode: 'heuristic' means same-document / heading-overlap expansion
  • rerankRuntimeMode: 'heuristic' means the built-in lexical reranker
  • deepResearchRuntimeMode: 'heuristic' means the built-in local-corpus research synthesis path

Hosts can inject real graphExpand, rerank, and deepResearch hooks in the constructor; those modes then become active.

See Query Router for the full contract and host-hook examples.

generateImage()​

import { generateImage } from '@framers/agentos';

// Provider-first: resolves to gpt-image-2.5-flare by default for openai.
const result = await generateImage({
  provider: 'openai',
  prompt: 'A cinematic neon city skyline reflected in rain at night.',
  outputFormat: 'png',
});

console.log(result.provider);
console.log(result.images[0]?.mimeType);

removeBackground()​

import fs from 'node:fs';
import { removeBackground } from '@framers/agentos';

// The subject on a transparent PNG, as a URL on fal's storage.
const { image } = await removeBackground({ image: 'https://example.com/hero.png' });
console.log(image.url);

// The bytes instead, with the matte.
const pngBytes = fs.readFileSync('hero.png');
const { image: cut, mask } = await removeBackground({ image: pngBytes, output: 'buffer', returnMask: true });

The built-in provider runs fal's BiRefNet v2 (fal-ai/birefnet/v2) with FAL_API_KEY. Options:

OptionDefaultWhat it does
image(required)An http(s) URL or a base64 data URL of a PNG, JPEG, WebP or GIF, sent as given, or the bytes (Buffer, Uint8Array) of one. Any other string is refused.
output'url''url' returns the cut-out's URL; 'buffer' returns its buffer and dataUrl (fal is asked to answer inline).
format'png''png' or 'webp'.
returnMaskfalseAlso return the matte as mask; an answer without it rejects.
model'fal-ai/birefnet/v2'The fal endpoint.
timeoutMs60000The call's deadline, download included; rejects with code: 'timeout'.
signalAborts the call with the signal's reason.
apiKey, baseUrlFAL_API_KEY, https://fal.runCredentials for this call.
providerOptions.falvariant (BiRefNet weights, default 'General Use (Light)'), operatingResolution (default '1024x1024'), refineForeground (default true), input (extra request fields; the image, output_mask and sync_mode the call sets stand over them), maxResponseBytes (default 64 MiB).
provider'fal'A provider registered with registerBackgroundRemovalProvider().

Failures reject with SegmentationProviderError (invalid_request, provider_failed or timeout, with the HTTP status of a failed answer). See Image Segmentation.

probeProviderKey()​

import { probeProviderKey } from '@framers/agentos';

const keys = process.env.ANTHROPIC_API_KEY!.split(',').map((key) => key.trim()).filter(Boolean);
for (const apiKey of keys) {
  const { state, detail, fingerprint } = await probeProviderKey({ provider: 'anthropic', apiKey });
  console.log(fingerprint, state, detail);
  // 3f9a1c0b7e2d exhausted no credit or a spend limit reached (HTTP 400): Your credit balance is too low to access the Anthropic API. ...
  // 81c4e6a90d55 ok key accepted: HTTP 200 on claude-haiku-4-5-20251001
}

It sends one key through the same billing probe the provider's key pool sends to recheck a key it took out as unpaid. The probe is the smallest billable request, and its answer is read with classifyKeyFailure(): ok (a 2xx), exhausted (no credit left or a spend limit reached), invalid (a bad, revoked or expired key), throttled (a rate limit, with retryAfterMs when the answer sent one) or unknown (a 5xx, a 403 or 404 for a model the key may not use, no answer within the timeout, a failed connection, the caller's abort, or no model to send to). It reads no key from the environment and builds no pool; it never throws for an answer. The one variable that changes what it sends is AGENTOS_OPERATION_ID_HEADER, which names the header an Anthropic, OpenAI or OpenRouter probe's requests carry an operation id in (Operation id across retries); a name those providers refuse rejects with a RangeError before any request.

OptionDefaultWhat it does
provider(required)anthropic, openai, gemini, openrouter or requesty (PROBEABLE_KEY_PROVIDERS); any other is refused with a RangeError.
apiKey(required)One key, trimmed; an empty key or a comma-separated list is refused with a TypeError.
modelthe probe's modelAnthropic claude-haiku-4-5-20251001, Gemini gemini-2.5-flash, OpenAI the first GPT chat model GET /models lists, else its first o-series model with a price (never a Responses-only one), OpenRouter and Requesty the cheapest chat model their catalog lists with a price above zero. A named OpenAI model that is Responses-only goes to POST /responses.
baseUrlthe provider's endpointThe API base as the provider's baseURL takes it; a blank string is unset.
timeoutMs10000How long the probe waits; a value that is not a positive finite number is refused with a RangeError, and one past 2147483647, the longest timer Node keeps, is cut to it.
signalEnds the probe, which reports unknown.

The result carries state, detail (the state, the HTTP status and the vendor's message, at most 300 characters, with the key replaced by [key] wherever the vendor echoed it; on a failed connection, fetch's error with the key and the base URL's credentials replaced by [redacted]), provider, fingerprint (the key's keyFingerprint()), and httpStatus, model and retryAfterMs when the answer had them. A passing probe costs one request of one output token (16 on OpenRouter, Requesty and OpenAI's reasoning models). See Key rotation.

generateVideo()​

import { generateVideo } from '@framers/agentos';

const result = await generateVideo({
  prompt: 'A drone flying over a misty forest at sunrise.',
  timeoutMs: 180_000,
  onProgress: (event) => console.log(event.status, event.progress, event.message),
  providerPreferences: {
    preferred: ['runway', 'replicate'],
    blocked: ['fal'],
  },
});

console.log(result.provider);
console.log(result.videos[0]?.url);

image routes the call to image-to-video and takes a Buffer, an https:// URL or a data: URL. resolution and referenceImages go to text-to-video and image-to-video alike; resolution is sent in the model's own vocabulary (720p, 1080p, or 512P and 768P for MiniMax on fal). providerOptions carry the model inputs the typed fields do not cover:

const clip = await generateVideo({
  provider: 'fal',
  model: 'fal-ai/minimax/hailuo-02/standard/image-to-video',
  prompt: 'The lantern swings in the wind',
  image: 'https://cdn.example.com/lantern.png',
  resolution: '768P',
  providerOptions: {
    fal: { end_image_url: 'https://cdn.example.com/lantern-end.png', prompt_optimizer: false },
    // Used when Replicate runs as a fallback, on its default Kling 2.5 Turbo Pro.
    replicate: { end_image: 'https://cdn.example.com/lantern-end.png' },
  },
});

Key providerOptions by provider id so each provider in a fallback chain receives only its own entry (an id added with registerVideoProviderFactory() keys a bag as the built-in ids do); a bag with no provider-id key goes whole to fal, Replicate and Gemini. They add inputs and never replace a field the call sets, so to send a model-specific value for such a field, leave the typed field unset and put the value there. Runway sends the prompt, duration, ratio (from aspectRatio) and the source image; it sends no resolution, reference images or provider options. A custom provider registered with registerVideoProviderFactory() receives image as the caller passed it, a Buffer or a string, and videoImageToUrl() turns either into a URL. timeoutMs goes to each provider at initialize(): the polling budget on fal, Replicate and Runway, and the deadline of every request on Gemini, which rounds a fractional value up to whole milliseconds and refuses one past 2147483647, the longest timer Node keeps.

Built-in Video Providers​

ProviderDefault modelAPI key env varImage input
runwaygen3a_turboRUNWAY_API_KEYprompt_image
replicatekwaivgi/kling-v2.5-turbo-proREPLICATE_API_TOKENstart_image (Kling), first_frame_image (MiniMax), else image
falfal-ai/kling-video/v2.5-turbo/pro/text-to-video (its image-to-video for an image)FAL_API_KEYimage_url
geminigemini-omni-1.1-flashGEMINI_API_KEYan inline image part

With no provider, AgentOS picks the first of RUNWAY_API_KEY, REPLICATE_API_TOKEN and FAL_API_KEY that is set and keeps the others as fallbacks. gemini runs only when provider: 'gemini' names it, so a Gemini key set for text never starts a video job.

  • fal sends the image and referenceImages as image_url and reference_image_urls (a URL as given, a Buffer as a data: URL), plus resolution, negative_prompt, aspect_ratio and seed. Kling and MiniMax endpoints take duration as a string ("5", "10"; "6", "10"), and the provider sends it that way. It polls the status_url and fetches the response_url fal returns, and raises a request that ends COMPLETED with an error.
  • Replicate posts { version: model, input }, where model is owner/name, owner/name:version or a version id. It sends referenceImages as reference_images, plus resolution, negative_prompt, duration, aspect_ratio and seed. A bare version id names no model family, so its image goes as image unless providerOptions: { replicate: { imageInputName: 'start_image' } } names the field; imageInputName itself is not sent.
  • Gemini runs Gemini Omni Flash on the Interactions API (POST /v1beta/interactions, key in the x-goog-api-key header). The source image, then the reference images, then the prompt go as input parts; aspectRatio (16:9, 9:16) and resolution (360p, 720p, 1080p, 4k) go in response_format. It fetches an image URL itself through the checked egress path (https on port 443 to a name whose every address is public, the connection pinned to the checked address, each of up to 3 redirects checked the same way, 20 MiB read across a call's images, redirect bodies included), sends the bytes inline without the API key, decodes a base64 or percent-encoded data: URL (counted against the same bound), and sends a Files API URI as a uri part. The provider's own imageEgress: { profile: 'local' } allows http and private addresses for a developer's machine, and maxImageBytes changes the bound (a value past Number.MAX_SAFE_INTEGER, Infinity included, is refused). It returns the video as base64; with providerOptions: { gemini: { response_format: { delivery: 'uri' } } }, which the Omni guide recommends for videos over 4 MB, it polls the Files API file until it is ACTIVE, downloads it with the key, and returns the bytes as base64 with the URI in providerMetadata.fileUri (URI delivery). providerOptions.gemini.background: true is refused before any image is fetched, since the provider does not poll a background interaction. pollIntervalMs sets the pause between Files API polls and, like timeoutMs, is refused past 2147483647. Google shut down the Veo models in the Gemini API (veo-3.0-* on 30 June 2026, the veo-3.1-*-preview models on 22 October 2026) and names gemini-omni-1.1-flash as their replacement (deprecations, Omni guide).

analyzeVideo()​

analyzeVideo() auto-creates a VisionPipeline, uses the structured VideoAnalyzer pipeline under the hood, and auto-wires STT when a supported speech provider credential is available (OPENAI_API_KEY, DEEPGRAM_API_KEY, ASSEMBLYAI_API_KEY, or Azure Speech env vars).

import { analyzeVideo } from '@framers/agentos';

const result = await analyzeVideo({
  videoUrl: 'https://example.com/demo.mp4',
  prompt: 'What is the product demo showing?',
  transcribeAudio: true,
  maxFrames: 12,
});

console.log(result.description);
console.log(result.fullTranscript);
console.log(result.scenes?.length);

Host requirement: ffmpeg and ffprobe must be installed and available on PATH.

generateMusic() and generateSFX()​

import { generateMusic, generateSFX } from '@framers/agentos';

const music = await generateMusic({
  prompt: 'Warm analog synthwave with a slow build.',
  timeoutMs: 180_000,
  onProgress: (event) => console.log('music', event.status, event.message),
  providerPreferences: {
    preferred: ['suno', 'udio'],
  },
});

const sfx = await generateSFX({
  prompt: 'Heavy vault door closing with metallic reverb.',
  timeoutMs: 60_000,
  onProgress: (event) => console.log('sfx', event.status, event.message),
  providerPreferences: {
    preferred: ['elevenlabs-sfx', 'stable-audio'],
  },
});

console.log(music.audio[0]?.url);
console.log(sfx.audio[0]?.url);

Cloud audio providers​

ProviderEndpointMusic defaultSFX defaultAPI key env var
stable-audioPOST https://api.stability.ai/v2beta/audio/stable-audio-2/text-to-audio (multipart)stable-audio-2.5stable-audio-2.5STABILITY_API_KEY
replicate-audioPOST https://api.replicate.com/v1/predictions, or /v1/models/{owner}/{name}/predictions for official modelsmeta/musicgen (version pinned)sepal/audiogen (version pinned)REPLICATE_API_TOKEN
fal-audioPOST https://queue.fal.run/{model} (queue)fal-ai/musicgenfal-ai/stable-audioFAL_API_KEY
  • stable-audio accepts stable-audio-2.5 and stable-audio-2. outputFormat takes mp3 (the default) or wav, and primitive providerOptions such as steps and cfg_scale become form fields.
  • replicate-audio takes owner/name, owner/name:version or a bare version id. Replicate runs community models only by version, so meta/musicgen and sepal/audiogen carry pinned versions, and the retired meta/audiogen id resolves to sepal/audiogen. Any other owner/name, such as stability-ai/stable-audio-2.5, runs on the official-model route. An owner/name:version or bare version id is reported in the result as given. outputFormat wav or mp3 is sent as the model's own output_format value (wav/mp3 for MusicGen and AudioGen, wav_cd_quality/mp3_standard for elevenlabs/music, nothing for stability-ai/stable-audio-2.5); flac, ogg and aac are left to the model's default. The result's MIME type is read from the output URL's extension, then from the format sent, and is audio/mpeg when neither names one. durationSec is sent as duration, or as music_length_ms in milliseconds for elevenlabs/music. seed is sent to every model except AudioGen and elevenlabs/music, which take none. providerOptions add model inputs (for example { model_version: 'stereo-large' } for MusicGen) and never replace the prompt, duration, output format or seed.
  • fal-audio sends seconds_total to Stable Audio models and duration to the others, and providerOptions add model inputs without replacing the prompt, duration or seed. Status polls and the result fetch stay on a custom baseURL. The provider reads the model's audio, audio_file or audio_url output. A request that fails in the queue is raised with fal's error text.
  • When no model is given, generateSFX() runs each provider on its SFX default, not on its music default.
const loop = await generateMusic({
  provider: 'replicate-audio',
  model: 'meta/musicgen',
  prompt: 'Slow ambient pads for a rainy night',
  durationSec: 30,
  outputFormat: 'wav',
  providerOptions: { model_version: 'stereo-large', normalization_strategy: 'peak' },
});

Media Provider Preferences​

Image, video, music, and SFX helpers accept providerPreferences so callers can reorder, block, or weight providers without hard-coding a single backend:

import type { MediaProviderPreference } from '@framers/agentos';

const preferredCloudOnly: MediaProviderPreference = {
  preferred: ['runway', 'replicate'],
  blocked: ['musicgen-local', 'audiogen-local'],
  weights: { runway: 3, replicate: 1 },
};

When weights are present, AgentOS chooses the primary provider from the resolved list using weighted selection and keeps the remaining providers in order as fallbacks.

Built-in Image Providers​

ProviderTypeDefault modelAPI key env var
openaiCloud APIgpt-image-2.5-flareOPENAI_API_KEY
stabilityCloud APIstable-diffusion-xl-1024-v1-0STABILITY_API_KEY
replicateCloud APIblack-forest-labs/flux-1.1-proREPLICATE_API_TOKEN
openrouterCloud API—OPENROUTER_API_KEY
bflCloud APIflux-pro-1.1BFL_API_KEY
falCloud APIfal-ai/flux/devFAL_API_KEY
minimaxCloud APIimage-01MINIMAX_API_KEY
stable-diffusion-localLocalv1-5-pruned-emaonlyNone (uses baseUrl)

Ollama serves text and embeddings only: generateImage({ provider: 'ollama' }) throws Image generation is not supported for provider "ollama".

Provider-Specific Options​

Use the common options for the simple path, then drop down to namespaced providerOptions when you need provider-native controls:

import { generateImage } from '@framers/agentos';

const poster = await generateImage({
  provider: 'stability',
  model: 'stable-image-core',
  prompt: 'An art deco travel poster for a moon colony',
  negativePrompt: 'text, watermark',
  providerOptions: {
    stability: {
      stylePreset: 'illustration',
      seed: 42,
      cfgScale: 8,
    },
  },
});

console.log(poster.images[0]?.mimeType);

Replicate and OpenRouter work the same way:

const replicateResult = await generateImage({
  provider: 'replicate',
  model: 'black-forest-labs/flux-schnell',
  prompt: 'A product photo of a titanium watch on black stone',
  aspectRatio: '16:9',
  providerOptions: {
    replicate: {
      outputQuality: 90,
      input: {
        go_fast: true,
      },
    },
  },
});

Local Image Generation​

Run Stable Diffusion locally without any API key, through an Automatic1111 or Forge WebUI (/sdapi/v1/txt2img) or ComfyUI (/prompt) server:

const sdLocal = await generateImage({
  provider: 'stable-diffusion-local',
  model: 'v1-5-pruned-emaonly',
  prompt: 'A brutalist house in fog',
  baseUrl: 'http://localhost:7860', // or set STABLE_DIFFUSION_LOCAL_BASE_URL
});

Custom Image Provider​

Register a provider factory for backends not covered by the built-ins:

import { generateImage, registerImageProviderFactory, type IImageProvider } from '@framers/agentos';

class ComfyUIProvider implements IImageProvider {
  providerId = 'comfyui';
  isInitialized = false;
  defaultModelId = 'sdxl';

  async initialize() {
    this.isInitialized = true;
  }

  async generateImage(request) {
    return {
      created: Math.floor(Date.now() / 1000),
      modelId: request.modelId,
      providerId: this.providerId,
      images: [{ url: 'https://example.invalid/image.png' }],
      usage: { totalImages: 1 },
    };
  }
}

registerImageProviderFactory('comfyui', () => new ComfyUIProvider());

await generateImage({
  provider: 'comfyui',
  model: 'sdxl',
  prompt: 'A brutalist house in fog',
});

agent()​

import { agent } from '@framers/agentos';

const researcher = agent({
  provider: 'openai',
  instructions: 'You are a concise research assistant.',
  maxSteps: 4,
});

const session = researcher.session('demo');

const first = await session.send('What is QUIC?');
console.log(first.text);

const second = await session.send('Compare it to TCP.');
console.log(second.text);

console.log(await session.usage());

agent({ tools }) accepts the same three forms as generateText({ tools }) and streamText({ tools }): named tool maps, ExternalToolRegistry (Record, Map, or iterable), and prompt-only ToolDefinitionForLLM[].

Citation verification​

Set verifyCitations: { embedFn, retrieve } to check generated claims against sources retrieved for the original user input. retrievalAugmentor can supply both hooks instead. The reply is returned unchanged: verification adds verdicts, not a filter. Missing retrieval wiring, empty sources or verification failure leave grounding undefined.

generate() returns these verdicts in result.grounding. On the default runtime, session send() does the same, including after a successful schema retry. Session stream() exposes result.grounding as a promise; consume the stream before awaiting it, because verification starts only after the text completes. Direct stream() and GMI session calls do not run this check.

const session = agent({
  provider: 'openai',
  verifyCitations: { embedFn, retrieve }, // Supply the embedding and retrieval functions.
}).session();

const reply = await session.send('Summarize the retrieved records.');
const verdicts = reply.grounding?.claims;

const streamed = session.stream('Explain the summary.');
for await (const text of streamed.textStream) process.stdout.write(text);
const streamedVerdicts = await streamed.grounding;

Schema validation and retries​

On the default runtime, agent({ responseSchema }) validates generate()'s final text with the supplied Zod schema and returns the value in result.parsed. It can extract JSON from a code fence. controls.maxValidationRetries defaults to 1, so an invalid reply gets one more attempt. Set it to 0 to throw ObjectGenerationError on the first invalid reply. output is the agency() option: agent() accepts it, does not apply it and names it in its construction warning, so pass an agent's schema as responseSchema. Repair attempts retain the completed transcript, including native tool calls and their results, and run without tools so they cannot repeat an action.

For a session, pass the schema to send() and read result.object:

import { z } from 'zod';

const result = await session.send('Count the records.', {
  responseSchema: z.object({ count: z.number() }),
  maxValidationRetries: 1,
});

Session sends default to zero extra attempts. Each retry includes the failed reply and validation feedback. Only the original user input and final valid reply enter session history; an exhausted send adds nothing. Usage totals count every completed attempt. Retry counts must be non-negative integers. Schema sends omit configured tools and their chain-of-thought instruction, as before. These retry options apply to the default runtime.

For schema calls, memoryProvider.observe records the original user input once and only the accepted answer, after caller hooks and validation. Failed attempts and exhausted retries record nothing. Memory retrieval and caller generation hooks remain active on every attempt.

Working-memory block​

On the default runtime, memory: { working: () => 'STATE: ...' } supplies a system-text block for each session send(). The callback may return a promise. It runs once per send, including sends with schema retries, and its text precedes the configured system text. An empty block is skipped. A thrown error or rejected promise skips the block and warns through the configured logger factory, without logging the error or state text.

The host owns the state. This callback creates no store and does not run on generate(), direct stream(), session stream() or the GMI runtime. The existing working: { enabled, maxTokens, strategy } buffer configuration remains accepted and deferred by the default runtime.

Sessions served by GMIs: runtime: 'gmi'​

import { agent } from '@framers/agentos';

const tutor = agent({
  runtime: 'gmi',
  provider: 'openai',
  instructions: 'You are a patient networking tutor.',
  memory: { embedding: { provider: 'openai', model: 'text-embedding-3-small' } },
});

const session = tutor.session('demo', { userId: 'learner-42' });
await session.send('What is QUIC?');
await tutor.close();

With runtime: 'gmi' (or gmi(), the same function under its own name), every session is a Generalized Mind Instance built from the options, and memory turns on cognitive memory: one manager for the sessions, scoped per user, embedding with memory.embedding (unset, a model picked from the global default provider or the environment's keys). cognition: 'full' adds sentiment tracking and the metaprompt presets. Of the MemoryConfig fields, this path reads embedding, consolidation, persistAccessState and store; it does not read types, working or shared. memory.store ({ path }, a SQLite file, or { adapter, brainId? }, a host's storage adapter) keeps the agent's memory and its learning store there, so an agent built again on it recalls what its users said; it needs learning.agentKey. The default runtime reads only the memory.working callback described above. GMIs from agent() lists what each option does on this path.

learning: true, or a LearningConfig, turns on the agent's learning layer: every GMI of the agent sends the persona's personality section as a system prompt of its own on every turn, in the words agent() writes, a session opened with a userId is bound to a learner when the id is not one the layer reads as anonymous (anonymous_user, guest and the others, with learning.anonymousUserIds) and the agent has a name or learning.agentKey, and agent.learning.manifest() says which loops run. agent.learning also carries the operator's calls (freeze, unfreeze, suspend, unsuspend, operatorSet, revert, exportLearnedState, importLearnedState and forgetUser), with the agent's key and namespace filled in. Unset or false (the default), the agent builds no layer and the persona's base prompt carries the section; agent() without runtime: 'gmi' ignores the option. Learning has the rules.

Per-agent identity via SOUL.md​

Pass a soul: option to give an agent its identity from a SOUL.md workspace. agent() builds its system prompt from the SOUL.md body first, then a ## Style section from STYLE.md, then the memory/index.md catalog, then instructions, name, personality and skills. A soul that fails to load logs a warning, and the agent starts without it.

// Workspace path: SOUL.md, STYLE.md and memory/index.md shape the prompt
agent({ provider: 'anthropic', soul: '~/.agentos/agents/aria' });

// Direct file path: read as SOUL.md, with the companion files beside it
agent({ provider: 'openai', soul: './personas/aria.soul.md' });

// Inline content: for tests and ephemeral agents
agent({ provider: 'openai', soul: { content: SOUL_MARKDOWN_STRING } });

agent({ soul }) uses the prose only. The frontmatter's structured fields (HEXACO scores, voice, mood, hard limits) land on the personaDefinition that loadSoul() returns, which the full runtime takes as a persona; give an agent() a trait paragraph with personality. See SOUL_FILES.md for the workspace files.

souledAgent(): soul plus a memory/ wiki​

agent({ soul }) loads identity but leaves long-term memory to you. souledAgent() is the async factory that wires the whole loop in one call: it loads the soul, opens one Memory store under the workspace's memory/.store/, and attaches the soul's memory/ markdown wiki as the agent's long-term memory.

import { souledAgent } from '@framers/agentos';

const aria = await souledAgent({
  provider: 'anthropic',
  model: 'claude-sonnet-4-6',
  soul: '~/.agentos/agents/aria',
});

What it wires:

  • Read: the memory/index.md catalog is injected into the system prelude, and the agent opens any page on demand with the read_memory_page tool. The tool is added when tools is absent or an array; a named tool map is passed through as given, without it.
  • Recall: before each call, up to eight traces from the store that match the user's message join the prompt as context.
  • Capture: conversation the agent observes is written to the same store as episodic traces.
  • Fold: those traces are merged into entity/concept pages when memory consolidates. souledAgent runs this on the agent's close(); call await aria.memory.compileWiki() to fold mid-session. Merges integrate new facts without clobbering human edits.

One Memory facade backs both the live memory and the wiki, so the markdown stays the source of truth and the vector/graph index is rebuilt from it. souledAgent() accepts every agent() option and returns the same Agent, plus agent.memory (the store) when the soul resolves to a workspace directory. An inline soul ({ content }) has no workspace, so it falls back to a plain agent(). Closing the agent also closes the store.

Runnable examples in the package source:

Full runtime: AgentOS​

import { AgentOS, AgentOSResponseChunkType } from '@framers/agentos';
import { createTestAgentOSConfig } from '@framers/agentos';

const agent = new AgentOS();
await agent.initialize(
  await createTestAgentOSConfig({
    tools: {
      open_profile: {
        description: 'Load a saved profile record by ID.',
        inputSchema: {
          type: 'object',
          properties: { profileId: { type: 'string' } },
          required: ['profileId'],
        },
        execute: async ({ profileId }) => ({
          success: true,
          output: { profile: { id: profileId, preferredTheme: 'solarized' } },
        }),
      },
    },
  })
);

for await (const chunk of agent.processRequest({
  userId: 'user-1',
  sessionId: 'session-1',
  textInput: 'Explain how TCP handshakes work',
})) {
  if (chunk.type === AgentOSResponseChunkType.TEXT_DELTA) {
    process.stdout.write(chunk.textDelta);
  }
}

AgentOSConfig.tools accepts the same three forms as the high-level helpers: named tool maps, ExternalToolRegistry (Record, Map, or iterable), and prompt-only ToolDefinitionForLLM[]. AgentOS normalizes those inputs during initialize(...) and registers them into the shared ToolOrchestrator, so direct processRequest() turns can plan against and execute them without helper wrappers. If a config-registered tool collides with an extension or pack tool name, the config tool wins at registration time.

If those external tool calls are AgentOS-registered tools, prefer processRequestWithRegisteredTools(...). It executes the registered tools with the correct live-turn ToolExecutionContext and resumes the stream for you:

import {
  AgentOS,
  AgentOSResponseChunkType,
  processRequestWithRegisteredTools,
} from '@framers/agentos';

const agent = await AgentOS.create();

for await (const chunk of processRequestWithRegisteredTools(agent, {
  userId: 'user-1',
  sessionId: 'session-1',
  textInput: 'Search memory for my preferences',
})) {
  if (chunk.type === AgentOSResponseChunkType.TEXT_DELTA) {
    process.stdout.write(chunk.textDelta);
  }
}

If a live turn can mix AgentOS-registered tools with a stable host-managed tool map, either configure externalTools once on AgentOS.initialize(...) or pass externalTools to processRequestWithRegisteredTools(...). It can be a record, Map, or iterable of tool-like executors, and only missing tool names will run through that host registry. Per-call externalTools override the configured registry by tool name. Use externalTools for helper-level fallback execution; use AgentOSConfig.tools when the tool should be permanently registered and prompt-visible on direct runtime turns too. If an externalTools entry also provides description and inputSchema, the helper temporarily registers a proxy tool so the model can see and plan against it during the turn. Execution-only entries without prompt metadata still work for fallback execution, but they are not visible to the model up front.

If you need fully dynamic routing instead of a fixed tool map, keep using fallbackExternalToolHandler.

For custom host-managed tools, keep using processRequestWithExternalTools(...) and provide your own execution callback.

If you are building a lower-level/custom GMI path and only need prompt-visible host tool schemas, configure AgentOSConfig.externalTools and call agent.listExternalToolsForLLM(). That returns only the prompt-aware host tools. You can turn those into raw OpenAI-style function schemas with formatToolDefinitionsForOpenAI(...) or directly from the registry with formatExternalToolsForOpenAI(...).

processRequestWithExternalTools(...) is the simplest path while the same AgentOS runtime stays alive. For restart-safe external tool execution, AgentOS also persists actionable external pauses into the conversation metadata. A fresh runtime can recover the pending request with getPendingExternalToolRequest(conversationId, userId) and continue on a new stream with resumeExternalToolRequest(...):

If the pending tool calls are AgentOS-registered tools, prefer resumeExternalToolRequestWithRegisteredTools(...). It executes the registered tools with the correct resume-time ToolExecutionContext and then resumes the stream for you.

import {
  AgentOS,
  AgentOSResponseChunkType,
  resumeExternalToolRequestWithRegisteredTools,
} from '@framers/agentos';

const agent = await AgentOS.create();
const pending = await agent.getPendingExternalToolRequest('conv-1', 'user-1');

if (pending) {
  for await (const chunk of resumeExternalToolRequestWithRegisteredTools(agent, pending, {
    organizationId: 'org-123',
  })) {
    if (chunk.type === AgentOSResponseChunkType.TEXT_DELTA) {
      process.stdout.write(chunk.textDelta);
    }
  }
}

If a persisted pause can mix AgentOS-registered tools with a stable host-managed tool map, either configure externalTools once on AgentOS.initialize(...) or pass externalTools to resumeExternalToolRequestWithRegisteredTools(...). The helper will execute the registered tool calls itself and only delegate missing tool names to that host registry before resuming the stream. Per-call externalTools override the configured registry by tool name. Prompt-aware entries with description and inputSchema are also registered temporarily during the resumed stream so follow-up model calls can plan against the same host tools.

If you need fully dynamic routing instead of a fixed tool map, keep using fallbackExternalToolHandler.

For custom host-managed tools that are not registered in AgentOS, keep using resumeExternalToolRequest(...) directly and supply your own tool results.

This recovery path assumes the conversation store is still available after the original process exits.

A pause is resumed once. When the conversation no longer holds the pause the snapshot names (its results were passed on its stream, an output guardrail blocked its stream, a newer turn ended it, or a newer pause replaced it), another resume of it is under way, or a result answers a call the pause does not hold, resumeExternalToolRequest(...) yields one error chunk with the code SYS_INVALID_STATE and the GMI takes none of the results. A resume ends the stream the pause was saved on. The same rule holds on that stream: handleToolResult(...) and handleToolResults(...) continue the pause once, with results for its calls. A second delivery while the pause is being continued, or a result for a call the stream's next pause does not hold, yields one error chunk with the code SYS_INVALID_STATE; a result passed after the stream closed (a resume took the pause, a newer turn ended it, or the continuation finished) yields the closed stream's error chunk (TOOL_ERROR). A resume refused for these reasons is refused before it acquires the session's GMI, so it changes neither the conversation nor that GMI: the preferredModelId and preferredProviderId a refused resume names are not recorded. Read the pending request again with getPendingExternalToolRequest(...) before a retry.

Pass every result of a pause together: a call the results leave out is answered with a tool result that says it has none (code EXTERNAL_TOOL_NOT_RUN), in the conversation and in the results the GMI reads when it takes results together (IGMI.handleToolResults(), as the shipped GMI does), and so is each call of a pause a new turn ends, so the model never reads a call without its result. A GMI that takes one result at a time reads the host's result alone.

A continuation that fails before the conversation records its results (the store refuses to save that the pause was taken, or a result has no JSON form, as an output holding a BigInt has none) yields an error chunk, ends its stream and leaves the pause as it was: getPendingExternalToolRequest(...) returns it, resumeExternalToolRequest(...) continues it, and a new turn ends it. A resume whose organizationId the runtime refuses (a single-tenant runtime with strictOrganizationIsolation and another organization) leaves the pause as it was too, and so does a new turn the runtime refuses before it records its input, for such an organization or because its turn plan fails. A continuation that fails after the conversation recorded its results keeps them, and a pause that a turn or a continuation saved before it failed (the store refuses that save) ends the way a new turn ends one, since the host never read its request.

The once-only rule holds within one AgentOS instance. Each instance keeps a conversation in memory and saves it whole, so two instances that resume one pause at the same time both resume it, and the later save overwrites the earlier one. A deployment that runs several instances over one conversation store sends each conversation's turns, tool results and resumes to one instance at a time. Within one instance AgentOS does not queue a conversation's operations either: a turn, a continuation and a resume each start when the host calls them. Start a conversation's next turn once the stream of its continuation or resume has ended: a turn that starts while one runs finds no pause to end, and its input lands among the continuation's messages.

AgentOSConfig.learning turns on the runtime's learning layer: every GMI sends its persona's personality section as a system prompt of its own on every turn, each session is bound to its user under the turn's resolved organization, and AgentOS.getLearningLayer() returns the layer (undefined while learning is off), which carries the operator's calls (freeze, unfreeze, suspend, unsuspend, operatorSet, revert, exportLearnedState, importLearnedState and forgetUser). Learning has the rules.

Guidance​

  • Show high-level examples first in README and landing guides.
  • Keep low-level AgentOS examples in architecture, advanced usage, extensions, workflows, and runtime-control docs.
  • Document both layers explicitly. They are complementary, not competing.
  • Keep generateImage() provider-agnostic at the API boundary, but expose provider-specific knobs through providerOptions when needed.
  • Do not force downstream libraries to adopt agent() unless the helper reaches feature parity with their runtime needs.