Skip to main content

Safety Primitives

Autonomous agents with LLM access can incur unbounded cost when a vendor API flakes, a retry policy misfires, or an output guardrail silently rejects every attempt. AgentOS ships small, independent primitives that bound the failure modes behind runaway spend, stuck loops and hung tools. Three run inside the runtime: generateText() and streamText() consult the provider health registry, processRequest() reserves against a configured spend meter, and generateText(), streamText(), generateObject(), embedText() and agent() check each provider call against a CostGuard when they are given a spend budget (Cost Optimization). A host calls the others around its own model and tool calls, and can call a CostGuard of its own the same way.

Each has defaults and works alone or composed with the others (How they work together).

These are operational guards — they don't read message content. For content-level safety (toxicity, PII, prompt injection, folder-level filesystem permissions) see Guardrails.

A host's guard chain​

Each layer is independent; use any subset.

CircuitBreaker​

Three-state (closed -> open -> half-open) pattern wrapping any async operation. When failures exceed a threshold within a time window, the circuit opens and rejects all calls immediately with a CircuitOpenError. After a cooldown period, it transitions to half-open and allows probe calls through. If probes succeed, it closes again.

Config​

OptionDefaultDescription
namerequiredBreaker identifier (used in errors and callbacks)
failureThreshold5Failures before opening
failureWindowMs60,000Window in ms for counting failures
cooldownMs30,000Time in open state before probing
halfOpenSuccessThreshold2Successes needed in half-open to close
onStateChangeundefinedCallback: (from, to, name) => void

Usage​

import { CircuitBreaker, CircuitOpenError } from '@framers/agentos';

const breaker = new CircuitBreaker({
  name: 'openai-api',
  failureThreshold: 3,
  cooldownMs: 60_000,
  onStateChange: (from, to, name) => {
    console.log(`[${name}] ${from} -> ${to}`);
  },
});

try {
  const response = await breaker.execute(async () => {
    return await openai.chat.completions.create({ model: 'gpt-4o-mini', messages });
  });
} catch (err) {
  if (err instanceof CircuitOpenError) {
    console.log(`Circuit open. Retry after ${err.cooldownRemainingMs}ms`);
  }
}

// Inspect state
const stats = breaker.getStats();
// { name: 'openai-api', state: 'closed', failureCount: 0, totalTripped: 0, ... }

LLMProviderHealthRegistry​

A status-aware, process-lifetime memory of LLM provider health, keyed by providerId. Wired into generateText and streamText so the next caller doesn't pay a full TLS round-trip to rediscover a provider that just returned 402 Insufficient Credits or 401 Invalid API key.

The plain CircuitBreaker above uses a single failure-threshold + cooldown pair per instance — fine for one-off operations. The router needs per-error-class behavior: open immediately on a payment or auth failure, but require a streak before tripping on a 429 or 5xx. This is what the registry adds.

Error classThresholdCooldown
402 insufficient credits1 fail5 min
401, 403 auth/forbidden1 fail30 min
429 rate limit3 fails30 s
5xx + unclassifiable5 fails60 s

The 5-minute window on 402 reflects operational reality: credits might get topped up while a batch job is in flight. 30 minutes on 401/403 is longer because those failures usually require an env change plus redeploy. 429 cooldowns are intentionally short because rate limits typically lift in a single billing interval.

How the router uses it​

  1. Before the primary call, generateText consults globalLLMProviderHealth.isOpen(resolvedProviderId). If the breaker is open, it throws a synthetic LLMProviderCircuitOpenError with httpStatus: 503. The existing isRetryableError check recognizes that status and routes the call into the fallback chain. No network round-trip, no TLS handshake, no waste.
  2. On a real provider error (anything caught in the outer try/catch except the breaker's own LLMProviderCircuitOpenError and a call the caller stopped: a spend budget's refusal, a hook's error under hookErrors: 'throw', or the call's abortSignal aborting), recordFailure(providerId, error) classifies the error by HTTP status and either trips immediately (for 401/402/403) or increments the streak counter (for 429/5xx).
  3. On success, recordSuccess(providerId) resets the streak counter so a future transient failure starts fresh. A single success does NOT shorten an already-open cooldown: the breaker is open precisely because we want to stop probing for a window.
  4. In the fallback chain loop, every fallback entry is checked against isOpen() before its attempt. A dead chain entry is skipped instantly, so the loop walks to the first healthy provider with O(N) constant-time checks rather than O(N) network calls.

Error classification​

The registry reads HTTP status from three sources, in order:

  1. [NNN] ... prefix in error.message — the shape OpenRouterProvider decorates its errors with so downstream regex-based routing can find them.
  2. error.statusCode numeric property — OpenRouterProviderError sets this explicitly.
  3. error.status numeric property — the Anthropic and OpenAI SDK shape.

If none of those resolves, the error is treated as the conservative transient class (5-failure threshold, 60 s cooldown). Better to under-protect on a one-off network blip than lock out a healthy provider.

Errors that judge the request rather than the provider do not count: a 4xx other than 401, 402, 403, 408 and 429 (a malformed body, an unknown model), a content-policy decline (code or type of content_filter, content_policy_violation or safety_violations), and a request larger than the model's context window.

Config​

The policy table is fixed in code; the registry takes no options.

Usage​

import { globalLLMProviderHealth, LLMProviderHealthRegistry } from '@framers/agentos/core/safety';

// Read state for an admin / diagnostics endpoint
const stats = globalLLMProviderHealth.getStats('openrouter');
if (stats?.state === 'open') {
  console.log(
    `OpenRouter circuit open; ${stats.cooldownRemainingMs}ms until close. ` +
    `Last status: ${stats.lastStatusCode}, total trips: ${stats.totalTrips}`,
  );
}

// Manually reset after a credit top-up so the next call probes immediately
globalLLMProviderHealth.reset('openrouter');

// Construct a private registry for a test
const isolated = new LLMProviderHealthRegistry();
isolated.recordFailure('mock-provider', new Error('[402] Test'));
expect(isolated.isOpen('mock-provider')).toBe(true);

Why a singleton​

Provider health is process-wide state. Two concurrent generateText calls inside the same Node process see the same OpenRouter: if one just discovered it's at 402, the other shouldn't redo the discovery. The globalLLMProviderHealth singleton is the natural granularity. Tests construct their own LLMProviderHealthRegistry instances to keep state isolated across cases.

The registry is ephemeral by design: it lives in memory and resets on server restart. Persistent provider-health tracking would add complexity (Redis, write-through cache invalidation) for a problem the in-process singleton already solves for the dominant case: a long-running batch job hammering a degraded provider.

ActionDeduplicator​

Hash-based recent action tracking with a configurable time window and LRU eviction. The caller computes the key string -- this class is intentionally generic. Use it to prevent duplicate votes, duplicate posts, or any repeated action within a window.

Config​

OptionDefaultDescription
windowMs3,600,000 (1 hr)Time window for dedup tracking
maxEntries10,000Maximum tracked entries before LRU eviction

Usage​

import { ActionDeduplicator } from '@framers/agentos';

const dedup = new ActionDeduplicator({ windowMs: 900_000 }); // 15-minute window

const key = `vote:${agentId}:${postId}`;

if (dedup.isDuplicate(key)) {
  console.log('Already voted on this post recently');
  return;
}

dedup.record(key);
await castVote(agentId, postId);

// Or use the combined check-and-record method:
const { isDuplicate, entry } = dedup.checkAndRecord(`like:${agentId}:${postId}`);
if (isDuplicate) {
  console.log(`Seen ${entry.count} times since ${new Date(entry.firstSeenAt)}`);
}

StuckDetector​

Detects agents producing identical outputs or errors repeatedly. Uses fast djb2 hashing (no crypto overhead) to track output history per agent within a sliding window.

Detects three patterns:

  • repeated_output -- The same output appears N times in a row
  • repeated_error -- The same error message appears N times in a row
  • oscillating -- Agent alternates between two outputs (A, B, A, B pattern)

Config​

OptionDefaultDescription
repetitionThreshold3Identical outputs before flagging stuck
errorRepetitionThreshold3Identical errors before flagging stuck
windowMs300,000 (5 min)Sliding window for history
maxHistoryPerAgent50Max entries tracked per agent

Usage​

import { StuckDetector } from '@framers/agentos';

const detector = new StuckDetector({ repetitionThreshold: 3 });

// After each LLM call, check for stuck behavior
const check = detector.recordOutput('agent-1', response.content);

if (check.isStuck) {
  console.log(`Agent stuck: ${check.reason}`);
  // check.reason is 'repeated_output' | 'repeated_error' | 'oscillating'
  // check.details has a human-readable description
  // check.repetitionCount tells you how many repeats were detected
  pauseAgent('agent-1');
}

// Also track errors
try {
  await callLLM();
} catch (err) {
  const errCheck = detector.recordError('agent-1', err.message);
  if (errCheck.isStuck) {
    // Same error 3 times in a row -- stop retrying
    break;
  }
}

// Clean up when an agent is removed
detector.clearAgent('agent-1');

CostGuard​

Per-agent spending caps with three levels: session, daily, and single operation, kept in process memory. CostGuard stops nothing itself: the host asks canAfford() before a call and records the cost after it, and onCapReached fires when a recorded cost reaches a cap. CostGuard never throws CostCapExceededError, and AgentOS runs no CostGuard of its own unless a call carries a spend budget: a budget on generateText(), streamText(), generateObject(), embedText() or agent() keeps a CostGuard (the one passed as guard, or one it makes), checks each provider call against it before the call and records the call's cost after it, and refuses a call that would pass the budget with CostCapExceededError unless the budget is set to warn. reserveSpend() and reserveOperationSpend() throw the same error, with the cap type daily, when an admission would take a day's committed spending past its cap; that day's total lives in the caller's own tables, not in a CostGuard.

Config​

OptionDefaultDescription
maxSessionCostUsd$1.00Maximum spend per agent session
maxDailyCostUsd$5.00Maximum spend per agent per day
maxSingleOperationCostUsd$0.50Maximum spend for a single operation
onCapReachedundefinedCallback: (agentId, capType, currentCost, limit) => void

Usage​

import { CostGuard } from '@framers/agentos';

const guard = new CostGuard({
  maxDailyCostUsd: 2.00,
  onCapReached: (agentId, capType, cost, limit) => {
    console.log(`${agentId} hit ${capType} cap: $${cost.toFixed(4)} / $${limit.toFixed(2)}`);
    pauseAgent(agentId); // the host's own pause
  },
});

// Before each operation, check affordability
const check = guard.canAfford('agent-1', 0.003); // estimated cost
// A refusal names its cap in check.capType, with the total so far and the cap in check.currentCostUsd and check.limitUsd
if (!check.allowed) {
  throw new Error(check.reason); // "Daily cost $5.0031 would exceed limit $5.00"
}

// After the operation, record actual cost
guard.recordCost('agent-1', actualCostUsd, 'llm-call-123');

// Per-agent overrides
guard.setAgentLimits('expensive-agent', { maxDailyCostUsd: 10.00 });

// Inspect spending
const snapshot = guard.getSnapshot('agent-1');
// { sessionCostUsd: 0.42, dailyCostUsd: 1.87, isSessionCapReached: false, ... }

// Daily totals reset at the process's local midnight. Manual reset:
guard.resetSession('agent-1');
guard.resetDailyAll();

Spend meter​

A persisted allowance per account and period, for a product that sells a number of turns a month, or per account over a rolling window, for one that allows a number of calls in any hour. CostGuard caps one process's spending in memory; the spend meter keeps the count in a database several processes share, so two servers cannot both sell the last turn, and a restart loses nothing.

processRequest reserves before anything reaches a provider and the turn settles the reservation when it ends:

What happensThe requestThe reservation
No operationId on the inputOne error chunk, SYS_VALIDATION_ERRORnone
The period has no roomOne error chunk, BILLING_ALLOWANCE_EXHAUSTED (402), with reason, period and remaining in detailsnone
The same operationId again, still running or already countedOne error chunk, SYS_ALREADY_EXISTSunchanged
The meter's store does not answer within its retry policyOne error chunk, SYS_SPEND_METER_UNAVAILABLE (503)none
The turn finishes, or stops for a tool the host runsThe reply, or the request for the toolconsumed, with the turn's usage
The turn fails before any text reached the stream, a turn that fails while it saves its pause for a tool includedThe errorreleased
The turn fails after some text reached the streamThe errorconsumed
An output guardrail blocks the replyThe guardrail's error chunkrefunded (released, outcome replaced)
The process dies mid-turnnothing moresettled by reconcile() once its lease runs out

Each refusal reaches the caller before any model call. The turn settles in the orchestrator, so a caller that stops reading early still has its turn counted.

Config​

SqlSpendMeter runs on a @framers/sql-storage-adapter store. Give it a Postgres adapter of its own (or the product's pool), never AgentOS's provenance-wrapped storage adapter.

OptionDefaultDescription
dbrequiredThe store. It must offer transactions, persistence and concurrent access unless requireShared: false
allowanceFor(accountId, period)requiredThe account's allowance in units for the period, read at every reservation
periodOf(now, accountId)required unless windowMs is setThe period a moment falls in for the account, such as its calendar month in its own time zone. Not read with a window
windowMsnoneA rolling window in milliseconds, in place of periods: the allowance holds the units consumed in the windowMs before each reservation and every unit still reserved, whatever its age
leaseMs45000How long a reservation lives without a heartbeat; the turn heartbeats on each tool iteration
maxAttemptsPerOperation3Reservations one operation may take, its retries after a release included
retry3 tries, 50 to 400 ms, 2 s deadlineRetries of transient store errors before SpendMeterUnavailableError
resolveUnknown(reservation)noneAsked by reconcile() what became of an expired reservation: consumed, released or unknown
resolveExpired(reservation)noneAsked by reconcile() for each expired reservation in place of resolveUnknown and unknownAfterLease: consumed, released, or deferred until retryAt with its evidence, which keeps the reservation reserved and counted. See Deferred expiry
unknownAfterLease'release'How an expired reservation settles when the answer is unknown
ensureSchematrueCreate the two tables when missing, and on tables that exist the indexes they lack; a product that runs its own migrations copies SPEND_METER_DDL, or spendMeterDdl(prefix) for its tablePrefix, and passes false
tablePrefix'agentos_spend'The start of the two tables' names, <prefix>_meter and <prefix>_reservations; their indexes are named idx_<prefix>_reservations_*, and idx_spend_reservations_* under the default. A lower-case letter and at most 37 more lower-case letters, digits or underscores, and not spend: the constructor throws on any other value, before any statement is built. At 38 characters the longest names, idx_<prefix>_reservations_account and idx_<prefix>_reservations_settled, take the 63 bytes Postgres keeps of an identifier
onWarning(warning)noneTold of each statement that failed while ensureSchema brought tables that exist up to the meter's indexes, with the statement and the error: an index the store refused, or the catalogue read before them. The meter uses the tables as they are

With windowMs the meter counts from the reservations themselves, inside each reservation's own transaction and after it has taken the account row's lock, so two reservations cannot both take the last unit of a window. Every row then carries the period window, which is the period allowanceFor is asked for. A consumed unit counts within its accounting window: from its reservation until more than windowMs have passed. A still-reserved hold counts until explicit settlement, even across window boundaries; expiration alone does not release it. A released reservation never counts, and one whose lease ran out still counts until reconcile() releases it. snapshot() answers the units consumed and reserved inside the window and what remains. A meter given neither periodOf nor windowMs throws when it is constructed.

The window's count reads the index idx_spend_reservations_account, and purge() finds the reservations settled before its before through idx_spend_reservations_settled, on settled_at and state (idx_<prefix>_reservations_account and idx_<prefix>_reservations_settled under another prefix). The DDL creates both with the tables. On tables that exist, ensureSchema reads the store's catalogue before the meter's first statement (pg_indexes on Postgres, sqlite_master on SQLite) and runs the CREATE INDEX IF NOT EXISTS statement of each of the meter's indexes the tables lack, so a store whose tables an earlier release made gains them. A meter whose tables hold all three runs no index statement, so on Postgres its start takes no SHARE lock on the reservations table, the lock a CREATE INDEX takes before it checks IF NOT EXISTS. The indexes change no answer of the meter, so tables that refuse one, as Postgres does for a role that does not own them, are used as they are, and the statement and its error go to onWarning. Postgres holds back writes to a table while it builds an index on it, so a product with a large reservations table creates the two indexes first in its own migration, with CREATE INDEX CONCURRENTLY IF NOT EXISTS and the names and columns spendMeterDdl(prefix) gives them, and runs the meter with ensureSchema: false.

Several meters in one database​

Meters with different rules take a prefix each. Every statement a meter runs names its own two tables, so its count reads only its own rows and its purge() deletes only its own rows. Meters under one prefix share its two tables and every row in them, and a purge by one reaches the settled reservations of the others.

import { SqlSpendMeter, spendMeterDdl } from '@framers/agentos';

// Twenty calls an account in any hour, in the default tables
const hourly = new SqlSpendMeter({ db, windowMs: 3_600_000, allowanceFor: () => 20 });

// Five hundred a month, in tables of its own: monthly_spend_meter and monthly_spend_reservations
const monthly = new SqlSpendMeter({
  db,
  tablePrefix: 'monthly_spend',
  periodOf: (now) => new Date(now).toISOString().slice(0, 7),
  allowanceFor: () => 500,
});

// The same tables and indexes for a product's own migration, with ensureSchema: false on the meter
const migration = spendMeterDdl('monthly_spend');

With no prefix, spendMeterDdl() answers SPEND_METER_DDL word for word; given a prefix the constructor refuses, it throws.

Usage​

import { AgentOS, SqlSpendMeter } from '@framers/agentos';
import { createPostgresAdapter } from '@framers/sql-storage-adapter';

const db = createPostgresAdapter({ connectionString: process.env.DATABASE_URL!, max: 3 });
await db.open();

const meter = new SqlSpendMeter({
  db,
  allowanceFor: async (accountId) => (await plans.planOf(accountId)).turnsPerMonth,
  periodOf: async (now, accountId) => monthIn(await accounts.timeZoneOf(accountId), now), // "2026-10"
  resolveUnknown: async ({ operationId }) => ((await replies.storedFor(operationId)) ? 'consumed' : 'unknown'),
});
meter.startReconciler(); // every 15 seconds

const agentos = await AgentOS.create({ spendMeter: { meter, accountIdOf: (input) => input.userId } });

for await (const chunk of agentos.processRequest({
  userId, sessionId, textInput,
  operationId: messageId, // the product's durable id of this message: a retry with it is never charged twice
})) {
  // ...
}

// The month so far, for the product's own meter line
const { used, remaining } = await meter.snapshot(userId);

// A plan change mid-period: the units used carry over, the new allowance applies at once
await meter.setAllowance(userId, 600);

Deferred expiry​

A call whose work may outlive its lease (a worker still running, an inference not yet stopped) must not be released or consumed by a sweeper that cannot prove it stopped. Reserve it with requireFence: true and give the meter resolveExpired:

const meter = new SqlSpendMeter({
  db, periodOf, allowanceFor,
  resolveExpired: async (reservation) => {
    const fence = await controlPlane.fenceOf(reservation.operationId); // the host's own evidence
    if (!fence.transportClosed || !fence.inferenceTerminated) return { outcome: 'deferred', retryAt: Date.now() + 15_000, evidence: fence };
    return { outcome: fence.served ? 'consumed' : 'released' };
  },
});
await meter.reserve({ accountId, operationId: callId, requireFence: true });

A deferred reservation stays reserved and counted, with its evidence and its next check kept on its row; a resolver that throws defers it for one lease. A meter without resolveExpired (a legacy sweeper over the same tables) leaves a requireFence reservation reserved and counts it pending; an explicit settle() still ends it. A rolling window counts a reservation still reserved whatever its age. releaseExpiredSpend(store, at, { resolve }) does the same for a day's bound, with SpendDayStore.deferReservation keeping the next check.

A reconcile() pass writes each decision to the reservation as it read it, in one conditional statement: the same attempt, still reserved, still past its lease and, for a settlement on a meter without resolveExpired, still unfenced. A reservation that an opted-in pass defers, or that a heartbeat renews, while a legacy pass asks resolveUnknown about it stays reserved. While resolveExpired is asked, a heartbeat's renewed lease stands, and so does another pass's deferral to a next check after the pass's now, with its evidence; an operation released and retried meanwhile keeps the retry reserved under its own lease, with none of the earlier attempt's fence, evidence or next check. A meter without resolveExpired reads the ordinary expired reservations into its page first and the fenced holds only into the room they leave, each in expiry order on the (state, expires_at) index, so the fenced holds it leaves never keep one from settling, whatever the limit, and no pass sorts every due row to find its page.

Retention​

A settled reservation's row stays, so that a retry of its operation is recognised, and a window meter keeps one row per account. purge({ before, limit }) deletes the reservations settled before before and, for a window meter, the account rows that hold nothing reserved and were last written before before. One call deletes at most limit of each (default 1,000) and answers how many of each it deleted, so a sweep calls it again while a count comes back full.

// Twenty calls an account in any hour
const hourly = new SqlSpendMeter({ db, windowMs: 3_600_000, allowanceFor: () => 20 });

// The sweep: what settled more than an hour ago is past the window, and goes
const { reservations, periods } = await hourly.purge({ before: Date.now() - 3_600_000 });

A reservation still reserved is never deleted, and neither is a period meter's period row, which holds the period's count: a month's snapshot() reads the same after its settled reservations are purged. For a window meter, a before at least one window back keeps every unit the window counts. Once an operation's row is purged the meter no longer knows the operation: a retry of it reserves and counts again, and a refund of it finds nothing, so before lies past the time either can come. purge() deletes across every account in its meter's two tables, so meters under one prefix share their retention, and a meter under a prefix of its own keeps its own.

Compound credit​

A host that sells metered calls reserves each call's most before any work starts, and settles it once from measured quantities. @framers/agentos/safety/runtime carries the arithmetic and the ledger; the host supplies rates, caps and its allocation policy.

import {
  InMemoryCompoundCreditStore,
  grantCreditLot,
  reserveCompoundCredit,
  settleCompoundCredit,
  ModelAttemptBudget,
} from '@framers/agentos/safety/runtime';

const store = new InMemoryCompoundCreditStore(); // SqlCompoundCreditStore on Postgres in a deployment
await grantCreditLot(store, { scopeId: 'scope-a', kind: 'paid', fundingRef: 'payment-1', principalMicro: 10_000_000, at: Date.now() });
const held = await reserveCompoundCredit(store, {
  scopeId: 'scope-a', operationId: 'call-1', kind: 'call', requestHash: 'h', rates: { primary: rates },
  mediaComponent: 'media_second', durationSeconds: 600,
  attempts: { maxAttempts: 64, perAttempt: { input_token: 8_192, output_token: 512 } },
  supplierDailyCapMicro: 10_000_000, at: Date.now(), expiresAt: Date.now() + 3_660_000,
});
if (held.status === 'reserved') {
  const budget = new ModelAttemptBudget({ store, holdId: held.hold.holdId });
  const attempt = await budget.authorize({ purpose: 'initial' }); // before every provider request
  await budget.record(attempt.attemptId, { input_token: 900, output_token: 120 }, true);
  await settleCompoundCredit(store, {
    holdId: held.hold.holdId, connectedMs: 61_500, at: Date.now(),
    fence: { mediaClosed: true, inferenceTerminated: true, evidence: 'room closed; worker exited', observedAt: Date.now() },
  });
}
  • A rate is rational (micro micro-dollars for every per units), and a component's charge is one ceiling of the exact product, on BigInt. quoteCompoundLiability prices the full duration and every permitted attempt at the dearest authorized snapshot; reasoning tokens are counted once when the snapshot says output includes them.
  • A hold is allocated to lots in the policy's order (promotionsFirstPolicy: earliest-expiring promotions, then paid lots); settlement consumes those shares in that order. A policy orders only the eligible lots (isLotEligible: active, unexpired, with a free balance), and the balance counts only those, whatever the policy. An expired promotion keeps the share an admitted hold took, and whatever that share does not consume retires.
  • The supplier side of a hold is checked against the platform's daily cap together with every hold still open, whatever day it was made on. Reservations take that check one at a time on one platform-wide lock (the supplier day of the epoch, 1970-01-01), so two on either side of midnight cannot both pass the cap.
  • Settlement needs evidence that the media closed and the inference stopped; without it the hold is deferred, still held. reconcileCompoundCredit defers when its resolver cannot tell or fails.
  • Every provider attempt (initial, tool continuation, retry, fallback) is authorized with authorizeModelAttempt before it is sent; an attempt that never reports usage keeps its supplier most open.
  • Refunds take only a paid lot's free principal, reserved before the payment rail is asked (refundCreditLot, then completeLotRefund). A refund id answers existing only for its own scope, lot and amount; another request under it, one racing it from another scope included, is refused as conflict and reserves nothing. A dispute freezes the lot and the account's admission; a lost reversal past the free balance is a receivable outside spendable credit.
  • marginReport reconciles a scope's period: paid-funded revenue, promotional usage, reversals, refunds, supplier cost including failed work, compute by measured runtime, and other costs quantified or excluded; a missing cost makes it unreconciled, with no contribution stated. Every hold settled in the period needs a measured runtime, an explicit zero included, or compute stays unreconciled. The period's settlements are the ones its settlement entries name, however many holds the scope settled before.

With credit funding, the scoped voice factory uses ModelAttemptBudget below provider request and retry loops; a call with local funding counts its attempts with CountingAttemptBudget, which holds no money. The OpenAI provider's request loop counts every call's attempts with CountingAttemptBudget as well, at the call's maxAttempts or the provider's configured count, so an attempt past the count is refused before it is sent (Attempts). Call minutes remain quantity-only in CallMinuteMeter; rational-rate arithmetic and final component rounding belong to the ledger. Stop evidence, final supplier quantities and the hold's rate snapshots and lot allocations determine settlement. Do not release a deferred hold because its process stopped reporting. marginReport aggregates the ledger's recorded revenue and supplier cost with the measured runtime the host supplies; it neither invents a tariff nor makes unknown supplier cost zero.

ToolExecutionGuard​

Wraps tool execution with a timeout and per-tool circuit breaker, so a hung tool returns a timeout result and a tool that keeps failing is refused while its breaker is open. Each tool gets its own circuit breaker instance and health tracking. execute() never throws: a timeout, an error or an open breaker comes back as { success: false }. A timed-out call is not cancelled; its promise runs on and its result is dropped.

Config​

OptionDefaultDescription
defaultTimeoutMs30,000Default timeout per tool execution
toolTimeoutsundefinedPer-tool timeout overrides (Record<string, number>)
enableCircuitBreakertrueWhether each tool gets its own circuit breaker
circuitBreakerConfigundefinedConfig applied to per-tool circuit breakers

Usage​

import { ToolExecutionGuard } from '@framers/agentos';

const guard = new ToolExecutionGuard({
  defaultTimeoutMs: 15_000,
  toolTimeouts: {
    'web-search': 45_000,  // Search gets more time
    'calculator': 5_000,   // Calculator should be fast
  },
});

const result = await guard.execute('web-search', async () => {
  return await searchTool.run(query);
});

if (result.success) {
  console.log(result.result);       // The tool's return value
  console.log(result.durationMs);   // How long it took
} else {
  console.log(result.error);        // Error message
  console.log(result.timedOut);     // true if it was a timeout
}

// Health monitoring
const health = guard.getToolHealth('web-search');
// { totalCalls: 47, failures: 2, timeouts: 1, avgDurationMs: 3200, circuitState: 'closed' }

// All tools at once
const allHealth = guard.getAllToolHealth();

How they work together​

A host composes the primitives around its own model call:

import { CostGuard, CircuitBreaker, StuckDetector } from '@framers/agentos';

const costGuard = new CostGuard({ maxDailyCostUsd: 2 });
const breaker = new CircuitBreaker({ name: 'llm' });
const stuck = new StuckDetector();

async function guardedCall(agentId: string, prompt: string) {
  const affordable = costGuard.canAfford(agentId, 0.001); // the host's estimate
  if (!affordable.allowed) throw new Error(affordable.reason);

  const response = await breaker.execute(() => callModel(prompt)); // the host's model call

  costGuard.recordCost(agentId, priceOf(response.usage)); // the host's pricing

  const check = stuck.recordOutput(agentId, response.text);
  if (check.isStuck) pauseAgent(agentId, check.details); // the host's own pause

  return response;
}

ActionDeduplicator fits around actions with side effects (a vote, a post, a message), and ToolExecutionGuard around tool calls.

Defense Matrix​

LayerProtectionDefault TriggerError Type
CircuitBreakerOpens after failures, cooldown before retry5 fails in 60sCircuitOpenError
LLMProviderHealthRegistrySkips a provider that keeps failing, in generateText() and streamText()402: 1 failure; 401/403: 1; 429: 3; 5xx: 5LLMProviderCircuitOpenError (status 503, routed to the fallback chain)
CostGuardSpending caps per session/day/operation, checked by the host or by a spend budget (Cost Optimization)$5/day per agent{ allowed: false, reason } from canAfford(); onCapReached callback; CostCapExceededError from a spend budget
Spend meterA persisted allowance per account and period, in processRequest()the host's allowanceBILLING_ALLOWANCE_EXHAUSTED error chunk
Spend reservationsA day's spending cap held in the caller's own tables, across processes (Cost Optimization)the cap each admission namesCostCapExceededError (daily) from reserveSpend() and reserveOperationSpend()
StuckDetectorFlags repeated output or oscillation3 identical outputs in 5 min{ isStuck: true, reason }
ToolExecutionGuardTimeout + per-tool circuit breaker30s timeout{ success: false, timedOut }
ActionDeduplicatorPrevent duplicate actions within window1 hr window, 10k entriesBoolean check

Imports​

The primitives are exported from the @framers/agentos package, the provider health registry from @framers/agentos/core/safety:

import {
  CircuitBreaker,
  CircuitOpenError,
  ActionDeduplicator,
  StuckDetector,
  CostGuard,
  CostCapExceededError,
  ToolExecutionGuard,
  ToolTimeoutError,
  SqlSpendMeter,
} from '@framers/agentos';
import { globalLLMProviderHealth, LLMProviderHealthRegistry } from '@framers/agentos/core/safety';

Killswitches, social rate limits and an action audit log (SafetyEngine, ActionAuditLog, ContentSimilarityDedup) belong to Wunderland's social network, not to AgentOS.

References​

Circuit breakers + bulkheads​

  • Nygard, M. T. (2018). Release It! Design and Deploy Production-Ready Software (2nd ed.). Pragmatic Bookshelf. — Foundational treatment of stability patterns: circuit breaker, bulkhead, timeout, and steady-state. The CircuitBreaker here implements the three-state (closed / open / half-open) machine from this book.
  • Fowler, M. (2014). CircuitBreaker. Martin Fowler's bliki. — Practical write-up of the circuit-breaker pattern with state-transition examples.

Cost guards + resource controls​

Stuck detection / liveness​

  • Brewer, E. A. (2000). Towards robust distributed systems. PODC 2000 keynote. — The CAP theorem framing that motivates aggressive timeout + stuck-detection in distributed agent runtimes where partial unavailability is normal.
  • Cantrill, B. (2006). Hidden in plain sight. ACM Queue, 4(1). — On instrumenting production systems to find pathological behaviour where it happens.

Implementation references​