Skip to main content

System Architecture

AgentOS organizes the runtime around long-running agent state rather than around a single turn loop. Cross-session conversations, parallel agent instances with independent personality and memory, conditional tool execution, human-in-the-loop approval, and a memory layer that distinguishes verified user input from model-generated content are first-class subsystems with their own modules.

The modules documented below are predominantly state-management subsystems. The turn loop itself is one component among them, not the central abstraction.

This page is the system map. For the what of each subsystem — components, lifecycle ownership, source-tree location — read on. For deep-dives into individual concerns, follow the table of contents.

For specific subsystem deep-dives, see:

AgentOS layered architecture: seven cooperating layers from the API surface (generateText, streamText, generateObject, generateImage, agent, agency, mission) through the cognitive substrate (GMI, GMIManager, PersonaOverlayManager, SentimentTracker, MetapromptExecutor), memory and RAG (working memory, five trace types, Ebbinghaus decay, 8 cognitive mechanisms, observer and reflector, HyDE, GraphRAG, hybrid retrieval, 7 vector stores), tools, skills and extensions (ToolOrchestrator, ExtensionRegistry, SkillRegistry, CapabilityDiscoveryEngine, ForgeToolMetaTool, 100+ extension packs, 88 SKILL.md skills), guardrails and HITL (ParallelGuardrailDispatcher, 4-tier PII redaction, ML classifiers, Grounding Guard, HumanInteractionManager), orchestration (workflow, mission, AgentGraph, CompiledExecutionGraph, ICheckpointStore), down to I/O and providers (voice pipeline, channels, media generation, 13 LLM providers, fallback chains).

The sections below follow the layers: layer 1 → API Surface Contract, layer 2 → GMI, layer 3 → Memory System and RAG System, layer 4 → Tool System and Extension & Guardrail Runtime, layer 5 → Guardrails and Human-in-the-Loop, layer 6 → Multi-Agent Coordination, layer 7 → Voice Pipeline and Channels. The class and function names on the diagram's pills are the names in the code.

Source Directory Layout​

The src/ tree has eleven top-level directories; each groups the modules of one domain, and foundational infrastructure lives under core/.

Perception model: Vision, hearing, and speech are separate modules under io/, following the biological perception analogy -- io/vision/ (OCR, scene detection, image analysis), io/hearing/ (STT providers, VAD, silence detection), and io/speech/ (TTS providers, resolver, session). Shared media generation (images, video, music, SFX) is under io/media/.

Key architectural patterns:

All paths below are under src/.

ModuleSubdirsPurpose
agents/agency/ · definitions/Agent definitions and multi-agent coordination classes (AgencyRegistry, AgencyMemoryManager, AgentCommunicationBus)
api/runtime/ · types/ · structured/ · server/ · interfaces/Public API surface — AgentOS, generateText, streamText, agent, agency, the media helpers, orchestrator collaborators, provider defaults, StructuredOutputManager; a few other folders re-export modules that moved
cognition/substrate/ · memory/ · rag/ · emergent/ · discovery/ · nlp/ · skills/ · marketplace/ · web-search/The GMI and personas (substrate/), cognitive memory, retrieval-augmented generation (vector stores, chunking, reranking, GraphRAG, HyDE), runtime tool forging, capability discovery, NLP utilities, the SKILL.md loader, marketplace listings and web search
config/—Configuration types for the embedding manager, memory lifecycle, retrieval augmentor, tool orchestrator and vector stores
core/config/ · conversation/ · embeddings/ · guardrails/ · llm/ · logging/ · providers/ · rate-limiting/ · safety/ · storage/ · streaming/ · tools/ · types/ · utils/ · vector-store/Foundational infrastructure: LLM providers and routing, shared interfaces, the IStorageAdapter, the StreamingManager, the ITool / ToolOrchestrator, embedding and vector-store abstractions
extensions/packs/Extension system: ExtensionPack, descriptor kinds, activation lifecycle, built-in packs
io/channels/ · hearing/ · speech/ · vision/ · voice-pipeline/ · media/ · avatar/ · segmentation/Messaging adapters, telephony and social posting; STT, VAD and TTS; OCR and image analysis; the real-time voice pipeline; image, video, music and SFX generation; avatars; image segmentation. The meeting-participant channel owns meeting runs, the capture interlock, transcript persistence and purging.
logging/—ILogger and the logger factory
orchestration/builders/ · checkpoint/ · compiler/ · events/ · hitl/ · ir/ · pipeline/ · planner/ · planning/ · runtime/ · tools/ · turn-planner/ · workflows/workflow(), mission() and AgentGraph builders, the graph IR, compiler and runtime, checkpoints, PlanningEngine, human-in-the-loop, the query, memory, ingest and read routers (pipeline/), the turn planner and the workflow engine
safety/guardrails/ · runtime/ · sandbox/ · provenance/ · evaluation/ · validation/ · auth/ · claims/Guardrails (IGuardrailService, ParallelGuardrailDispatcher), runtime safety (CircuitBreaker, CostGuard, StuckDetector), code execution (node:vm) and CLISubprocessBridge / CLIRegistry, provenance and blockchain anchoring, evaluation and OpenTelemetry tracing, and deterministic claim evidence (EvidenceClaimVerifier)
utils/—Shared error helpers

Architecture Layers​

The diagram at the top of this page is the canonical layered view. From top to bottom:

  1. API surface — generateText / streamText / agent / agency / generateImage, plus the AgentOS lifecycle facade.
  2. GMI — per-mind state: ConversationHistoryManager, CognitiveMemoryBridge, SentimentTracker, MetapromptExecutor, persona overlays.
  3. Memory & RAG — cognitive memory (five trace types, working memory, Ebbinghaus decay, 8 mechanisms such as retrieval-induced forgetting and reconsolidation), 7 vector stores, HyDE, GraphRAG, hybrid retrieval, CitationVerifier.
  4. Tools & Extensions — the extension and skill catalog (100+ extension packs, 88 skills) with capability discovery and runtime tool forging.
  5. Safety & Guardrails — guardrail packs (PII, toxicity, grounding, code safety, topicality, content policy), circuit breakers and the cost guard, and HITL approvals (the agency() approval gates and the runtime HumanInteractionManager).
  6. Orchestration — DAG runtime, workflow(), mission(), AgentGraph with its human nodes, checkpointing, planning engine.
  7. I/O & providers — 13 LLM providers, OpenRouter among them, with fallback chains in generateText(), streamText(), agent() and agency(); vision (OCR), hearing (STT, VAD), speech (TTS, voice pipeline), 12 messaging adapters, telephony.

The diagram above the prose shows how a typical request enters at layer 1 and traverses downward.

API Surface Contract​

generateText(), streamText(), agent(), agency(), and the AgentOS runtime share some configuration names, but the shared config surface does not imply identical enforcement.

  • agent() is the lightweight stateful facade for prompt assembly, sessions, tools, hooks, personality shaping, and usage-ledger forwarding.
  • generateText() / streamText() are low-level helper loops for provider selection, direct tool execution, and text-fallback tool calling.
  • The full AgentOS runtime owns the deeper runtime systems: emergent tooling, guardrails, discovery, RAG bootstrapping, HITL, channels and provenance. agency() applies HITL approvals, emergent specialists (on hierarchical), output schemas and provenance records at the agency level; it accepts guardrails, RAG, permissions and security tiers without applying them (Agencies).

GMI (Generalized Mind Instance)​

GMI is what an agent actually is between turns: persona, working memory, mood, reasoning trace, conversation history. Each instance is a single mind bound to one persona. The dedicated GMI page covers the turn loop, completion options, conversation history, the completion gateway and the output stream; this section covers how the GMI plugs into the wider runtime.

GMI Lifecycle​

ERRORED records that the last turn failed; the next turn starts from it as from READY (GMIPrimeState, which also defines INITIALIZING and REFLECTING).

Initialization​

GMI.initialize(persona, config) validates required dependencies, wires collaborators, and loads state:

const gmi = new GMI('my-gmi-id');
await gmi.initialize(researchAssistantPersona, {
  workingMemory,
  promptEngine,
  toolOrchestrator,
  llmProviderManager,
  utilityAI,
  cognitiveMemory,       // Optional: enables CognitiveMemoryBridge
  retrievalAugmentor,    // Optional: enables RAG
});

Required dependencies: workingMemory, promptEngine, toolOrchestrator, llmProviderManager, utilityAI. Optional: cognitiveMemory, retrievalAugmentor, and completionGateway, which routes each model step and falls back across providers; with it, llmProviderManager is a GatewayProviderManager (Model calls through a completion gateway).

Collaborators​

The GMI delegates to four extracted collaborators to keep the core class focused:

CollaboratorResponsibility
ConversationHistoryManagerMaintains chat history, supports hydration from external stores
CognitiveMemoryBridgeBridges GMI turns to the CognitiveMemoryManager (encode/retrieve/observe)
SentimentTrackerTracks user sentiment via IUtilityAI, emits GMIEvent types (frustration, confusion, etc.)
MetapromptExecutorHandles metaprompt triggers, self-reflection, and state updates

Turn Processing​

processTurnStream() is an async generator that yields GMIOutputChunk objects and returns the turn's GMIOutput:

const turn = gmi.processTurnStream(turnInput);
let next = await turn.next();
while (!next.done) {
  const chunk = next.value;
  switch (chunk.type) {
    case GMIOutputChunkType.TEXT_DELTA:            // streamed text
    case GMIOutputChunkType.TOOL_CALL_REQUEST:     // tool calls the model requested
    case GMIOutputChunkType.USAGE_UPDATE:          // a provider usage report
    case GMIOutputChunkType.STEP_FINISHED:         // a model step completed
    case GMIOutputChunkType.TOOL_RESULT:           // a result of the GMI's tool round
    case GMIOutputChunkType.ERROR:                 // the turn failed
    case GMIOutputChunkType.FINAL_RESPONSE_MARKER: // the turn's last chunk
  }
  next = await turn.next();
}
const output = next.value; // GMIOutput: responseText, toolCalls, usage, error

The chunk types, their payloads and their order are on the GMI page.

AgentOS Facade​

AgentOS (api/AgentOS.ts) is the public-facing facade that manages GMI instances, streaming, and cross-cutting concerns. It exposes processRequest() as the primary entry point and coordinates:

AgentOSConfig is the configuration object (about 40 fields) that wires all subsystems together. Key optional features activated via config: ragConfig, turnPlanning, emergent, observability, standaloneMemory, workflowEngineConfig.

Request Lifecycle​

A request to the full runtime passes through five stages. The stage boundaries below are the ones in the code; the facade owns guardrails, the pipeline owns preparation, the GMI owns the model call.

  1. Facade (AgentOS.processRequest()) fills selectedPersonaId from defaultPersonaId when the request has none, applies the self-improvement session overrides and the skill prompt context (SelfImprovementSessionManager), negotiates the language, evaluates the input guardrails with evaluateInputGuardrails (the dispatcher runs sanitizers first, then classifiers, in parallel through ParallelGuardrailDispatcher), and hands the turn to the orchestrator. A blocked input ends the request with the guardrail's own stream (createGuardrailBlockedStream) before any turn starts, and while a guard named in requiredGuardrails is not active the request gets one SYS_GUARDRAIL_REQUIRED_MISSING error chunk and runs nothing. The facade performs no input validation (that is the pipeline's first phase), no authentication and no rate limiting; the host does the last two before calling it.
  2. Orchestrator (AgentOSOrchestrator) registers the stream and runs the pre-model pipeline.
  3. Preparation (TurnExecutionPipeline.prepareTurn()), twelve phases: input validation (selectedPersonaId must be present; the facade fills it from defaultPersonaId), GMI acquisition through GMIManager.getOrCreateGMIForSession(), stream context registration, GMI input construction, turn planning (when a turn planner is configured), adaptive execution policies, organization context and long-term memory policy, inbound message persistence, rolling summary compaction, prompt profile routing, long-term memory retrieval, and conversation history assembly with metadata and memory-sink persistence. The rolling summary compaction runs under the compaction profile the turn's mode picks; the profiles share the conversation's one summary state, which records the head it was built with, and the history assembly sends that head and the messages newer than the summary's boundary (Conversation history). The result is a PreparedTurnContext.
  4. The GMI turn (GMI.processTurnStream()): sentiment scoring when the persona enables it, the RAG trigger, memory context assembly through CognitiveMemoryBridge (only when a cognitive memory manager is attached), prompt construction by the PromptEngine, the streaming model call through the provider manager, the tool loop through ToolOrchestrator (up to maxToolLoopIterations, five by default), history and memory updates, and the metaprompts. The turn yields GMIOutputChunks. Metaprompts run after the model call; they do not build the prompt.
  5. Delivery: the orchestrator converts GMI chunks to AgentOSResponseChunks with GMIChunkTransformer and pushes them into the StreamingManager. The facade registers an AsyncStreamClientBridge as one client of that stream, wraps the bridge's output with wrapOutputGuardrails, and yields the guarded chunks to its caller. The output guardrails apply to the stream processRequest() returns (in hold mode its TEXT_DELTA chunks wait for the final verdict); any other client registered on the stream receives the chunks before and without them. When the model requests a tool the host executes, the facade yields that chunk and returns; the host continues the same turn through handleToolResult(), handleToolResults() or resumeExternalToolRequest(), whose streams pass through the same output guardrails. Delivery does not call the model. Tracing spans are recorded throughout (Tracer).

An error inside the GMI turn reaches the stream as an ERROR chunk carrying the original message, with the code GMI_PROCESSING_ERROR unless the error carries its own (an error chunk in the provider's stream carries LLM_PROVIDER_ERROR); the turn's FINAL_RESPONSE follows it with the same error and the usage counted before the failure. An error outside the GMI turn ends the stream with an ERROR chunk (The turn lifecycle).

Sequence Diagram​

Key Types​

TypeModulePurpose
AgentOSInputapi/types/Normalized request envelope (text, audio, images, metadata)
AgentOSResponseapi/types/Streamed response chunks (TEXT_DELTA, TOOL_CALL_REQUEST, FINAL_RESPONSE, ERROR and others)
GMITurnInputcognition/substrate/IGMIInternal turn representation consumed by the GMI
GMIOutputChunkcognition/substrate/IGMIPer-chunk output from the cognitive engine
ConversationContextcore/conversation/Session state: history, active persona, user context

Extension & Guardrail Runtime​

The extension runtime is centered on three core pieces:

  1. ExtensionManifest / ExtensionPack -- Declarative loading of tool bundles, guardrails, and channel adapters.
  2. ExtensionManager -- Descriptor activation and runtime access.
  3. ISharedServiceRegistry -- Lazy singleton reuse across packs (for NLP pipelines, ONNX classifiers, embedding functions).
interface ExtensionPack {
  name: string;
  version?: string;
  descriptors: ExtensionDescriptor[];
  onActivate?: (context: ExtensionLifecycleContext) => Promise<void> | void;
  onDeactivate?: (context: ExtensionLifecycleContext) => Promise<void> | void;
}

Creating an Extension Pack​

Extension packs are the unit of distribution. Each pack bundles one or more descriptors of the same or different kinds (tool, guardrail, workflow, provenance, etc.) and can hook into the activation lifecycle to perform setup and teardown.

import type { ExtensionPack, ExtensionLifecycleContext } from '@framers/agentos/extensions';
import { EXTENSION_KIND_TOOL } from '@framers/agentos/extensions';

export function createMyExtensionPack(): ExtensionPack {
  return {
    name: 'my-custom-tools',
    version: '1.0.0',
    descriptors: [
      {
        id: 'search_documents',
        kind: EXTENSION_KIND_TOOL,
        payload: {
          id: 'my-search-tool',
          name: 'search_documents',
          displayName: 'Document Search',
          description: 'Search internal documents by query.',
          inputSchema: {
            type: 'object',
            properties: { query: { type: 'string' } },
            required: ['query'],
          },
          execute: async (args) => {
            const results = await searchIndex(args.query);
            return { success: true, output: results };
          },
        },
      },
    ],
    onActivate: async (ctx: ExtensionLifecycleContext) => {
      const apiKey = ctx.getSecret?.('MY_API_KEY');
      // Initialize resources, warm caches, etc.
    },
    onDeactivate: async () => {
      // Release resources
    },
  };
}

Packs are loaded by including them in the extensionManifest passed to AgentOS.create() ({ packs: [{ factory: () => createMyExtensionPack() }] }, or a package or module entry), or by using the schema-on-demand meta-tools (extensions_list, extensions_enable) at runtime.

Descriptor Kinds​

Every descriptor carries an id, a kind and a payload, plus optional priority, enableByDefault, requiredSecrets and lifecycle hooks (ExtensionDescriptor).

KindConstantPayloadDescription
toolEXTENSION_KIND_TOOLIToolCallable tool registered in ToolOrchestrator
guardrailEXTENSION_KIND_GUARDRAILIGuardrailServiceInput/output guardrail
workflowEXTENSION_KIND_WORKFLOWWorkflowDescriptorPayloadReusable workflow definition
provenanceEXTENSION_KIND_PROVENANCEprovenance pack payloadContent anchoring provider
meeting-providerEXTENSION_KIND_MEETING_PROVIDERIMeetingBotProviderMeeting participant registered by the host on MeetingManager

The other kinds are response-processor, workflow-executor, persona, planning-strategy, hitl-handler, communication-channel, memory-provider, stt-provider, tts-provider, vad-provider, wake-word-provider, messaging-channel, http-handler, streaming-stt-provider, streaming-tts-provider, diarization-provider, endpoint-detector, bargein-handler, vad-session-provider and audio-endpoint-provider.

Guardrail Dispatch Model​

ParallelGuardrailDispatcher uses a two-phase execution model:

  1. Phase 1 (sequential sanitizers) -- Guardrails with config.canSanitize === true run in registration order and can chain SANITIZE results deterministically. A BLOCK in Phase 1 short-circuits the entire pipeline.
  2. Phase 2 (parallel classifiers) -- All remaining guardrails run concurrently via Promise.allSettled. A Phase 2 SANITIZE is downgraded to FLAG because concurrent sanitization would produce non-deterministic results.

The final outcome uses worst-wins aggregation: BLOCK (3) > FLAG (2) > ALLOW (0).

GuardrailOutputPayload carries ragSources?: RagRetrievedChunk[] so grounding-aware guardrails can verify claims against retrieved evidence.

Each guardrail service can also configure timeouts via config.timeoutMs. If a guardrail exceeds its timeout or throws, it fails open (returns null) rather than blocking the pipeline, unless its config.failClosed is true, in which case the failure counts as a BLOCK.

Built-in Guardrail Packs​

Six built-in packs ship on npm as @framers/agentos-ext-<id>. The source of ml-classifiers, topicality and content-policy-rewriter is in agentos-extensions/registry/curated/safety/; the pii-redaction, code-safety and grounding-guard folders there hold a README that points to their npm packages:

  • pii-redaction — sanitizer; redacts personally identifiable information before tokens leave the runtime
  • ml-classifiers — toxicity / hate-speech / harm classification via on-device ONNX models
  • topicality — input check that blocks a prompt matching a blocked topic and flags one outside the allowed topics, by embedding similarity, with an LLM judge and then keyword matching as fallbacks
  • code-safety — a default set of 25 regex rules for dangerous code patterns, run over the code fences in user input and in the agent's streamed text, and over the code argument of a shell_execute, run_sql, write_file, create_file or edit_file tool call (Tool Calls says what a block stops)
  • grounding-guard — verifies output claims against retrieved RAG sources (citation faithfulness)
  • content-policy-rewriter — output check that rewrites text in a category set to sanitize and blocks text in a category set to block; by default only the illegal_harmful category is on, and it blocks

For details on writing custom guardrails, see Creating Guardrails and Guardrails Usage.

Persona System​

Personas define the identity, expertise, and behavioral configuration for a GMI instance.

Key files:

  • cognition/substrate/personas/IPersonaDefinition.ts -- The IPersonaDefinition interface
  • cognition/substrate/personas/PersonaLoader.ts -- Loads persona JSON files from disk or registry
  • cognition/substrate/personas/PersonaValidation.ts -- Schema validation
  • cognition/substrate/persona_overlays/PersonaOverlayManager.ts -- Runtime persona layering

A persona definition includes:

  • Identity -- id, name, description, version, strengths, and the baseSystemPrompt
  • Model settings -- defaultModelId, defaultProviderId, defaultModelCompletionOptions, tool ids and capabilities
  • Memory and context -- memoryConfig (RAG retrieval and ingestion triggers), cognitiveMemoryConfig, conversationContextConfig
  • Adaptation -- moodAdaptation, sentimentTracking, metaPrompts, contextualPromptElements
  • Personality traits -- personalityTraits, where the HEXACO scores go (honesty, emotionality, extraversion, agreeableness, conscientiousness, openness)

HEXACO Trait Modulation​

The HEXACO model provides six personality dimensions. In cognitive memory (the traits a host passes as CognitiveMemoryConfig.traits), each one weights a content feature at encoding and scales one mechanism, and openness scales two:

HEXACO TraitRangeCognitive Effect
Honesty-Humility0-1Ethical-content attention at encoding; stronger source-confidence decay of reflections.
Emotionality0-1Emotional-content attention at encoding; faster reconsolidation drift toward the current mood.
Extraversion0-1Social-content attention at encoding; a lower feeling-of-knowing threshold, so more partial recalls surface.
Agreeableness0-1Cooperative-content attention at encoding; a faster emotion-regulation reappraisal rate.
Conscientiousness0-1Procedural-content attention at encoding; stronger retrieval-induced forgetting; above 0.6, one working-memory slot fewer.
Openness0-1Novelty attention at encoding; a higher involuntary-recall probability and a larger schema-encoding boost for novel traces; above 0.6, one working-memory slot more.

Persona Definition Example​

const researchAssistant: IPersonaDefinition = {
  id: 'research-assistant',
  name: 'Research Assistant',
  description: 'Academic research aide',
  version: '1.0.0',
  baseSystemPrompt: 'You are a meticulous research assistant...',
  strengths: ['literature review', 'data analysis', 'citation management'],
  personalityTraits: {
    honesty: 0.9,
    emotionality: 0.3,
    extraversion: 0.5,
    agreeableness: 0.7,
    conscientiousness: 0.9,
    openness: 0.8,
  },
  memoryConfig: {
    enabled: true,
    ragConfig: {
      enabled: true,
      retrievalTriggers: { onUserQuery: true },
    },
  },
  moodAdaptation: { enabled: true, defaultMood: 'neutral', sensitivityFactor: 0.3 },
  defaultModelId: 'gpt-4o',
  defaultProviderId: 'openai',
};

The PersonaOverlayManager supports runtime persona blending -- applying temporary overlays (e.g., "be more formal") on top of the base persona definition without mutating the original.

For the bundled persona definitions, see Personas.

Prompt Construction​

The GMI builds each model call's prompt with the PromptEngine, which fits the parts to the model's context window and assembles the messages. MetapromptExecutor (cognition/substrate/MetapromptExecutor.ts) does not build the prompt: it runs metaprompts after the model call, on three trigger types: turn_interval (periodic self-reflection), event_based (driven by SentimentTracker events like frustration or confusion), and manual (flags in working memory).

Prompt Assembly Order​

The GMI uses the openai_chat template for every provider; each provider converts the messages to its own wire format. The template assembles them in this order:

The diagram shows a turn's first model call. On the turn's later calls, after a tool round, the user's message is part of the history.

Token Budget Strategy​

The PromptEngine measures the prompt against the model's context window (optimalContextTokens, else maxContextTokens) and enforces two shares of it:

  • Conversation history, 35%. When the history is over its share, or the whole prompt over the window, the turn's own messages stay whole, with a tool round the first of them belongs to, and the earlier history is reduced. The prompt engine's utility AI summarizes it when one is configured and the history is over historyManagement.summarizationTriggerRatio of its share; otherwise the oldest messages are dropped, a tool round (a call and its results) at a time.
  • Retrieved context, 20%. When the retrieved context is over its share, or the whole prompt over the window, the utility AI summarizes it, or it is cut to its share when there is none.

The GMI's ConversationHistoryManager keeps a window of messages (20 by default) and nothing else; the persona fields overflowStrategy and summarizationTriggerTokens are not read.

The conversation the full runtime records (ConversationContext) keeps maxHistoryLengthMessages messages (100 by default): past that it drops the oldest messages after the first messagesToKeepVerbatimHead, keeping the summary right after them and the last summary after them, or summarizes them when enableAutomaticSummarization is on, and getHistory(limit) returns the newest limit, from the first message of a tool round they reach into. None of these cuts splits a tool round: a round the kept messages reach into is kept whole, past the limit, and a summary leaves out a round whose calls still wait for their results.

Built-in Metaprompt Handlers​

MetapromptExecutor includes pre-built handlers for common situations:

  • Frustration recovery -- Triggered by negative sentiment events
  • Confusion clarification -- When the user signals misunderstanding
  • Satisfaction reinforcement -- When the user is pleased
  • Error recovery -- After tool failures
  • Engagement boost -- When the conversation stalls
  • Self-reflection (gmi_self_trait_adjustment) -- Periodic review that updates the GMI's mood, the user's skill level and the task complexity from evidence; it does not change HEXACO traits

See Adaptive Prompt Intelligence for the full guide: the three trigger types, the five preset templates, the state surfaces metaprompts mutate, and concrete cost numbers.

Memory System​

The cognitive memory system replaces flat key-value memory with a personality-modulated, decay-aware architecture grounded in cognitive science.

Core Model​

Memory traces follow the Ebbinghaus forgetting curve:

S(t) = S0 * e^(-dt / stability)

where S0 (initial encoding strength) is set by personality traits, emotional arousal, and content features. The stability time constant grows with each successful retrieval via the desirable difficulty effect -- memories that were harder to retrieve (lower current strength at retrieval time) receive a larger stability boost.

From cognition/memory/core/decay/DecayModel.ts:

// Ebbinghaus forgetting curve
function computeCurrentStrength(trace: MemoryTrace, now: number): number {
  const elapsed = Math.max(0, now - trace.lastAccessedAt);
  return trace.encodingStrength * Math.exp(-elapsed / trace.stability);
}

Traces below a configurable pruning threshold are soft-deleted (isActive = false) during consolidation.

Memory Type Taxonomy​

Five memory types (Tulving's taxonomy plus relational) across four ownership scopes:

TypeDescriptionExample
episodicPersonal experiences and events"User mentioned they're moving to Berlin on Tuesday"
semanticFacts, concepts, general knowledge"The user's preferred language is Python"
proceduralHow-to knowledge, learned procedures"When deploying, run tests first, then build, then push"
prospectiveFuture intentions and reminders"Remind user about the deadline next Monday"
relationalTrust signals, boundaries and emotional bonds between the agent and the user"The user asked not to be messaged after 9 pm"
ScopeVisibilityShared Across
threadSingle conversation threadNothing
userAll conversations with one userThreads
personaAll users of one personaUsers
organizationAll personas in an orgPersonas

Architecture​

Cognitive Pipeline (per-message smart orchestration)​

Above the storage substrate sits an LLM-as-judge orchestration layer that picks strategy per message at three pipeline boundaries. Each stage is its own router primitive — independently shippable, independently testable, composable via the CognitivePipeline facade. This is smart orchestration, not safety guardrails — orchestration picks strategies, guardrails enforce safety/policy at the output stage. They live in different modules on purpose.

Every router has the same internal structure: a classifier (LLM-as-judge that maps input to a category/intent token), a pure select* function (category + routing table + budget policy → strategy decision), a dispatcher (registry of executors per strategy), and shipping presets (three for the memory and read routers, four for the ingest router) calibrated from LongMemEval-S Phase B N=500 measurements.

PrimitiveSubpathCategoriesStrategies
Memory Router@framers/agentos/memory-router6 query categories3 backends (canonical-hybrid, OM-v10, OM-v11)
Ingest Router@framers/agentos/ingest-router6 content kinds6 strategies (raw / summarized / observational / fact-graph / hybrid / skip)
Read Router@framers/agentos/read-router5 read intents5 strategies (single-call / two-call extract+answer / commit-vs-abstain / verbatim / scratchpad)
Cognitive Pipeline@framers/agentos/orchestration/pipeline(composition)wires all three stages
Adaptive Memory Router@framers/agentos/memory-router(self-calibrating)derives routing tables from your own calibration data

Each classifier is provider-agnostic — talks to a small IXClassifierLLM adapter interface, not an SDK. One OpenAI key reproduces the entire pipeline; no Claude / Gemini accounts required for the shipping configuration.

See the dedicated Cognitive Pipeline guide for the unified architecture overview, or the per-stage docs (Memory Router, Ingest Router, Read Router, Adaptive Memory Router) for the routing tables and presets each stage exposes.

The MemoryTrace Envelope​

Every memory is stored as a MemoryTrace (defined in cognition/memory/core/types.ts; abridged):

interface MemoryTrace {
  id: string;
  type: MemoryType;                    // episodic | semantic | procedural | prospective | relational
  scope: MemoryScope;                  // thread | user | persona | organization
  scopeId: string;                     // Id of the thread, user, persona or organization
  content: string;                     // The memory content
  entities: string[];                  // Extracted entity references
  tags: string[];                      // Classification tags
  provenance: MemoryProvenance;        // Source type, confidence, verification count
  emotionalContext: EmotionalContext;   // PAD model: valence, arousal, dominance
  encodingStrength: number;            // S0: initial strength at creation
  stability: number;                   // Time constant (ms), grows with retrieval
  retrievalCount: number;              // Successful retrieval count
  lastAccessedAt: number;              // Unix ms of last access
  reinforcementInterval: number;       // Spaced repetition interval (ms)
  associatedTraceIds: string[];        // Graph linkage to related traces
  createdAt: number;                   // Unix ms of creation
  updatedAt: number;                   // Unix ms of the last update
  isActive: boolean;                   // Soft-delete flag
}

Retrieval Scoring​

Retrieval combines six weighted signals to rank candidate traces (RetrievalPriorityScorer; a recall's scoringWeights replaces the weights for that call):

SignalWeightSource
Strength/decay0.25computeCurrentStrength() from DecayModel
Vector similarity0.35Cosine similarity from IVectorStore
Recency0.10A boost for traces accessed within about a day (24-hour half-life)
Emotional congruence0.15Current mood valence times the trace's valence, when both have the same sign
Graph activation0.10Spreading activation score from IMemoryGraph (0 without a graph)
Importance0.05The trace's provenance confidence, scaled to 0.5–1

Eight Cognitive Mechanisms​

Located in cognition/memory/mechanisms/; seven of the eight are scaled by a HEXACO trait, all but temporal gist extraction. They run when CognitiveMemoryManager is initialized with a cognitiveMechanisms config:

MechanismHEXACO ModulatorEffect
ReconsolidationEmotionalityMemories become labile during retrieval; high E increases drift
Retrieval-induced forgettingConscientiousnessRetrieving one trace suppresses competitors; high C strengthens suppression
Involuntary recallOpennessSpontaneous memory surfacing; high O increases trigger sensitivity
Feeling-of-knowingExtraversionMetacognitive confidence judgment; high X lowers sharing threshold
Temporal gist extractionNoneCompresses old, rarely retrieved traces into a gist at consolidation
Schema encodingOpennessStrengthens novel traces and weakens schema-congruent ones at encoding, once the host sets cluster centroids; high O strengthens novel traces more
Source confidence decayHonesty-HumilityShortens the stability of inferred and reflected traces at consolidation; high H decays reflections faster
Emotion regulationAgreeablenessReappraises high-arousal traces at consolidation; high A reappraises faster

GMI Integration​

  1. Before prompt construction: CognitiveMemoryBridge.assembleContext() calls the manager's assembleForPrompt(), which retrieves and formats memory within a token budget
  2. After the turn: CognitiveMemoryBridge.syncForTurn() encodes the user's input as an episodic trace (user_statement) and the reply as a semantic trace (agent_inference), each with personality-modulated strength
  3. Before each encode: the bridge passes the text to the manager's observe(), which feeds the MemoryObserver buffer when an observer is configured

For full details, see Cognitive Memory (theory + mechanism implementation reference) and the Memory System Overview (composition, archive, vendor comparison).

RAG System​

The RAG subsystem provides retrieval-augmented generation with multiple vector backends and retrieval strategies.

The AgentOS bootstrap path wires EmbeddingManager -> VectorStoreManager -> RetrievalAugmentor. UnifiedRetriever is a higher-level orchestration layer that runs only when a host wires it in.

Retrieval Pipeline​

The GMI integrates with RAG through persona-configurable hooks:

  • shouldTriggerRAGRetrieval() checks ragConfig.retrievalTriggers (on user query, on tool failure, on intent detection)
  • retrievalAugmentor.retrieveContext() runs the default runtime retrieval pipeline
  • performPostTurnIngestion() summarizes and embeds conversation turns

When a host wires QueryRouter.setUnifiedRetriever(...), plan-aware retrieval runs through UnifiedRetriever instead of the legacy dispatcher path. BM25 + dense fusion is available through HybridSearcher and, for memory traces, HybridRetriever.

Within the default QueryRouter path, cacheResults provides in-memory route() result caching, and verifyCitations can attach QueryResult.grounding by running CitationVerifier over retrieved chunks when embeddings are available.

Vector Store Backends​

Seven IVectorStore implementations ship in cognition/rag/vector_stores/:

BackendPersistenceBest For
InMemoryVectorStoreNoneDevelopment / testing
HnswlibVectorStoreFile-basedSelf-hosted approximate nearest-neighbour search
SqlVectorStoreAny @framers/sql-storage-adapter backend (SQLite, PostgreSQL, IndexedDB, Capacitor)Embedded, edge and browser apps
PostgresVectorStorePostgreSQL with pgvectorSQL-native production
QdrantVectorStoreManaged or self-hosted QdrantOpen-source vector database
PineconeVectorStoreManaged cloudVendor-managed scale
Neo4jVectorStoreNeo4jVector search beside a knowledge graph

Retrieval Strategies​

StrategyMethodTradeoff
Dense onlyEmbedding cosine similarityFast, good for semantic match
Sparse onlyBM25 keyword matchingPrecise term matching, no semantic understanding
HybridDense + Sparse with reciprocal rank fusionBest recall, slightly higher latency
HyDEGenerate hypothetical answer, embed thatBetter recall for vague queries, extra LLM call
GraphRAGEntity graph + community summariesBest for multi-hop reasoning, highest setup cost

GraphRAG Engine​

GraphRAGEngine (cognition/memory/retrieval/graph/graphrag/, re-exported from cognition/rag/graphrag/) implements Microsoft GraphRAG-inspired retrieval:

  1. Ingestion: Entity extraction (LLM or pattern-based) -> graph construction (graphology) -> Louvain community detection -> hierarchical meta-graph -> LLM community summarization
  2. Global search: Query community summary embeddings, synthesize across matched communities
  3. Local search: Query entity embeddings, 1-hop graph expansion, include community context

Chunking Strategies​

SemanticChunker in cognition/rag/chunking/ splits text at headings, then paragraphs, then sentences, keeps a fenced code block whole unless it is longer than maxSize (2,000 characters unless set), and falls back to fixed-size splits for text with no boundaries (Chunking and Snippets).

Reranking​

Pluggable providers in cognition/rag/reranking/providers/:

For configuration details, see RAG Memory Configuration and HyDE Retrieval.

Multi-Agent Coordination​

Agency System​

The agency system enables multi-agent coordination across seven strategies (defined in AgencyStrategy in src/api/types.ts):

StrategyBehavior
sequentialEach agent runs after the previous one completes; output of one feeds the next
parallelAll agents run concurrently against the same input; the agency's model synthesizes their results
debateAgents critique and refine each other's outputs across multiple rounds
review-loopOne agent produces, another reviews; loop continues until reviewer accepts or maxRounds
hierarchicalA coordinator agent delegates to sub-agents and synthesizes their results
graphExplicit DAG via dependsOn on each sub-agent; runs roots first, then dependents
panelEvery agent answers the same input on its own model; the agents that returned text are counted against a quorum, and a chair synthesizes their answers (chair: false returns them labelled)

agency() keeps its coordination state in the call itself. Three classes under src/agents/agency/ serve the full runtime and hosts; agency() uses none of them:

Workflow DAG​

The orchestration engine compiles workflow definitions into directed acyclic graphs for parallel execution:

Workflow definitions live in orchestration/workflows/ with these key types:

WorkflowEngine runs these definitions for the full runtime. The workflow(), mission() and AgentGraph builders are a separate path: GraphCompiler in orchestration/compiler/ compiles them to the graph IR, and GraphRuntime in orchestration/runtime/ runs it (Unified Orchestration).

Agent Communication Bus​

AgentCommunicationBus (agents/agency/AgentCommunicationBus.ts) provides structured messaging between the agents a host registers on it:

  • Direct send -- Targeted messages to specific agents
  • Broadcast -- Send to all agents in an agency
  • Request/Response -- Query agents and await responses
  • Handoff -- Transfer context between agents with state, findings, and memory references

Message types: task_delegation, status_update, question, answer, finding, decision, critique, handoff, acknowledgment, error, broadcast, heartbeat.

Planning Engine​

PlanningEngine (orchestration/planner/PlanningEngine.ts) converts high-level goals into multi-step ExecutionPlan objects using the ReAct (Reasoning and Acting) pattern. Supports plan generation, task decomposition, plan refinement, and autonomous plan-execute-reflect loops.

Human-in-the-Loop​

HumanInteractionManager (orchestration/hitl/HumanInteractionManager.ts) provides structured collaboration between AI agents and human operators:

  • Approval requests for high-risk actions (with severity levels and reversibility flags)
  • Clarification requests for ambiguous situations
  • Escalations for transferring control to humans

The ToolOrchestrator integrates HITL directly: with hitl.enabled in its config (off by default), a tool with hasSideEffects waits for the hitlManager's approval before it runs, within approvalTimeoutMs; without a manager the call fails unless autoApproveWhenNoManager is set.

Using the API​

import { agency } from '@framers/agentos';

// Hierarchical agency with runtime agent synthesis. The manager LLM gets
// delegate_to_<name> tools for each static agent plus a spawn_specialist
// tool that lets it mint new specialists for sub-tasks the static roster
// doesn't cover. EmergentAgentForge validates each spec; EmergentAgentJudge
// gates it on safety/scope/risk before activation.
const research = agency({
  provider: 'openai', model: 'gpt-4o',
  agents: {
    researcher: { instructions: 'Find authoritative sources and pull verbatim quotes.' },
    writer: { instructions: 'Write clear, well-cited prose.' },
  },
  strategy: 'hierarchical',
  emergent: {
    enabled: true,
    judge: true,
    planner: { maxSpecialists: 3, requireJustification: true },
  },
});

const result = await research.generate(
  'Research and summarize recent advances in retrieval-augmented generation.',
);

See Emergent Capabilities for the full worked example of multi-GMI synthesis via spawn_specialist, runtime sequence, and tested rejection paths.

Checkpoint/Restore​

The orchestration engine supports checkpointing for long-running workflows via ICheckpointStore (orchestration/checkpoint/). Checkpoints capture the full execution state (completed tasks, pending tasks, intermediate results) and support fork/resume semantics -- you can snapshot a workflow at any point and resume it later, or fork from a checkpoint to explore alternative execution paths.

InMemoryCheckpointStore ships as the default implementation; persistent stores can be plugged in via the ICheckpointStore interface.

For details, see Planning Engine, HITL, Agency API, and Agent Communication.

Tool System​

ToolOrchestrator (core/tools/ToolOrchestrator.ts) manages tool registration, discovery, permission enforcement, and execution. It acts as a facade over ToolPermissionManager and ToolExecutor.

ITool Interface​

Every tool implements the ITool interface (core/tools/ITool.ts):

interface ITool<TInput = any, TOutput = any> {
  readonly id: string;              // Globally unique ID (e.g. "web-search-v1")
  readonly name: string;            // LLM-facing name (e.g. "search_web")
  readonly displayName: string;     // Human-readable title
  readonly description: string;     // Detailed description for LLM tool selection
  readonly inputSchema: JSONSchemaObject;   // JSON Schema for arguments
  readonly outputSchema?: JSONSchemaObject; // Optional output schema
  readonly requiredCapabilities?: string[]; // Permission requirements
  readonly category?: string;              // Grouping (e.g. "data_analysis")
  readonly hasSideEffects?: boolean;       // Triggers HITL gating when true

  execute(
    args: TInput,
    context: ToolExecutionContext,
  ): Promise<ToolExecutionResult<TOutput>>;
}

Custom Tool Example​

import type { ITool, ToolExecutionResult, ToolExecutionContext } from '@framers/agentos/core/tools/ITool';

const weatherTool: ITool = {
  id: 'weather-lookup-v1',
  name: 'get_weather',
  displayName: 'Weather Lookup',
  description: 'Get current weather for a city. Use when the user asks about weather conditions.',
  inputSchema: {
    type: 'object',
    properties: {
      city: { type: 'string', description: 'City name' },
      units: { type: 'string', enum: ['celsius', 'fahrenheit'], default: 'celsius' },
    },
    required: ['city'],
  },
  hasSideEffects: false,
  async execute(args: { city: string; units?: string }, ctx: ToolExecutionContext) {
    const data = await fetchWeatherAPI(args.city, args.units);
    return { success: true, output: data };
  },
};

Tool Execution Flow​

  1. LLM emits a tool_call chunk with name and arguments
  2. ToolOrchestrator resolves the tool by name from its registry
  3. ToolPermissionManager checks persona capabilities and user subscription
  4. If hasSideEffects and the orchestrator's hitl.enabled is set, HumanInteractionManager gates the execution
  5. ToolExecutor validates arguments against inputSchema and calls execute()
  6. Result is formatted as ToolCallResult and fed back to the LLM

Capability Discovery​

The CapabilityDiscoveryEngine (cognition/discovery/) replaces static tool schema dumps in the prompt with a three-tier semantic search system:

TierContentDefault token budgetWhen Used
Tier 0Category summaries200Always included in system prompt
Tier 1Top-5 semantic matches (name, description, key parameters)800Per-turn, based on user query
Tier 2Full JSON schemas2,000On-demand via discover_capabilities meta-tool

The engine pipeline: User Message -> CapabilityIndex.search() -> CapabilityGraph.rerank() -> CapabilityContextAssembler.assemble() -> CapabilityDiscoveryResult.

Extension-Provided Tools​

Tools are typically loaded via ExtensionPack descriptors. The curated registry in @framers/agentos-extensions holds more than 100 extension manifests, 37 of them channels and 16 voice extensions.

For details, see Tool Calling & Loading and Capability Discovery.

Guardrails​

GuardrailAction Enum​

Four possible outcomes from any guardrail evaluation:

enum GuardrailAction {
  ALLOW    = 'allow',     // Pass through unchanged
  FLAG     = 'flag',      // Pass through, record metadata for audit
  SANITIZE = 'sanitize',  // Replace content with modified version
  BLOCK    = 'block',     // Reject / terminate the interaction
}

IGuardrailService Interface​

interface IGuardrailService {
  config?: {
    evaluateStreamingChunks?: boolean;  // Evaluate during streaming
    maxStreamingEvaluations?: number;   // Rate limit per stream
    canSanitize?: boolean;              // Runs in Phase 1 (sequential)
    timeoutMs?: number;                 // Per-evaluation timeout
    failClosed?: boolean;               // Block instead of passing on a timeout or an error
  };
  evaluateInput?(payload: GuardrailInputPayload): Promise<GuardrailEvaluationResult | null>;
  evaluateOutput?(payload: GuardrailOutputPayload): Promise<GuardrailEvaluationResult | null>;
}

Security Tiers​

SecurityTier names five levels: dangerous, permissive, balanced, strict and paranoid (types.ts). agency() accepts one as security.tier and as guardrails.tier. AgentOS defines no guardrail set for any tier and applies none: a host that wants tiers maps each one to its own list of guardrail packs.

Custom Guardrail Example​

import { GuardrailAction, type IGuardrailService } from '@framers/agentos/safety/guardrails';

const domainRestrictionGuard: IGuardrailService = {
  config: { canSanitize: false, timeoutMs: 1000 },
  async evaluateInput({ input, context }) {
    const text = input.textInput ?? '';
    if (text.match(/\b(stock|invest|trade)\b/i)) {
      return {
        action: GuardrailAction.BLOCK,
        reason: 'Financial advice is outside this agent\'s scope.',
        reasonCode: 'DOMAIN_RESTRICTION',
      };
    }
    return { action: GuardrailAction.ALLOW };
  },
};

ParallelGuardrailDispatcher runs guardrails in two phases (sanitizers sequentially, classifiers in parallel). The safety runtime also includes CircuitBreaker, CostGuard, and StuckDetector in safety/runtime/.

For details, see Safety Primitives, Creating Guardrails, and Guardrails Usage.

Voice Pipeline​

The real-time voice conversation pipeline lives in io/voice-pipeline/ and is orchestrated by VoicePipelineOrchestrator, a state machine that coordinates audio capture, speech recognition, endpoint detection, agent inference, text-to-speech synthesis, and barge-in handling.

State Machine​

IDLE -------> startSession() ---------> LISTENING
LISTENING --> turn_complete ----------> PROCESSING
PROCESSING -> LLM tokens start -------> SPEAKING
SPEAKING ---> TTS flush_complete -----> LISTENING (once played out, on a transport that reports it)
SPEAKING ---> barge-in (cancel) ------> INTERRUPTING -> LISTENING
ANY --------> transport disconnect ---> CLOSED
ANY --------> stopSession() ----------> CLOSED

Component Wiring​

Provider Interfaces​

InterfacePurposeImplementations
IStreamTransportBidirectional audio/text transportWebSocketStreamTransport, WebRTCStreamTransport, LiveKitStreamTransport
IStreamingSTTSpeech-to-text recognitionIn tree (providers/): Deepgram streaming, ElevenLabs streaming, OpenAI Realtime transcription, and batch fallbacks over Deepgram pre-recorded and OpenAI Whisper
IStreamingTTSText-to-speech synthesisIn tree: OpenAI, ElevenLabs, Deepgram Aura, Cartesia and Hume, streaming and batch
IEndpointDetectorDetect when the user finishes speakingHeuristicEndpointDetector, AcousticEndpointDetector
IBargeinHandlerHandle user interruptions during playbackHardCutBargeinHandler, SoftFadeBargeinHandler
IDiarizationEngineMulti-speaker identification(optional, provider-specific)

Audio Types​

  • AudioFrame -- Raw PCM audio (Float32Array samples, sampleRate, timestamp). Typically 20ms frames at 16 kHz for STT.
  • EncodedAudioChunk -- Compressed output (Buffer, format: pcm/mp3/opus, durationMs, text). Carries the synthesized text for barge-in tracking.

A watchdog timer prevents the pipeline from staying in LISTENING indefinitely if the user walks away (default 30s, resets after each completed turn).

For details, see Voice Pipeline and Speech Providers.

Channels​

Twelve messaging adapters live in src/io/channels/adapters/, plus four telephony providers in src/io/channels/telephony/providers/ (Twilio, Telnyx, Plivo, plus a mock for tests). Additional social-platform adapters ship as separate extension packs in registry/curated/channels/ of agentos-extensions. Each adapter implements the IChannelAdapter interface; a host initializes an in-tree adapter and registers it on a ChannelRouter, or loads a channel extension pack through the manifest.

The meeting-participant channel in src/io/channels/meetings/ adds MeetingManager, provider packs of kind meeting-provider, injected stores and the listen-session sink; see Meeting bots.

Platform Table​

In-tree messaging adapters (src/io/channels/adapters/):

PlatformAdapterCategory
DiscordDiscordChannelAdapterMessaging
SlackSlackChannelAdapterMessaging
TelegramTelegramChannelAdapterMessaging
WhatsAppWhatsAppChannelAdapterMessaging
Twitter/XTwitterChannelAdapterSocial
RedditRedditChannelAdapterSocial
SignalSignalChannelAdapterMessaging
IRCIRCChannelAdapterMessaging
WebChatWebChatChannelAdapterWeb
TeamsTeamsChannelAdapterEnterprise
Google ChatGoogleChatChannelAdapterEnterprise
SMS (Plivo)PlivoSmsChannelAdapterMessaging

Telephony (src/io/channels/telephony/providers/): Twilio, Telnyx, Plivo. Additional social-platform adapters (LinkedIn, Bluesky, Mastodon, Threads, etc.) ship as extension packs in registry/curated/channels/ of agentos-extensions rather than in-tree.

Channel Routing​

import { ChannelRouter } from '@framers/agentos/channels';

const router = new ChannelRouter();
router.registerAdapter(telegramAdapter); // an initialized IChannelAdapter

// Bind a conversation to an agent; onMessage handlers run only for bound conversations
router.addBinding({
  bindingId: 'support-telegram',
  seedId: 'support-agent',
  ownerUserId: 'owner-1',
  platform: 'telegram',
  channelId: '123456789',
  conversationType: 'direct',
  isActive: true,
  autoBroadcast: false,
});

router.onMessage(async (message, binding, session) => {
  await router.sendMessage(binding.seedId, message.platform, message.conversationId, {
    blocks: [{ type: 'text', text: `You said: ${message.text}` }],
  });
});

Social Posting​

SocialPostManager and ContentAdaptationEngine (in channels/social-posting/) handle cross-platform publishing. The adaptation engine reformats content for each platform's constraints (character limits, media formats, hashtag conventions).

The multi-channel-post, social-analytics, media-upload and bulk-scheduler tools ship as extension packs in registry/curated/tools/ of agentos-extensions.

For details, see Channels, Social Posting, and Telephony Providers.

Observability​

AgentOS provides opt-in observability through OpenTelemetry integration, configured via AgentOSObservabilityConfig.

Tracing​

When observability.tracing.enabled is true, AgentOS creates spans for:

  • Runtime turns (agentos.turn), GMI acquisition (agentos.gmi.get_or_create) and the GMI turn (agentos.gmi.process_turn_stream)
  • Tool-result handoffs (agentos.tool_result) and resumed external tool calls (agentos.resume_external_tool_request)
  • Conversation saves (agentos.conversation.save)
  • The high-level helpers (agentos.api.generate_text, agentos.api.stream_text, agentos.api.generate_image and the other media calls)

The Tracer class (safety/evaluation/observability/Tracer.ts) wraps @opentelemetry/api and uses the configured tracer name (default "@framers/agentos"). Trace context is propagated through AgentOSResponse metadata when includeTraceInResponses is enabled, allowing client-side correlation.

Metrics​

When observability.metrics.enabled is true, AgentOS exports (otel.ts):

  • agentos.turns -- Counter of completed turns
  • agentos.turn.duration_ms and agentos.turn.first_part_ms -- Histograms of turn latency
  • agentos.turn.tokens.total, .prompt, .completion, .cache_read, .cache_creation -- Histograms of tokens per turn
  • agentos.turn.cost.usd and agentos.turn.task_success_score -- Histograms of cost and outcome score per turn
  • agentos.tool_results and agentos.tool_result.duration_ms -- Tool-result handoffs and their duration

Logging​

PinoLogger injects trace_id and span_id fields when observability.logging.includeTraceIds is true. Optional exportToOtel emits LogRecord objects via @opentelemetry/api-logs. With content-safe telemetry on (enableContentSafeTelemetry from @framers/agentos/logging), it writes no record: the host's sink receives one fixed event per record instead, and spans, metrics and log export stay off.

Evaluation Framework​

Evaluator and LLMJudge (safety/evaluation/) provide a grading framework for agent outputs. SqlTaskOutcomeTelemetryStore persists per-turn outcome KPI windows so rolling quality metrics survive restarts.

For details, see Observability, Logging, and Evaluation Framework.

Emergent Capabilities​

The cognition/emergent/ module enables agents to create new tools at runtime within safety bounds.

SandboxedToolForge​

When emergent: true is set in AgentOSConfig, the agent gains access to the forge_tool meta-tool. The forge pipeline works as follows:

  1. The agent generates JavaScript code for a new tool (name, description, input schema, implementation)
  2. SandboxedToolForge performs static validation, rejecting dangerous patterns (eval, Function, process, require, import, child_process, and every file-mutating call but fs.writeFile and fs.unlink, which need the request to name fs.write and fs.delete)
  3. Validated code executes on the forge's executor: by default an in-process node:vm context (not a security boundary, per Node's documentation) via CodeSandbox, or a QuickJS WebAssembly instance per call with QuickJSExecutor, with configurable bounds:
    • Memory: observed as a heap delta in-process; QuickJSExecutor stops the guest at it
    • Timeout: 5,000 ms default
    • API allowlist without a ceiling: fetch (any method and host; fetchDomainAllowlist checks the first URL's host when a host sets it), fs.readFile (under fsReadRoots, the working directory by default; 1 MB, checked after the read), crypto (hash, HMAC, UUID)
    • Under a ceiling (emergentConfig.capabilities): each capability scoped to the host's hosts, roots and bounds through one broker that records every call; fs.write, fs.delete and the state-changing fetch methods can be granted on QuickJSExecutor, and a forge's test cases run as dry runs (see EMERGENT_CAPABILITIES.md)
  4. EmergentJudge evaluates the tool against safety criteria before permanent registration
  5. EmergentToolRegistry persists approved tools via IStorageAdapter

Additional Emergent Tools​

For details, see Emergent Capabilities and Self-Extension.