Capability Discovery

Overview

Putting every registered tool schema into the system prompt grows the prompt with each tool, skill and channel, and the model reads all of it on every turn, most of it irrelevant to the turn. Research calls the quality loss that follows context rot (Chroma 2025).

The Capability Discovery Engine solves this with a three-tier context model:

Tier	Default budget	Content	When
Tier 0	200 tokens	Category summaries	Always in context
Tier 1	800 tokens	Top-5 capability summaries	Per-turn semantic retrieval
Tier 2	2,000 tokens	Full schema/content for top-2	Per-turn deep pull
Total	at most 3,000 tokens

The budgets, tier1TopK and tier2TopK are configurable (Configuration). Agents also get a meta-tool, discover_capabilities, for active search when the passive tiers miss something.

Architecture

Per-turn data flow:

User Message
  → CapabilityIndex.search()                // semantic vector search
  → CapabilityGraph.rerank()                // boost related capabilities via graph edges
  → CapabilityContextAssembler.assemble()   // token-budgeted tier assembly
  → CapabilityDiscoveryResult               // injected into prompt

Component	Responsibility
`CapabilityDiscoveryEngine`	Top-level orchestrator — coordinates index, graph, and assembler
`CapabilityIndex`	Normalizes sources into `CapabilityDescriptor`; embeds and stores in vector index
`CapabilityGraph`	Graphology relationship graph (4 edge types); provides re-ranking boosts
`CapabilityContextAssembler`	Builds Tier 0/1/2 context within hard token budgets
`CapabilityEmbeddingStrategy`	Constructs intent-oriented embedding text per capability
`CapabilityManifestScanner`	Scans for `CAPABILITY.yaml` manifests; hot-reload via `fs.watch`
`createDiscoverCapabilitiesTool`	Factory for the `discover_capabilities` meta-tool

Three-Tier Context Model

Tier 0 — Always in Context (200-token default budget)

Category summaries giving the model a high-level map. Regenerated only when capabilities change (version-tracked cache).

Available capability categories:
- Communication: telegram, discord, slack, whatsapp (+16 more) (20)
- Information: web-search, news-search, web-browser (3)
- Developer-tools: github, git, cli-executor (3)
Use discover_capabilities tool to get details on any capability.

Tier 1 — Semantic Retrieval (800-token default budget)

Per-turn top-5 retrieval as compact summaries. Minimum relevance threshold: 0.3.

Relevant capabilities:
web-search (tool). Search the web for current information. Params: query, max_results
news-search (tool). Search news articles by keyword. Params: query, date_range
web-browser (tool). Browse a URL and extract content. Params: url, selector
github (skill). Use the GitHub CLI for issues, PRs, repos. Requires: gh
summarizer (skill). Summarize long documents into key points

Tier 2 — Full Details (2,000-token default budget)

Full schema or SKILL.md content for the top-2 from Tier 1:

# Web Search
Kind: tool | Category: information

Search the web for current information using the Serper API.

## Input Schema
  query (string, required): The search query
  max_results (number): Maximum results to return (default: 5)
  search_type (string): Type of search [search|news|images] (default: "search")

Required secrets: SERPER_API_KEY

The assembler holds each tier to its budget, counting four characters as one token.

Source Normalization

CapabilityIndex normalizes five source types into unified CapabilityDescriptor objects:

Source	ID Convention	Kind	Example
Tools (`ITool`)	`tool:{name}`	`tool`	`tool:web-search`
Skills (`SKILL.md`)	`skill:{name}`	`skill`	`skill:github`
Extensions (catalog)	`extension:{name}`	`extension`	`extension:giphy`
Channels (platform)	`channel:{platform}`	`channel`	`channel:telegram`
Manifests (CAPABILITY.yaml)	`{kind}:{name}` or custom	any	`tool:my-custom-tool`

Each descriptor carries: id, kind, name, displayName, description, category, tags, requiredSecrets, requiredTools, available, hasSideEffects, and lazy-load fields fullSchema (Tier 2 tool schemas) and fullContent (Tier 2 skill content). A sourceRef discriminated union points back to the original source for on-demand loading.

Normalization is deterministic and runs during initialize(). Skills derive displayName by capitalizing name segments. Channels always get category: 'communication'. Extensions inherit availability from the catalog entry.

Embedding Strategy

CapabilityEmbeddingStrategy constructs a concise text per capability (100-300 tokens) optimized for semantic matching against user intents. Informed by ToolLLM (NDCG@5 of 84.9 on 16K+ APIs) and MCP-RAG parameter-level decomposition.

Field	Why	Example
Name + display name	Exact-match queries	`"Web Search (web-search)"`
Description	Core semantic content	`"Search the web for current information"`
Category	Categorical queries	`"Category: information"`
Tags	Use-case queries	`"Use cases: search, api, news"`
Parameter names (tools)	Action queries	`"Parameters: query, max_results"`
Dependencies	Composition queries	`"Requires: gh, git"`

Fields not embedded: fullSchema, fullContent, requiredSecrets — these are Tier 2 data loaded on demand to keep embedding text lean.

Graph Relationships

CapabilityGraph uses graphology (shared with GraphRAGEngine) for neighbor lookups. All edges are built deterministically from metadata.

Edge Types

Edge Type	Source	Weight	Example
`DEPENDS_ON`	`requiredTools` frontmatter	1.0	`skill:github` -> `tool:cli-executor`
`COMPOSED_WITH`	Preset co-occurrence	0.5	`tool:web-search` <-> `skill:summarizer`
`TAGGED_WITH`	Shared tags (>= 2 overlap)	0.3/tag	`tool:news-search` <-> `tool:web-search`
`SAME_CATEGORY`	Same `kind:category` (2-8 group)	0.1	`tool:web-search` <-> `tool:web-browser`

Re-Ranking Algorithm

CapabilityGraph.rerank() runs after semantic search:

For each result, look up 1-hop graph neighbors.
If a neighbor is also in results, mutual boost: score += graphBoostFactor * edgeWeight.
If a neighbor is not in results but has DEPENDS_ON or COMPOSED_WITH, pull it in: score = parentScore * graphBoostFactor * edgeWeight.
Re-sort by score descending.

If a user asks about GitHub issues and skill:github ranks high, tool:cli-executor (its dependency) gets pulled in even without a direct query match.

Meta-Tool: `discover_capabilities`

When passive tiers miss something, agents actively search via the discover_capabilities tool:

discover_capabilities({ query: "send a message on Discord", kind: "channel" })
// → { capabilities: [{ id: "channel:discord", relevance: 0.91, ... }], totalIndexed: 66 }

The tool runs through the same discover() pipeline and returns Tier 1 results. It is always included in listDiscoveredTools() output regardless of query relevance.

import { createDiscoverCapabilitiesTool } from '@framers/agentos/discovery';
// With the tool orchestrator as the second argument, each result also says whether it is
// loadable at run time (an extension in the registry catalog that is not loaded yet) and its extensionId.
const metaTool = createDiscoverCapabilitiesTool(discoveryEngine, toolOrchestrator);
await toolOrchestrator.registerTool(metaTool);

File-Based Discovery

Custom capabilities defined via CAPABILITY.yaml, scanned by CapabilityManifestScanner.

Default scan directories (scanner.getDefaultDirs(), in order; scan(dirs) takes your own list):

~/.wunderland/capabilities/ (user-global)
./.wunderland/capabilities/ (workspace-local)
$WUNDERLAND_CAPABILITY_DIRS (env var, colon-separated)

Directory structure (one folder per capability):

~/.wunderland/capabilities/my-custom-tool/
  CAPABILITY.yaml   # required (or CAPABILITY.yml)
  SKILL.md          # optional (loaded as fullContent when skillContent is not set)
  schema.json       # optional (loaded as fullSchema when inputSchema is not set)

CAPABILITY.yaml format:

id: tool:my-custom-tool
kind: tool
name: my-custom-tool
displayName: My Custom Tool
description: Searches a proprietary API for internal documents
category: information
tags: [search, internal, documents]
requiredSecrets: [INTERNAL_API_KEY]
hasSideEffects: false
inputSchema: {"type": "object", "properties": {"query": {"type": "string"}}}
skillContent: ./SKILL.md

Required fields: name, kind, description; a manifest without one is skipped with a warning. The id defaults to ${kind}:${name}, displayName to the name and category to custom. The scanner reads a simple YAML subset: scalars, lists (inline, or one indented - item per line), and objects written inline as JSON, so inputSchema is JSON on one line or lives in schema.json.

skillContent names a file inside the capability's folder, its names joined by / after an optional leading ./; no name in it is . or .. or starts with a dot (resolveSkillContentPath). A manifest that sets it to an empty value, null or no value, false or a number breaks the rule too. The manifest, the skill's file and schema.json are read only when each is a regular file reached through no symbolic link at the capability's folder or below it; on Linux the scanner also checks that the file it opened lies below the capability's own folder, and when that check cannot run (no /proc, or a folder's real path it cannot get) it reads without it and writes one warning per scan. A capability that breaks either rule is left out with a warning, and scanner.scanWithRefusals(dirs) returns it in refused with its manifestPath, field and problem.

Hot-reload via fs.watch with debouncing (default 500ms); the callback also gets the refused capabilities:

const scanner = new CapabilityManifestScanner();
scanner.watch(scanner.getDefaultDirs(), async (descriptors, refused) => {
  await discoveryEngine.refreshIndex({ manifests: descriptors });
  for (const { manifestPath, field, problem } of refused) {
    console.warn(`capability left out: ${manifestPath} (${field}: ${problem})`);
  }
});

Integration Points

PromptBuilder — render discovery result into system prompt text:

const result = await discoveryEngine.discover(userMessage);
const contextText = discoveryEngine.renderForPrompt(result);

ToolOrchestrator — filter tool schemas to only discovered tools via listDiscoveredTools():

const toolSchemas = await toolOrchestrator.listDiscoveredTools(discoveryResult);
// Returns only Tier 1/2 tools + discover_capabilities meta-tool

A host's own loop — wire it before prompt composition (promptBuilder and provider stand for the host's own; the AgentOS runtime does this itself through the turn planner below):

const discoveryResult = await discoveryEngine.discover(userMessage);
const tools = await toolOrchestrator.listDiscoveredTools(discoveryResult);
const capabilityContext = discoveryEngine.renderForPrompt(discoveryResult);
const prompt = promptBuilder.build({ capabilityContext, ...otherInputs });
const response = await provider.complete({ prompt, tools });

AgentOS Turn Planner (Core Integration)

The AgentOS runtime runs a turn planner before each GMI turn:

Sets tool failure policy (fail_open or fail_closed)
Applies dynamic tool selection (discovered or all)
Injects discovery context into prompt metadata

Defaults:

defaultToolFailureMode: "fail_open"
discovery enabled
fallback to full toolset when discovery fails or yields no tool matches

With autoInitializeEngine (the default), the runtime builds the engine itself on an in-memory vector store: it indexes the runtime's tools, the loaded extension packs, the messaging channels, the curated registry's capability manifests and what discovery.sources adds (skills, manifests, extra extensions and channels), and registers the discover_capabilities and load_capability_extension tools unless registerMetaTool is false.

A curated manifest describes a pack as the catalog lists it (some packs as not available until set up, with no input schema), so it never replaces what the host holds: a tool the host has loaded keeps its own descriptor (available, with its input schema) and stays among the capabilities onlyAvailable discovery returns, a manifest in discovery.sources.manifests keeps its id, and a pack the catalog lists under several entries is indexed once per tool. CapabilityDiscoveryInitializer.indexSourcesWithCurated() builds these sources, and curatedDescriptorsBeside() does the filtering.

await agentos.initialize({
  // ...
  turnPlanning: {
    enabled: true,
    defaultToolFailureMode: 'fail_open',
    allowRequestOverrides: true,
    discovery: {
      enabled: true,
      autoInitializeEngine: true,
      registerMetaTool: true,
      onlyAvailable: true,
      defaultToolSelectionMode: 'discovered',
      includePromptContext: true,
      maxRetries: 1,
      retryBackoffMs: 150,
    },
  },
});

Per-request overrides can be provided via options.customFlags:

toolFailureMode: fail_open | fail_closed
toolSelectionMode: all | discovered
capabilityDiscoveryKind: tool | skill | extension | channel | any

Runtime metadata updates include executionLifecycle transitions:

planned -> executing
degraded (if discovery fallback is applied in fail-open mode)
recovered (optional, when turn completes successfully after degradation)
completed or errored

AgentOS also emits taskOutcome in metadata updates at the end of each turn:

status: success | partial | failed
score: normalized score in [0, 1]
source: heuristic or request_override

When task outcome telemetry is enabled (default), AgentOS also emits taskOutcomeKpi as a rolling stream payload (windowed success stats):

scopeKey: aggregation key (global / org / org+persona)
sampleCount, successCount, partialCount, failedCount
successRate
averageScore / weightedSuccessRate

taskOutcome can be overridden per request via options.customFlags:

taskOutcome: success | partial | failed | numeric 0..1
taskSuccess: boolean

Task outcome telemetry can be configured under orchestratorConfig.taskOutcomeTelemetry:

enabled (default true)
rollingWindowSize (default 100)
scope: global | organization | organization_persona (default)
emitAlerts (default true)
alertBelowWeightedSuccessRate (default 0.55)
alertMinSamples (default 8)
alertCooldownMs (default 60000)

When alerting is enabled and KPI degrades, metadata updates include taskOutcomeAlert with severity/reason/threshold/value so clients can trigger automated remediation.

To persist KPI windows across restarts, set AgentOSConfig.taskOutcomeTelemetryStore (the runtime passes it to the orchestrator). The store contract is:

loadWindows(): Promise<Record<string, TaskOutcomeKpiWindowEntry[]>>
saveWindow(scopeKey, entries): Promise<void>

AgentOS includes a built-in SQL implementation:

import { SqlTaskOutcomeTelemetryStore } from '@framers/agentos';

const taskOutcomeTelemetryStore = new SqlTaskOutcomeTelemetryStore({
  // @framers/sql-storage-adapter resolution options, plus an optional tableName
  // (default agentos_task_outcome_kpi_windows).
  priority: ['better-sqlite3', 'sqljs'],
  filePath: './data/agentos_task_outcomes.db',
});

Adaptive recovery can be configured under orchestratorConfig.adaptiveExecution:

enabled (default true)
minSamples (default 5)
minWeightedSuccessRate (default 0.7)
forceAllToolsWhenDegraded (default true)
forceFailOpenWhenDegraded (default true)

When enabled, if rolling task KPI degrades below threshold, AgentOS can automatically switch turn policy from toolSelectionMode=discovered to toolSelectionMode=all to recover task success rate. It can also force toolFailureMode=fail_open unless the request explicitly pinned toolFailureMode=fail_closed via options.customFlags.

Configuration

interface CapabilityDiscoveryConfig {
  tier0TokenBudget: number;     // Default: 200
  tier1TokenBudget: number;     // Default: 800
  tier2TokenBudget: number;     // Default: 2000
  tier1TopK: number;            // Default: 5
  tier2TopK: number;            // Default: 2
  tier1MinRelevance: number;    // Default: 0.3 (0-1 scale)
  useGraphReranking: boolean;   // Default: true
  collectionName: string;       // Default: 'capability_index'
  embeddingModelId?: string;    // Default: undefined (use system default)
  graphBoostFactor: number;     // Default: 0.15 (0-1 scale)
}

Override at construction time or per-query:

const engine = new CapabilityDiscoveryEngine(embeddingManager, vectorStore, {
  tier1TopK: 8, tier2TopK: 3, graphBoostFactor: 0.2,
});

const result = await engine.discover("search the web", {
  config: { tier2TokenBudget: 3000 },
  kind: 'tool',
  onlyAvailable: true,
});

Cost

initialize() embeds every indexed capability once; refreshIndex() embeds and upserts the capabilities it is given, then rebuilds the graph.
Each discover() embeds the query once, searches the vector store and re-ranks over the in-memory graph.
The prompt context is held to the tier budgets: at most 3,000 tokens at the defaults, counted at four characters a token. The discover_capabilities schema adds one tool to the tool list.

Usage Example

import {
  CapabilityDiscoveryEngine,
  CapabilityManifestScanner,
  createDiscoverCapabilitiesTool,
  type CapabilityIndexSources,
} from '@framers/agentos/discovery';

// Stand-ins: an embedding manager and a vector store from your memory wiring,
// the runtime's tool orchestrator, and your catalog entries in the
// CapabilityIndexSources shape (tools, skills, extensions, channels).
declare const embeddingManager: any;
declare const vectorStore: any;
declare const toolOrchestrator: any;
declare const catalog: Omit<CapabilityIndexSources, 'manifests'>;

// --- Initialization (once at startup) ---

const engine = new CapabilityDiscoveryEngine(embeddingManager, vectorStore);
const scanner = new CapabilityManifestScanner();
const manifests = await scanner.scan();

await engine.initialize({ ...catalog, manifests });

await toolOrchestrator.registerTool(createDiscoverCapabilitiesTool(engine, toolOrchestrator));

scanner.watch(scanner.getDefaultDirs(), async (descs) => {
  await engine.refreshIndex({ manifests: descs });
});

console.log(engine.getStats());
// { capabilityCount, graphNodes, graphEdges, indexVersion }

// --- Per-turn discovery ---

const discoveryResult = await engine.discover('Search the web for AI news and summarize it');
// discoveryResult.tokenEstimate.totalTokens stays within the tier budgets

const tools = await toolOrchestrator.listDiscoveredTools(discoveryResult);
// the Tier 1/2 tools plus discover_capabilities

const capabilityContext = engine.renderForPrompt(discoveryResult);
// Inject into the system prompt

Source Files

All source lives in src/cognition/discovery/:

File	Export
`types.ts`	All types, `DEFAULT_DISCOVERY_CONFIG`
`CapabilityDiscoveryEngine.ts`	`CapabilityDiscoveryEngine`
`CapabilityIndex.ts`	`CapabilityIndex`
`CapabilityGraph.ts`	`CapabilityGraph`
`CapabilityContextAssembler.ts`	`CapabilityContextAssembler`
`CapabilityEmbeddingStrategy.ts`	`CapabilityEmbeddingStrategy`
`CapabilityManifestScanner.ts`	`CapabilityManifestScanner`
`DiscoverCapabilitiesTool.ts`	`createDiscoverCapabilitiesTool()`
`index.ts`	Barrel re-exports for `@framers/agentos/discovery`

Overview​

Architecture​

Three-Tier Context Model​

Tier 0 — Always in Context (200-token default budget)​

Tier 1 — Semantic Retrieval (800-token default budget)​

Tier 2 — Full Details (2,000-token default budget)​

Source Normalization​

Embedding Strategy​

Graph Relationships​

Edge Types​

Re-Ranking Algorithm​

Meta-Tool: discover_capabilities​

File-Based Discovery​

Integration Points​

AgentOS Turn Planner (Core Integration)​

Configuration​

Cost​

Usage Example​

Source Files​