Skip to main content

Class: EmergentJudge

Defined in: packages/agentos/src/cognition/emergent/EmergentJudge.ts:212

Evaluates forged tools for safety, correctness, and quality using LLM-as-judge.

Three evaluation modes, each scaled to the risk level of the operation:

ModeLLM callsWhen used
reviewCreation1Newly forged tool — full code audit + test validation
validateReuse0Every invocation — pure programmatic schema check
reviewPromotion2Tier promotion — dual-judge safety + correctness panel

Example​

const judge = new EmergentJudge({
  judgeModel: 'gpt-4o-mini',
  promotionModel: 'gpt-4o',
  generateText: async (model, prompt) => callLlm(model, prompt),
});

// Creation review
const verdict = await judge.reviewCreation(candidate);
if (verdict.approved) { registry.register(tool, 'session'); }

// Reuse validation (no LLM call)
const reuse = judge.validateReuse('tool-1', output, outputSchema);
if (!reuse.valid) { throw new Error(reuse.schemaErrors.join(', ')); }

// Promotion panel
const promotion = await judge.reviewPromotion(tool);
if (promotion.approved) { registry.promote(tool.id, 'agent'); }

Constructors​

Constructor​

new EmergentJudge(config): EmergentJudge

Defined in: packages/agentos/src/cognition/emergent/EmergentJudge.ts:223

Create a new EmergentJudge instance.

Parameters​

config​

EmergentJudgeConfig

Judge configuration specifying models and the LLM callback. The generateText function is called for creation reviews and promotion panels but never for reuse validation (which is purely programmatic).

Returns​

EmergentJudge

Methods​

reviewCreation()​

reviewCreation(candidate): Promise<CreationVerdict>

Defined in: packages/agentos/src/cognition/emergent/EmergentJudge.ts:250

Full code + test review for a newly forged tool.

Builds a structured prompt from the candidate's details (name, description, schemas, source code, sandbox allowlist, test results) and asks the LLM to evaluate four dimensions: SAFETY, CORRECTNESS, DETERMINISM, BOUNDED.

The tool is approved only if both safety.passed AND correctness.passed are true in the LLM response.

If the LLM returns malformed JSON that cannot be parsed, a rejected verdict is returned with confidence 0 and a reasoning string explaining the parse failure. This prevents bad LLM output from accidentally approving a tool.

Parameters​

candidate​

ToolCandidate

The tool candidate to evaluate. Must include source code and at least one test result.

Returns​

Promise<CreationVerdict>

A CreationVerdict indicating approval or rejection with per-dimension scores and reasoning.

reviewPromotion()​

reviewPromotion(tool): Promise<PromotionVerdict>

Defined in: packages/agentos/src/cognition/emergent/EmergentJudge.ts:357

Two-judge panel for tier promotion. Both must approve.

Sends two independent LLM calls in parallel using the promotion model:

  1. Safety auditor: Reviews the tool's source code and usage history for security concerns (data exfiltration, resource exhaustion, API abuse).
  2. Correctness reviewer: Reviews the tool's source code and all historical outputs for correctness issues (schema violations, edge case failures).

Both reviewers must return approved: true for the promotion to pass. If either reviewer's response fails to parse as JSON, the promotion is rejected.

Parameters​

tool​

EmergentTool

The emergent tool to evaluate for promotion. Must have usage stats and judge verdicts from prior reviews.

Returns​

Promise<PromotionVerdict>

A PromotionVerdict containing both sub-verdicts and the combined approval decision.

validateReuse()​

validateReuse(_toolId, output, schema): ReuseVerdict

Defined in: packages/agentos/src/cognition/emergent/EmergentJudge.ts:323

Pure schema validation on each reuse — no LLM call.

Validates that output conforms to the declared schema using basic type checking. This runs on every tool invocation so it must be fast — no LLM calls, no network I/O, no async operations.

Checks performed:

  • If schema declares type: 'object', verify output is a non-null object.
  • If schema declares properties, verify each declared property key exists on the output object.
  • If schema declares required, verify each required property key exists.
  • If schema declares type: 'string', verify output is a string.
  • If schema declares type: 'number' or type: 'integer', verify output is a number.
  • If schema declares type: 'boolean', verify output is a boolean.
  • If schema declares type: 'array', verify output is an array.

Parameters​

_toolId​

string

The ID of the tool being reused (reserved for future anomaly detection; currently unused).

output​

unknown

The actual output value produced by the tool invocation.

schema​

JSONSchemaObject

The tool's declared output JSON Schema.

Returns​

ReuseVerdict

A ReuseVerdict with valid: true if the output conforms, or valid: false with a schemaErrors array describing each mismatch.