Skip to main content

Class: EmergentJudge

Defined in: packages/agentos/src/cognition/emergent/EmergentJudge.ts:304

Evaluates forged tools for safety, correctness, and quality using LLM-as-judge.

Three evaluation modes, each scaled to the risk level of the operation:

ModeLLM callsWhen used
reviewCreation1Newly forged tool — full code audit + test validation
validateReuse0Every invocation — pure programmatic schema check
reviewPromotion2Tier promotion — dual-judge safety + correctness panel

Example​

const judge = new EmergentJudge({
judgeModel: 'gpt-4o-mini',
promotionModel: 'gpt-4o',
generateText: async (model, prompt) => callLlm(model, prompt),
});

// Creation review
const verdict = await judge.reviewCreation(candidate);
if (verdict.approved) { registry.register(tool, 'session'); }

// Reuse validation (no LLM call)
const reuse = judge.validateReuse('tool-1', output, outputSchema);
if (!reuse.valid) { throw new Error(reuse.schemaErrors.join(', ')); }

// Promotion panel
const promotion = await judge.reviewPromotion(tool);
if (promotion.approved) { registry.promote(tool.id, 'agent'); }

Constructors​

Constructor​

new EmergentJudge(config): EmergentJudge

Defined in: packages/agentos/src/cognition/emergent/EmergentJudge.ts:315

Create a new EmergentJudge instance.

Parameters​

config​

EmergentJudgeConfig

Judge configuration specifying models and the LLM callback. The generateText function is called for creation reviews and promotion panels but never for reuse validation (which is purely programmatic).

Returns​

EmergentJudge

Methods​

judgeAuthor()​

judgeAuthor(): object

Defined in: packages/agentos/src/cognition/emergent/EmergentJudge.ts:446

The model and provider this judge's creation review asks for, as its configuration names them, for a forge trace's provenance. A providerId function is called now. A provider fallback in the host's generateText can answer with another model.

Returns​

object

The configured judgeModel and the provider, each null when unset or unknown.

model​

model: string | null

providerId​

providerId: string | null


reviewCreation()​

reviewCreation(candidate): Promise<CreationVerdict>

Defined in: packages/agentos/src/cognition/emergent/EmergentJudge.ts:342

Full code + test review for a newly forged tool.

Builds a structured prompt from the candidate's details (name, description, schemas, source code, sandbox allowlist, test results) and asks the LLM to evaluate four dimensions: SAFETY, CORRECTNESS, DETERMINISM, BOUNDED.

The tool is approved only if both safety.passed AND correctness.passed are true in the LLM response.

If the LLM returns malformed JSON that cannot be parsed, a rejected verdict is returned with confidence 0 and a reasoning string explaining the parse failure. This prevents bad LLM output from accidentally approving a tool.

Parameters​

candidate​

ToolCandidate

The tool candidate to evaluate. Must include source code and at least one test result.

Returns​

Promise<CreationVerdict>

A CreationVerdict indicating approval or rejection with per-dimension scores and reasoning.


reviewCreationWithDetail()​

reviewCreationWithDetail(candidate): Promise<{ author: { model: string | null; providerId: string | null; }; detail: ForgeTraceJudgeDetail; verdict: CreationVerdict; }>

Defined in: packages/agentos/src/cognition/emergent/EmergentJudge.ts:371

The creation review of reviewCreation, with the reply's own fields that its verdict drops.

Makes the same one call and returns the same verdict: approved only when the reply's safety.passed and correctness.passed are both true, with determinism and bounded at 1.0 or 0.5. Beside it, detail holds what the reply said for itself: its own approved, each pass and likely flag (null when the reply's value is not a boolean), every safety concern (a non-string written as its JSON text), the finite numbers of its failed-test list in order, and the determinism and boundedness reasoning (a non-string written as its JSON text, null when absent). Texts and lists are kept whole; a forge trace cuts them to its caps.

When the call fails, detail.judgeError is 'llm_call_failed'; when the reply does not parse as JSON, it is 'reply_unparsable'. In both, the reply's fields are null and its lists empty, and neither the provider's error nor the raw reply is kept.

author is judgeAuthor as it reads when the request is sent: its provider is read in the same synchronous step as the call to the host's callback.

Parameters​

candidate​

ToolCandidate

The tool candidate to evaluate.

Returns​

Promise<{ author: { model: string | null; providerId: string | null; }; detail: ForgeTraceJudgeDetail; verdict: CreationVerdict; }>

The verdict reviewCreation returns, the reply's detail, and the model and provider the request was sent to.


reviewPromotion()​

reviewPromotion(tool): Promise<PromotionVerdict>

Defined in: packages/agentos/src/cognition/emergent/EmergentJudge.ts:527

Two-judge panel for tier promotion. Both must approve.

Sends two independent LLM calls in parallel using the promotion model:

  1. Safety auditor: Reviews the tool's source code and usage history for security concerns (data exfiltration, resource exhaustion, API abuse).
  2. Correctness reviewer: Reviews the tool's source code and all historical outputs for correctness issues (schema violations, edge case failures).

Both reviewers must return approved: true for the promotion to pass. If either reviewer's response fails to parse as JSON, the promotion is rejected.

Parameters​

tool​

EmergentTool

The emergent tool to evaluate for promotion. Must have usage stats and judge verdicts from prior reviews.

Returns​

Promise<PromotionVerdict>

A PromotionVerdict containing both sub-verdicts and the combined approval decision.


validateReuse()​

validateReuse(_toolId, output, schema): ReuseVerdict

Defined in: packages/agentos/src/cognition/emergent/EmergentJudge.ts:491

Pure schema validation on each reuse — no LLM call.

Validates that output conforms to the declared schema using basic type checking. This runs on every tool invocation so it must be fast — no LLM calls, no network I/O, no async operations.

Checks performed:

  • If schema declares type: 'object', verify output is a non-null object.
  • If schema declares properties, verify each declared property key exists on the output object.
  • If schema declares required, verify each required property key exists.
  • If schema declares type: 'string', verify output is a string.
  • If schema declares type: 'number' or type: 'integer', verify output is a number.
  • If schema declares type: 'boolean', verify output is a boolean.
  • If schema declares type: 'array', verify output is an array.

Parameters​

_toolId​

string

The ID of the tool being reused (reserved for future anomaly detection; currently unused).

output​

unknown

The actual output value produced by the tool invocation.

schema​

The tool's declared output JSON Schema; a tool forged with none (forge_tool does not ask for one) gives every output valid: true.

JSONSchemaObject | undefined

Returns​

ReuseVerdict

A ReuseVerdict with valid: true if the output conforms, or valid: false with a schemaErrors array describing each mismatch.