Class: EmergentJudge
Defined in: packages/agentos/src/cognition/emergent/EmergentJudge.ts:304
Evaluates forged tools for safety, correctness, and quality using LLM-as-judge.
Three evaluation modes, each scaled to the risk level of the operation:
| Mode | LLM calls | When used |
|---|---|---|
reviewCreation | 1 | Newly forged tool — full code audit + test validation |
validateReuse | 0 | Every invocation — pure programmatic schema check |
reviewPromotion | 2 | Tier promotion — dual-judge safety + correctness panel |
Example
const judge = new EmergentJudge({
judgeModel: 'gpt-4o-mini',
promotionModel: 'gpt-4o',
generateText: async (model, prompt) => callLlm(model, prompt),
});
// Creation review
const verdict = await judge.reviewCreation(candidate);
if (verdict.approved) { registry.register(tool, 'session'); }
// Reuse validation (no LLM call)
const reuse = judge.validateReuse('tool-1', output, outputSchema);
if (!reuse.valid) { throw new Error(reuse.schemaErrors.join(', ')); }
// Promotion panel
const promotion = await judge.reviewPromotion(tool);
if (promotion.approved) { registry.promote(tool.id, 'agent'); }
Constructors
Constructor
new EmergentJudge(
config):EmergentJudge
Defined in: packages/agentos/src/cognition/emergent/EmergentJudge.ts:315
Create a new EmergentJudge instance.
Parameters
config
Judge configuration specifying models and the LLM callback.
The generateText function is called for creation reviews and promotion
panels but never for reuse validation (which is purely programmatic).
Returns
EmergentJudge
Methods
judgeAuthor()
judgeAuthor():
object
Defined in: packages/agentos/src/cognition/emergent/EmergentJudge.ts:446
The model and provider this judge's creation review asks for, as its configuration names them, for a forge
trace's provenance. A providerId function is called now. A provider fallback in the host's generateText can
answer with another model.
Returns
object
The configured judgeModel and the provider, each null when unset or unknown.
model
model:
string|null
providerId
providerId:
string|null
reviewCreation()
reviewCreation(
candidate):Promise<CreationVerdict>
Defined in: packages/agentos/src/cognition/emergent/EmergentJudge.ts:342
Full code + test review for a newly forged tool.
Builds a structured prompt from the candidate's details (name, description, schemas, source code, sandbox allowlist, test results) and asks the LLM to evaluate four dimensions: SAFETY, CORRECTNESS, DETERMINISM, BOUNDED.
The tool is approved only if both safety.passed AND correctness.passed
are true in the LLM response.
If the LLM returns malformed JSON that cannot be parsed, a rejected verdict is returned with confidence 0 and a reasoning string explaining the parse failure. This prevents bad LLM output from accidentally approving a tool.
Parameters
candidate
The tool candidate to evaluate. Must include source code and at least one test result.
Returns
Promise<CreationVerdict>
A CreationVerdict indicating approval or rejection with per-dimension scores and reasoning.
reviewCreationWithDetail()
reviewCreationWithDetail(
candidate):Promise<{author: {model:string|null;providerId:string|null; };detail:ForgeTraceJudgeDetail;verdict:CreationVerdict; }>
Defined in: packages/agentos/src/cognition/emergent/EmergentJudge.ts:371
The creation review of reviewCreation, with the reply's own fields that its verdict drops.
Makes the same one call and returns the same verdict: approved only when the reply's safety.passed and
correctness.passed are both true, with determinism and bounded at 1.0 or 0.5. Beside it, detail holds
what the reply said for itself: its own approved, each pass and likely flag (null when the reply's value is not
a boolean), every safety concern (a non-string written as its JSON text), the finite numbers of its failed-test
list in order, and the determinism and boundedness reasoning (a non-string written as its JSON text, null when
absent). Texts and lists are kept whole; a forge trace cuts them to its caps.
When the call fails, detail.judgeError is 'llm_call_failed'; when the reply does not parse as JSON, it is
'reply_unparsable'. In both, the reply's fields are null and its lists empty, and neither the provider's error
nor the raw reply is kept.
author is judgeAuthor as it reads when the request is sent: its provider is read in the same synchronous
step as the call to the host's callback.
Parameters
candidate
The tool candidate to evaluate.
Returns
Promise<{ author: { model: string | null; providerId: string | null; }; detail: ForgeTraceJudgeDetail; verdict: CreationVerdict; }>
The verdict reviewCreation returns, the reply's detail, and the model and provider the request was sent to.
reviewPromotion()
reviewPromotion(
tool):Promise<PromotionVerdict>
Defined in: packages/agentos/src/cognition/emergent/EmergentJudge.ts:527
Two-judge panel for tier promotion. Both must approve.
Sends two independent LLM calls in parallel using the promotion model:
- Safety auditor: Reviews the tool's source code and usage history for security concerns (data exfiltration, resource exhaustion, API abuse).
- Correctness reviewer: Reviews the tool's source code and all historical outputs for correctness issues (schema violations, edge case failures).
Both reviewers must return approved: true for the promotion to pass. If
either reviewer's response fails to parse as JSON, the promotion is rejected.
Parameters
tool
The emergent tool to evaluate for promotion. Must have usage stats and judge verdicts from prior reviews.
Returns
Promise<PromotionVerdict>
A PromotionVerdict containing both sub-verdicts and the combined approval decision.
validateReuse()
validateReuse(
_toolId,output,schema):ReuseVerdict
Defined in: packages/agentos/src/cognition/emergent/EmergentJudge.ts:491
Pure schema validation on each reuse — no LLM call.
Validates that output conforms to the declared schema using basic type
checking. This runs on every tool invocation so it must be fast — no LLM
calls, no network I/O, no async operations.
Checks performed:
- If schema declares
type: 'object', verify output is a non-null object. - If schema declares
properties, verify each declared property key exists on the output object. - If schema declares
required, verify each required property key exists. - If schema declares
type: 'string', verify output is a string. - If schema declares
type: 'number'ortype: 'integer', verify output is a number. - If schema declares
type: 'boolean', verify output is a boolean. - If schema declares
type: 'array', verify output is an array.
Parameters
_toolId
string
The ID of the tool being reused (reserved for future anomaly detection; currently unused).
output
unknown
The actual output value produced by the tool invocation.
schema
The tool's declared output JSON Schema; a tool forged with none (forge_tool does not ask for
one) gives every output valid: true.
JSONSchemaObject | undefined
Returns
A ReuseVerdict with valid: true if the output conforms,
or valid: false with a schemaErrors array describing each mismatch.