Skip to main content

Class: LLMJudge

Defined in: packages/agentos/src/safety/evaluation/LLMJudge.ts:142

LLM-based judge for semantic evaluation

Constructors​

Constructor​

new LLMJudge(config): LLMJudge

Defined in: packages/agentos/src/safety/evaluation/LLMJudge.ts:151

Parameters​

config​

LLMJudgeConfig

Returns​

LLMJudge

Methods​

batchJudge()​

batchJudge(evaluations, criteria?, concurrency?): Promise<JudgmentResult[]>

Defined in: packages/agentos/src/safety/evaluation/LLMJudge.ts:294

Batch evaluate multiple outputs

Parameters​

evaluations​

object[]

criteria?​

JudgeCriteria[]

concurrency?​

number = 3

Returns​

Promise<JudgmentResult[]>

compare()​

compare(input, outputA, outputB, criteria?): Promise<{ reasoning: string; scoreA: number; scoreB: number; winner: "A" | "B" | "tie"; }>

Defined in: packages/agentos/src/safety/evaluation/LLMJudge.ts:260

Compare two outputs and determine which is better

Parameters​

input​

string

outputA​

string

outputB​

string

criteria?​

JudgeCriteria[]

Returns​

Promise<{ reasoning: string; scoreA: number; scoreB: number; winner: "A" | "B" | "tie"; }>

createScorer()​

createScorer(criteria?): ScorerFunction

Defined in: packages/agentos/src/safety/evaluation/LLMJudge.ts:249

Create a scorer function for use with Evaluator

Parameters​

criteria?​

JudgeCriteria[]

Returns​

ScorerFunction

judge()​

judge(input, actualOutput, expectedOutput?, criteria?): Promise<JudgmentResult>

Defined in: packages/agentos/src/safety/evaluation/LLMJudge.ts:173

Judge an AI output against criteria

Parameters​

input​

string

actualOutput​

string

expectedOutput?​

string

criteria?​

JudgeCriteria[]

Returns​

Promise<JudgmentResult>