Class Evaluator

Agent evaluation framework implementation.

Implements

IEvaluator

Index

Constructors

Methods

runEvaluation evaluateTestCase score registerScorer getRun listRuns compareRuns generateReport

Constructors

constructor

new Evaluator(): Evaluator
Returns Evaluator
- Defined in core/evaluation/Evaluator.ts:210

Methods

runEvaluation

runEvaluation(name, testCases, agentFn, config?): Promise<EvalRun>
Runs an evaluation suite against an agent.
Parameters
- name: string
  Name for this evaluation run
- testCases: EvalTestCase[]
  Test cases to evaluate
- agentFn: ((input, context?) => Promise<string>)
  Function that takes input and returns agent output
  - - (input, context?): Promise<string>
    - Parameters
      input: string
      Optional context: string
      Returns Promise<string>
- Optional config: EvalConfig
  Evaluation configuration
Returns Promise<EvalRun>
The completed evaluation run
Implementation of IEvaluator.runEvaluation
- Defined in core/evaluation/Evaluator.ts:220

evaluateTestCase

evaluateTestCase(testCase, actualOutput, config?): Promise<EvalTestResult>
Evaluates a single test case.
Parameters
- testCase: EvalTestCase
  The test case
- actualOutput: string
  The agent's actual output
- Optional config: EvalConfig
  Evaluation configuration
Returns Promise<EvalTestResult>
Test result
Implementation of IEvaluator.evaluateTestCase
- Defined in core/evaluation/Evaluator.ts:306

score

score(scorer, actual, expected?, references?): Promise<number>
Scores output using a specific scorer.
Parameters
- scorer: string
  Scorer name
- actual: string
  Actual output
- Optional expected: string
  Expected output
- Optional references: string[]
  Reference outputs
Returns Promise<number>
Score (0-1)
Implementation of IEvaluator.score
- Defined in core/evaluation/Evaluator.ts:365

registerScorer

registerScorer(name, fn): void
Registers a custom scorer.
Parameters
- name: string
  Scorer name
- fn: ScorerFunction
  Scoring function
Returns void
Implementation of IEvaluator.registerScorer
- Defined in core/evaluation/Evaluator.ts:378

getRun

getRun(runId): Promise<undefined | EvalRun>
Gets an evaluation run by ID.
Parameters
- runId: string
  Run ID
Returns Promise<undefined | EvalRun>
The evaluation run or undefined
Implementation of IEvaluator.getRun
- Defined in core/evaluation/Evaluator.ts:382

listRuns

listRuns(limit?): Promise<EvalRun[]>
Lists recent evaluation runs.
Parameters
- limit: number = 50
  Maximum runs to return
Returns Promise<EvalRun[]>
Array of runs
Implementation of IEvaluator.listRuns
- Defined in core/evaluation/Evaluator.ts:386

compareRuns

compareRuns(runId1, runId2): Promise<EvalComparison>
Compares two evaluation runs.
Parameters
- runId1: string
  First run ID
- runId2: string
  Second run ID
Returns Promise<EvalComparison>
Comparison results
Implementation of IEvaluator.compareRuns
- Defined in core/evaluation/Evaluator.ts:392

generateReport

generateReport(runId, format): Promise<string>
Generates a report for a run.
Parameters
- runId: string
  Run ID
- format: "json" | "markdown" | "html"
  Report format
Returns Promise<string>
Report content
Implementation of IEvaluator.generateReport
- Defined in core/evaluation/Evaluator.ts:433