Skip to main content

Evaluation Framework

Overview​

The evaluation framework runs test cases through an agent function and scores the answers:

  • Test cases with expected outputs and weighted criteria
  • Runs in batches of a configurable size, with a per-attempt timeout and retries
  • Scorers, built in or registered, and an LLM judge
  • Run comparison over four aggregate metrics
  • Reports in JSON, Markdown or HTML

Runs live in the Evaluator instance's memory.

Quick Start​

import { Evaluator } from '@framers/agentos';
import type { EvalTestCase } from '@framers/agentos';

const evaluator = new Evaluator();

// Define test cases
const testCases: EvalTestCase[] = [
  {
    id: 'math-1',
    name: 'Basic Addition',
    input: 'What is 2 + 2?',
    expectedOutput: '4',
    criteria: [
      { name: 'correctness', description: 'Contains correct answer', weight: 1, scorer: 'contains' }
    ]
  },
  {
    id: 'greeting-1',
    name: 'Greeting Response',
    input: 'Hello!',
    expectedOutput: 'Hello! How can I help you today?',
    criteria: [
      { name: 'politeness', description: 'Polite response', weight: 1, scorer: 'contains' },
      { name: 'similarity', description: 'Similar to expected', weight: 2, scorer: 'levenshtein' }
    ]
  }
];

// Define your agent function
async function myAgent(input: string): Promise<string> {
  // Your GMI or agent logic here
  return `Response to: ${input}`;
}

// Run evaluation
const run = await evaluator.runEvaluation(
  'My Agent Evaluation v1.0',
  testCases,
  myAgent,
  { concurrency: 5, timeoutMs: 30000 }
);

// Generate report
const report = await evaluator.generateReport(run.runId, 'markdown');
console.log(report);

Built-in Scorers​

The framework includes several built-in scorers:

ScorerDescriptionBest For
exact_matchReturns 1 if the strings are equal after trimming, ignoring casePrecise answers
containsReturns 1 if actual contains expected, ignoring caseChecking for key terms
levenshtein1 minus the normalized character edit distance (0-1), ignoring caseTypo tolerance
semantic_similarityWord overlap (0-1): shared words longer than 2 characters over the larger set; no embeddingsShared vocabulary
bleuUnigram precision with a brevity penalty (0-1)Translation quality
rougeROUGE-L F1 scoreSummarization quality
word_error_rate1 less the word error rate, never below 0Speech-to-text transcripts

Using Scorers Directly​

// Score individual outputs
const score = await evaluator.score('levenshtein', 'actual output', 'expected output');
console.log(`Similarity: ${(score * 100).toFixed(1)}%`);

Word Error Rate​

word_error_rate scores a transcript against a reference transcript by its word error rate. wordErrorRate() returns the counts behind the score and normalizeTranscript() the words it compares; both are exported beside Evaluator, from @framers/agentos and from @framers/agentos/safety/evaluation.

Both texts go through one normalisation before they are counted, so that measurements of different speech-to-text systems agree:

  • Unicode's composed form (NFC), so é written as one character and e followed by a combining accent are the same letter.
  • Lower case.
  • Letters, digits and a word's inner apostrophe are kept (cat's, don't), with the combining marks on them, such as an accent or a vowel sign, so कि and क are different words. The typographic apostrophes ’ (U+2019), ‘ (U+2018) and ʼ (U+02BC) count as ', so don’t and don't are the same word.
  • Other punctuation and symbols are dropped, with any combining mark on them (an emoji's variation selector, for one), and so are bracketed marks such as [laughter] and the filler words um, uh, er, ah, hmm and mm.
  • Numbers are left as written, so 3 and three are different words.

The counts are the substitutions, deletions and insertions of a word-level edit distance from the reference to the transcript: a deletion is a reference word the transcript lacks, an insertion a transcript word the reference lacks. The rate is their sum over the reference's word count, taken as 1 when the reference is empty, so insertions can take it above 1. The scorer gives 1 - rate, never below 0, and 0 when a test case has no expected output.

import { Evaluator, normalizeTranscript, wordErrorRate } from '@framers/agentos';

normalizeTranscript("Um, so the [laughter] meeting's at 3.");
// ['so', 'the', "meeting's", 'at', '3']

wordErrorRate('so the meeting is at three', "Um, so the [laughter] meeting's at 3.");
// { substitutions: 2, deletions: 0, insertions: 1, referenceWords: 5, rate: 0.6 }

const evaluator = new Evaluator();
await evaluator.score('word_error_rate', 'so the meeting is at three', "so the meeting's at 3");
// 0.4

Custom Scorers​

Register custom scorers for domain-specific evaluation:

// A scorer is (actual, expected, references, metadata) => number | Promise<number>
evaluator.registerScorer('json_valid', (actual) => {
  try {
    JSON.parse(actual);
    return 1;
  } catch {
    return 0;
  }
});

// Use in test cases
const testCase: EvalTestCase = {
  id: 'json-1',
  name: 'JSON Output',
  input: 'Generate JSON for user profile',
  expectedOutput: '{"name": "John", "age": 30}',
  criteria: [
    { name: 'valid_json', description: 'Output is valid JSON', weight: 1, scorer: 'json_valid' },
    { name: 'similar', description: 'Similar structure', weight: 2, scorer: 'levenshtein' }
  ]
};

Configuration Options​

interface EvalConfig {
  concurrency?: number;      // Cases per batch (default: 3); a batch waits for its slowest case
  timeoutMs?: number;        // Per attempt (default: 60000); the agent call is not cancelled
  retries?: number;          // Extra attempts after an error or a timeout (default: 1)
  continueOnError?: boolean; // Declared, not read: a failed case scores 0 and the run goes on
  thresholds?: {
    pass?: number;           // Minimum weighted score to pass (default: 0.7)
    warn?: number;           // Declared, not read (default: 0.5)
  };
  customScorers?: Record<string, ScorerFunction>; // Declared, not read: use registerScorer()
}

Test Case Structure​

interface EvalTestCase {
  id: string;                    // Unique identifier
  name: string;                  // Human-readable name
  category?: string;             // A label (the aggregates do not group by it)
  input: string;                 // Input to the agent function
  expectedOutput?: string;       // Reference answer
  referenceOutputs?: string[];   // Extra references (bleu, semantic_similarity)
  context?: string;              // Second argument of the agent function
  expectedToolCalls?: Array<{ toolName: string; args?: Record<string, unknown> }>; // Declared, not scored
  criteria?: EvalCriteria[];     // { name, description, weight, scorer, threshold? }; default: levenshtein
  metadata?: Record<string, unknown>; // Fourth argument of every scorer
}

A case's score is the weighted mean of its criteria; a criterion's threshold is not read, and a criterion naming an unknown scorer is skipped.

Comparing Runs​

Track improvements across versions:

const runV1 = await evaluator.runEvaluation('v1.0', testCases, agentV1);
const runV2 = await evaluator.runEvaluation('v2.0', testCases, agentV2);

const comparison = await evaluator.compareRuns(runV1.runId, runV2.runId);

console.log('Improvements:', comparison.summary.improved);
console.log('Regressions:', comparison.summary.regressed);

for (const metric of comparison.metrics) {
  console.log(`${metric.name}: ${metric.delta.toFixed(2)} (${metric.percentChange.toFixed(1)}%)`);
}

The comparison covers passRate, avgScore, avgLatencyMs and p95LatencyMs. A metric counts as improved when the second run's value is higher, for the latencies too, so a slower second run reports its latencies as improvements. Both runs must belong to the same Evaluator instance.

Aggregate Metrics​

Each evaluation run includes aggregate metrics:

interface AggregateMetrics {
  totalTests: number;
  passedTests: number;
  failedTests: number;
  passRate: number;        // 0-1
  avgScore: number;        // 0-1
  scoreStdDev: number;     // Standard deviation
  avgLatencyMs: number;
  p50LatencyMs: number;    // Median
  p95LatencyMs: number;    // 95th percentile
  p99LatencyMs: number;    // 99th percentile
  totalTokens: number;     // 0: runEvaluation() records no token usage
  totalCostUsd: number;    // 0: runEvaluation() records no cost
  byCategory?: Record<string, { passRate: number; avgScore: number; count: number }>; // Not filled
}

Integration with GMI​

Evaluate GMI responses through the runtime. Each test case runs on a session of its own, so one case's history does not reach the next:

import { AgentOS, AgentOSResponseChunkType, BUILT_IN_PERSONAS } from '@framers/agentos';

const agentos = await AgentOS.create({ personas: BUILT_IN_PERSONAS });
let caseNumber = 0;

// Create wrapper function for evaluation
async function gmiAgent(input: string): Promise<string> {
  let response = '';
  for await (const chunk of agentos.processRequest({
    userId: 'eval-user',
    sessionId: `eval-case-${++caseNumber}`,
    selectedPersonaId: 'v_researcher',
    textInput: input,
  })) {
    if (chunk.type === AgentOSResponseChunkType.TEXT_DELTA) {
      response += chunk.textDelta;
    }
  }
  return response;
}

const run = await evaluator.runEvaluation('GMI Evaluation', testCases, gmiAgent);

Best Practices​

  1. Start with representative test cases - Cover common user queries and edge cases
  2. Use multiple criteria - Combine exactness and semantic measures
  3. Weight criteria appropriately - Prioritize what matters most
  4. Track baselines - Store run IDs to compare against
  5. Automate in CI/CD - Run evaluations on each deployment
  6. Review failures - Read the failing cases' actualOutput and error

Report Formats​

import { writeFileSync } from 'node:fs';

// JSON: the whole EvalRun
const data = JSON.parse(await evaluator.generateReport(run.runId, 'json'));

// Markdown: summary, latency percentiles, one line per case
const markdown = await evaluator.generateReport(run.runId, 'markdown');

// HTML: the same content as tables
writeFileSync('report.html', await evaluator.generateReport(run.runId, 'html'));

LLM-as-Judge​

LLMJudge grades with a model through an AIModelProviderManager. Without modelId, it uses the judge resolver's model (OpenAI gpt-5.6, or AGENTOS_JUDGE_PROVIDER / AGENTOS_JUDGE_MODEL / AGENTOS_JUDGE_EFFORT):

import { LLMJudge, CRITERIA_PRESETS } from '@framers/agentos';

const judge = new LLMJudge({
  llmProvider: aiModelProviderManager,
  modelId: 'gpt-4-turbo',
  temperature: 0.1,
});

// Single judgment
const result = await judge.judge(
  'What is photosynthesis?',
  'Photosynthesis is how plants make food from sunlight.',
  'Photosynthesis is the process by which plants convert light energy into chemical energy.'
);

console.log(`Score: ${result.score}`);
console.log(`Reasoning: ${result.reasoning}`);
console.log(`Feedback: ${result.feedback.join(', ')}`);

// Use preset criteria
const codeResult = await judge.judge(
  'Write a function to reverse a string',
  actualCode,
  expectedCode,
  CRITERIA_PRESETS.codeGeneration
);

// Compare two outputs
const comparison = await judge.compare(
  'Summarize this article',
  summaryA,
  summaryB,
  CRITERIA_PRESETS.summarization
);
console.log(`Winner: ${comparison.winner}`);

// Register as a scorer; it reads the input from the test case's metadata.input
evaluator.registerScorer('llm_judge', judge.createScorer());
  • judge() asks for a JSON answer at temperature 0.1. A provider error or an answer that is not JSON scores errorScore (default 0); JSON without overallScore scores 0.5.
  • compare() judges the two outputs separately and calls a tie when their scores differ by less than 0.05.
  • batchJudge(items, criteria, concurrency = 3) returns the judgments in the order they finish, not in the order of items.

Available Criteria Presets​

PresetUse Case
codeGenerationEvaluate generated code
summarizationEvaluate summaries
questionAnsweringEvaluate Q&A responses
creativeWritingEvaluate creative content
safetyEvaluate for harmlessness

References​

LLM-as-judge methodology​

Human-LLM agreement + evaluator calibration​

Bias + reliability concerns​

Implementation references​