Skip to main content

Structured Output

Overview​

The Structured Output Manager ensures LLM outputs conform to predefined JSON Schemas, enabling reliable parsing, validation, and type-safe consumption of agent responses.

Key Features​

  • JSON Schema Validation: Full JSON Schema draft 2020-12 support
  • Multiple Strategies: JSON mode, function calling, prompt engineering
  • Parallel Function Calls: Execute multiple tools in a single response
  • Entity Extraction: Pull structured data from unstructured text
  • Automatic Retry: Retry with feedback on validation failures
  • Robust Parsing: Handle malformed JSON from LLMs

Quick Start​

Basic Structured Generation​

import { StructuredOutputManager, JSONSchema } from '@framers/agentos/core/structured';

const manager = new StructuredOutputManager({
  llmProviderManager,
  defaultProviderId: 'openai',
  defaultModelId: 'gpt-4o',
});

// Define your schema
const personSchema: JSONSchema = {
  type: 'object',
  properties: {
    name: { type: 'string', minLength: 1 },
    age: { type: 'integer', minimum: 0, maximum: 150 },
    email: { type: 'string', format: 'email' },
    interests: {
      type: 'array',
      items: { type: 'string' },
      minItems: 1,
    },
  },
  required: ['name', 'email'],
};

// Generate structured output
const result = await manager.generate({
  prompt: 'Extract person info from: John Doe, 30 years old, john@example.com, likes hiking and photography',
  schema: personSchema,
  schemaName: 'Person',
});

if (result.success) {
  console.log(result.data);
  // { name: 'John Doe', age: 30, email: 'john@example.com', interests: ['hiking', 'photography'] }
}

Parallel Function Calling​

const result = await manager.generateFunctionCalls({
  prompt: 'Get weather for New York and current stock price of AAPL',
  functions: [
    {
      name: 'get_weather',
      description: 'Get current weather for a city',
      parameters: {
        type: 'object',
        properties: {
          city: { type: 'string' },
          units: { type: 'string', enum: ['celsius', 'fahrenheit'] },
        },
        required: ['city'],
      },
      handler: async (args) => await weatherAPI.get(args.city, args.units),
    },
    {
      name: 'get_stock_price',
      description: 'Get current stock price',
      parameters: {
        type: 'object',
        properties: {
          symbol: { type: 'string', pattern: '^[A-Z]{1,5}$' },
        },
        required: ['symbol'],
      },
      handler: async (args) => await stockAPI.getPrice(args.symbol),
    },
  ],
  maxParallelCalls: 10,
});

// Both functions called in parallel
result.calls.forEach(call => {
  console.log(`${call.functionName}:`, call.executionResult);
});

Entity Extraction​

const result = await manager.extractEntities({
  text: `
    Meeting attendees:
    - John Smith (john@company.com) - Engineering Lead
    - Sarah Johnson (sarah@company.com) - Product Manager
    - Mike Wilson (mike@company.com) - Designer
  `,
  entitySchema: {
    type: 'object',
    properties: {
      name: { type: 'string' },
      email: { type: 'string', format: 'email' },
      role: { type: 'string' },
    },
    required: ['name', 'email'],
  },
  taskName: 'MeetingAttendeeExtraction',
  extractAll: true,
});

console.log(result.entities);
// [
//   { name: 'John Smith', email: 'john@company.com', role: 'Engineering Lead' },
//   { name: 'Sarah Johnson', email: 'sarah@company.com', role: 'Product Manager' },
//   { name: 'Mike Wilson', email: 'mike@company.com', role: 'Designer' },
// ]

Generation Strategies​

JSON Mode (json_mode)​

Uses the provider's native JSON mode (OpenAI, OpenRouter, Ollama). Best for simple schemas.

const result = await manager.generate({
  prompt: 'List 3 colors',
  schema: { type: 'array', items: { type: 'string' }, minItems: 3, maxItems: 3 },
  schemaName: 'Colors',
  strategy: 'json_mode',
});

Function Calling (function_calling)​

Uses tool/function calling API. Best for complex nested schemas.

const result = await manager.generate({
  prompt: 'Generate a complex report',
  schema: complexReportSchema,
  schemaName: 'Report',
  strategy: 'function_calling',
});

Prompt Engineering (prompt_engineering)​

Instructs in the prompt and parses output. Fallback for providers without native support.

const result = await manager.generate({
  prompt: 'Generate data',
  schema: mySchema,
  schemaName: 'Data',
  strategy: 'prompt_engineering',
});

Auto Selection (auto)​

Automatically selects the best strategy based on provider capabilities and schema complexity.

const result = await manager.generate({
  prompt: 'Generate data',
  schema: mySchema,
  schemaName: 'Data',
  strategy: 'auto', // Default
});

JSON Schema Support​

Supported Keywords​

CategoryKeywords
Typetype, enum, const
StringminLength, maxLength, pattern, format
Numberminimum, maximum, exclusiveMinimum, exclusiveMaximum, multipleOf
Arrayitems, minItems, maxItems, uniqueItems
Objectproperties, required, additionalProperties, minProperties, maxProperties
CompositionallOf, anyOf, oneOf, not
References$ref, $defs

Format Validators​

FormatDescription
emailEmail address
uriFull URI
uri-referenceURI or relative reference
uuidUUID v4
date-timeISO 8601 datetime
dateISO 8601 date
timeISO 8601 time
hostnameDNS hostname
ipv4IPv4 address
ipv6IPv6 address
regexValid regex pattern

Validation​

Manual Validation​

const issues = manager.validate(
  { name: 'John', age: -5 },
  personSchema,
  true // strict mode
);

if (issues.length > 0) {
  issues.forEach(issue => {
    console.log(`${issue.path}: ${issue.message}`);
    // "age: Value must be >= 0"
  });
}

Custom Validators​

const result = await manager.generate({
  prompt: 'Generate user data',
  schema: userSchema,
  schemaName: 'User',
  customValidator: (data) => {
    const issues = [];
    
    // Business logic validation
    if (data.endDate < data.startDate) {
      issues.push({
        path: 'endDate',
        message: 'End date must be after start date',
        keyword: 'custom',
        severity: 'error',
      });
    }
    
    return issues;
  },
});

Retry Logic​

The manager automatically retries on validation failure:

const result = await manager.generate({
  prompt: 'Generate data',
  schema: strictSchema,
  schemaName: 'Data',
  maxRetries: 5, // Default: 3
});

console.log(`Succeeded after ${result.retryCount} retries`);

Schema Registration​

Register schemas for reuse:

// Register common schemas
manager.registerSchema('Address', {
  type: 'object',
  properties: {
    street: { type: 'string' },
    city: { type: 'string' },
    country: { type: 'string' },
    postalCode: { type: 'string' },
  },
  required: ['street', 'city', 'country'],
});

// Use in other schemas via $ref
const orderSchema = {
  type: 'object',
  properties: {
    orderId: { type: 'string' },
    shippingAddress: { $ref: '#/$defs/Address' },
    billingAddress: { $ref: '#/$defs/Address' },
  },
  required: ['orderId', 'shippingAddress'],
  $defs: {
    Address: manager.getSchema('Address'),
  },
};

Statistics​

Track structured output performance:

const stats = manager.getStatistics();

console.log(`Success rate: ${(stats.successRate * 100).toFixed(1)}%`);
console.log(`Average retries: ${stats.avgRetries.toFixed(2)}`);
console.log(`Average latency: ${stats.avgLatencyMs.toFixed(0)}ms`);
console.log(`Total tokens: ${stats.totalTokensUsed}`);
console.log('Top validation errors:', stats.topValidationErrors);

// Reset if needed
manager.resetStatistics();

Error Handling​

import { StructuredOutputError } from '@framers/agentos/core/structured';

try {
  const result = await manager.generate({
    prompt: 'Generate data',
    schema: strictSchema,
    schemaName: 'Data',
    maxRetries: 3,
  });
} catch (error) {
  if (error instanceof StructuredOutputError) {
    console.log('Validation failed after retries:', error.validationErrors);
    console.log('Raw output was:', error.rawOutput);
    console.log('Strategy used:', error.strategy);
    console.log('Retry count:', error.retryCount);
  }
}

Provider Capabilities​

ProviderJSON ModeFunction CallingParallel CallsStrict Mode
OpenAI✅✅✅✅
Anthropic❌✅✅❌
OpenRouter✅✅✅❌
Ollama✅❌❌❌

Best Practices​

1. Use Descriptive Schemas​

// ✅ Good: Rich descriptions help the LLM
const schema = {
  type: 'object',
  description: 'A product review with sentiment analysis',
  properties: {
    summary: {
      type: 'string',
      description: 'One sentence summary of the review',
      maxLength: 200,
    },
    sentiment: {
      type: 'string',
      enum: ['positive', 'neutral', 'negative'],
      description: 'Overall sentiment of the review',
    },
    score: {
      type: 'integer',
      minimum: 1,
      maximum: 5,
      description: 'Rating from 1 (worst) to 5 (best)',
    },
  },
};

// ❌ Bad: Minimal schema
const schema = {
  type: 'object',
  properties: {
    summary: { type: 'string' },
    sentiment: { type: 'string' },
    score: { type: 'number' },
  },
};

2. Start Simple​

// Start with simple schemas, add constraints as needed
const v1Schema = {
  type: 'object',
  properties: {
    name: { type: 'string' },
    age: { type: 'number' },
  },
};

// Later, add constraints based on real-world issues
const v2Schema = {
  type: 'object',
  properties: {
    name: { type: 'string', minLength: 1, maxLength: 100 },
    age: { type: 'integer', minimum: 0, maximum: 150 },
  },
  required: ['name'],
};

3. Use Appropriate Retries​

// Simple extraction: fewer retries needed
const result1 = await manager.generate({
  schema: simpleSchema,
  maxRetries: 2,
});

// Complex generation: may need more retries
const result2 = await manager.generate({
  schema: complexSchema,
  maxRetries: 5,
});