Skip to main content

Vision Pipeline (OCR & Image Understanding)

VisionPipeline runs an image through up to five tiers: OCR (PaddleOCR or Tesseract.js), handwriting recognition (TrOCR), document layout (Florence-2), a cloud vision model called through generateText(), and an image embedding (CLIP). createVisionPipeline() builds a pipeline from the packages installed and the API keys in the environment.

Every tier runs against the current releases of the packages it names: ppu-paddle-ocr 6.6.1, tesseract.js 7.0.0 and @huggingface/transformers 3.8.1. Limitations lists what the pipeline does not do.

Quick Start​

import { createVisionPipeline } from '@framers/agentos';
import { readFileSync } from 'node:fs';

// Needs an OCR package (npm install tesseract.js) for 'progressive'.
const vision = await createVisionPipeline({ strategy: 'progressive' });

const result = await vision.process(readFileSync('./document.png'));
console.log(result.text);        // text of the tier with the highest confidence
console.log(result.confidence);  // that tier's confidence, 0-1
console.log(result.category);    // 'printed-text' | 'handwritten' | 'document-layout' | 'photograph' | 'mixed' | ...
console.log(result.tiers);       // tiers that ran, e.g. ['ocr', 'cloud-vision']
console.log(result.tierResults); // each tier's text, confidence and duration

await vision.dispose();

The other methods each run one tier:

const text = await vision.extractText(image);      // OCR tier, returns a string
const layout = await vision.analyzeLayout(image);  // Florence-2 tier, returns a DocumentLayout
const vector = await vision.embed(image);          // CLIP tier, returns number[]

// Cloud tier only: a description of the image plus the text it shows.
const described = await vision.process(image, { tiers: ['cloud-vision'] });

// Cloud tier only, asked for the text alone, exactly as written.
const transcribed = await vision.process(image, { tiers: ['cloud-vision'], cloudTask: 'transcribe' });

image is a Buffer, a file path, an http(s) URL or a data URL. Preprocessing applies to a Buffer only. A file path works for the local tiers only: the cloud tier sends a string to the provider as the image URL, and a provider cannot read a path on your machine. Pass a local file as a Buffer when the cloud tier can run. The local tiers fetch an http(s) URL string themselves, and nothing checks where it points: for a URL from a model or a user, read it with imageToBuffer(url, { untrusted: true }) (see Image Editing) and pass the Buffer.

Tiers​

Tier (VisionTier)What runsNeedsConfidence it reports
ocrPaddleOCR (ppu-paddle-ocr) or Tesseract.js (tesseract.js, English), per the ocr optionthe packagePaddleOCR: the mean of the region confidences. Tesseract.js: the page confidence divided by 100
handwritingTrOCR, Xenova/trocr-base-handwritten (Microsoft's checkpoint converted to ONNX), Transformers.js image-to-text task@huggingface/transformers and handwriting: true0.75 when it returns text, else 0
document-aiFlorence-2, onnx-community/Florence-2-base-ft, loaded as a model and a processor and run with the <OCR_WITH_REGION> task@huggingface/transformers and documentAI: true0.8 when it returns text, else 0
cloud-visiongenerateText() with the image and one of two fixed prompts: by default, describe the image, extract all visible text and name the kind of content; with cloudTask: 'transcribe', give the text alone, exactly as written, in reading ordercloudProvider0.95, fixed; 0 for an answer with no text
embeddingCLIP, Xenova/clip-vit-base-patch32, Transformers.js image-feature-extraction task: 512 numbers in the space of CLIP's text embeddings@huggingface/transformers and embedding: truenone; fills result.embedding

result.text and result.confidence come from the tier with the highest confidence, so a cloud result (0.95) wins over every local tier that ran. result.regions holds that tier's text regions. The Florence-2 tier reads the image line by line: result.layout holds one page the size of the image, with a text block for each line and the line's bounding box in the image's pixels, and the tier's text is the lines in reading order. It does not label headings, tables or figures.

A tier that was due to run and failed, such as an OCR engine that could not start or a model that did not load, is left out of result.tiers and listed in result.failedTiers with its error, and the run goes on with the other tiers: an OCR failure leaves the category mixed, so both model tiers run, and the cloud tier after them as the strategy allows. process() throws only when no tier gave a result, naming each failure. dispose() waits for calls in progress, and concurrent first calls share one load of each engine.

Each local model tier downloads its model from the Hugging Face Hub on first use and caches it: about 1.3 GB for TrOCR, 1.1 GB for Florence-2 and 350 MB for CLIP's vision tower, in fp32, the precision Transformers.js loads by default in Node. The Vision models CI job runs the three tiers from the built package against these models every week and on every change to the vision code.

The cloud tier sends a Buffer as a data URL whose media type comes from the image's first bytes (PNG, JPEG, GIF or WebP); a string goes to the provider as given.

Strategies​

Strategyocrhandwriting, document-aicloud-visionembedding
progressive (default)alwayswhen the OCR confidence is below the threshold and the category calls for the tierwhen the best local confidence is below the thresholdwhen enabled
local-onlyalwayswhen the category calls for the tierneverwhen enabled
cloud-onlyneverneveralwayswhen enabled
parallelalwayswhen the category calls for the tieralwayswhen enabled
  • The threshold is confidenceThreshold, default 0.7. In progressive, an OCR result at or above it ends the run.
  • The text tiers run one after another. Only the embedding tier runs alongside them. parallel runs the local-only sequence and then the cloud tier.
  • The cloud tier runs only when cloudProvider is set; cloud-only without one throws.
  • The handwriting tier runs for the categories handwritten and mixed; the document tier for document-layout and mixed.
  • A handwriting, document or embedding tier that fails is skipped. A cloud tier that fails is skipped when a local tier produced a result; otherwise process() throws, as it does when the OCR tier fails.

process(image, { tiers: [...] }) runs the listed tiers instead of the strategy's sequence, whatever the handwriting, documentAI and embedding flags say; the handwriting and document tiers still need a matching category.

Content Category​

The category comes from the OCR result, checked in this order:

ConditionCategory
no OCR resultmixed
confidence above 0.85printed-text
confidence below 0.5 and at least one single-character regionhandwritten
more than 20 regionsdocument-layout
confidence below 0.6 and fewer than 5 regionsphotograph
anything elsemixed

diagram and screenshot are never detected; they come only from forceCategory:

const result = await vision.process(image, { forceCategory: 'handwritten' });

forceCategory sets the reported category and routes the handwriting and document tiers. In progressive the OCR tier still runs first, and an OCR result at or above the threshold still ends the run. To run TrOCR alone, pass { tiers: ['handwriting'] }.

The cloud tier describes the image unless cloudTask: 'transcribe' is passed, and its answer has confidence 0.95, so it becomes result.text over every local tier that ran. An answer with no text in it, which the transcription prompt asks for when the image shows none, has confidence 0, so a local tier's text stands. A call that wants text passes 'transcribe'. TrOCR is a single-line model (its card): its processor resizes the whole image to 384 by 384 pixels and generation stops at 20 tokens, so a page of handwriting is read with the cloud tier listed too:

const page = await vision.process(scan, {
  tiers: ['handwriting', 'cloud-vision'],
  forceCategory: 'handwritten',
  cloudTask: 'transcribe',
});

createVisionPipeline() Options​

createVisionPipeline(config?) takes a partial VisionPipelineConfig. A field left out is filled in by detection: the optional packages installed, and the API keys in the environment. new VisionPipeline(config) detects nothing: its handwriting, documentAI and embedding default to off and its ocr to 'paddle'.

interface VisionPipelineConfig {
  /** How the tiers combine. Default: 'progressive'. */
  strategy: 'progressive' | 'local-only' | 'cloud-only' | 'parallel';

  /** OCR engine. Detected: 'paddle' when ppu-paddle-ocr is installed, else 'tesseract' when tesseract.js is, else 'none'. */
  ocr?: 'paddle' | 'tesseract' | 'none';

  /** TrOCR handwriting, Florence-2 document layout and CLIP embeddings. Detected: on when @huggingface/transformers is installed. */
  handwriting?: boolean;
  documentAI?: boolean;
  embedding?: boolean;

  /**
   * Cloud vision provider, a provider id generateText() knows ('openai',
   * 'anthropic', 'gemini', 'openrouter', ...). Detected: 'openai' when
   * OPENAI_API_KEY is set, else 'anthropic' (ANTHROPIC_API_KEY), else 'gemini'
   * (GEMINI_API_KEY, or GOOGLE_API_KEY, which detection passes as the key),
   * else 'openrouter' (OPENROUTER_API_KEY); unset and undetected, there is no
   * cloud tier. Gemini fetches an https image URL itself, up to 15 MB a
   * file, and takes a Buffer or a data URL inline, up to 100 MB of base64
   * per request.
   */
  cloudProvider?: string;
  /** Cloud model. Default: the provider's default text model. */
  cloudModel?: string;
  /**
   * Key for the cloud provider. Default: its environment variable
   * (OPENAI_API_KEY and so on). Refused, with cloudBaseUrl, when neither
   * cloudProvider nor a cloudModel prefix ('openai:gpt-4o') names the
   * provider, and when the two name different providers.
   */
  cloudApiKey?: string;
  /**
   * Base URL for the cloud provider, such as a proxy. Refused as cloudApiKey
   * is. An empty or whitespace-only value is unset: never refused, and the
   * provider's base-URL variable, then its default, apply as with none.
   */
  cloudBaseUrl?: string;
  /**
   * Providers the cloud tier may try, in order, after cloudProvider fails
   * (generateText's fallbackProviders). Default: none, so the image goes to
   * cloudProvider alone, whatever other keys the environment holds.
   */
  cloudFallbackProviders?: Array<{ provider: string; model?: string }>;

  /** Confidence below which 'progressive' runs the next tier. Default: 0.7. */
  confidenceThreshold?: number;

  /** Applied with sharp to a Buffer before any tier runs. Without sharp installed, the image passes unchanged. */
  preprocessing?: {
    grayscale?: boolean;
    resize?: { maxWidth?: number; maxHeight?: number };  // scales down only
    sharpen?: boolean;
    normalize?: boolean;
  };
}

VisionResult Shape​

interface VisionResult {
  text: string;                 // text of the winning tier (the cloud tier's: a description with the text, or with cloudTask 'transcribe' the text alone)
  confidence: number;           // confidence of the winning tier, 0-1
  category: ContentCategory;    // 'printed-text' | 'handwritten' | 'document-layout' | 'photograph' | 'diagram' | 'screenshot' | 'mixed'
  tiers: VisionTier[];          // tiers that ran: 'ocr' | 'handwriting' | 'document-ai' | 'embedding' | 'cloud-vision'
  tierResults: TierResult[];    // one entry per text tier that produced a result, in run order
  embedding?: number[];         // CLIP output, when the embedding tier ran
  layout?: DocumentLayout;      // { pages: [{ pageNumber, width, height, blocks }] }, when Florence-2 ran
  regions?: TextRegion[];       // text regions of the winning tier
  failedTiers?: FailedTier[];   // tiers that were due to run and failed, each { tier, error }; absent when none did
  durationMs: number;           // wall-clock time of process()
}

interface TierResult {
  tier: VisionTier;
  provider: string;             // 'paddle', 'tesseract', 'trocr', 'florence-2', or the cloud provider that answered
  model?: string;               // the cloud tier's model, as generateText reports it
  text: string;
  confidence: number;
  durationMs: number;
  regions?: TextRegion[];
}

interface FailedTier {
  tier: VisionTier;
  error: string;                // the error's message
}

interface TextRegion {
  text: string;
  confidence: number;
  bbox: { x: number; y: number; width: number; height: number };
}

Indexing Images for Retrieval​

The multimodal indexer describes an image with a vision provider, embeds the description with your text embedding manager and stores it with modality: 'image'; search() embeds a text query the same way. createMultimodalIndexerFromResolver() wraps a pipeline as that vision provider, so the indexed text is process(image).text.

import { createVisionPipeline } from '@framers/agentos';
import {
  createMultimodalIndexerFromResolver,
  type IEmbeddingManager,
  type IVectorStore,
} from '@framers/agentos/cognition/rag';
import { readFileSync } from 'node:fs';

declare const embeddingManager: IEmbeddingManager; // your text embedding manager
declare const vectorStore: IVectorStore;           // your initialized vector store

const visionPipeline = await createVisionPipeline({ strategy: 'cloud-only', cloudProvider: 'openai' });
const indexer = createMultimodalIndexerFromResolver({ visionPipeline, embeddingManager, vectorStore });

await indexer.indexImage({ image: readFileSync('./receipt.jpg'), metadata: { source: 'upload' } });
const hits = await indexer.search('receipt total');

Pass the pipeline through this factory (or as visionProvider: new PipelineVisionProvider(pipeline)), not as new MultimodalIndexer({ visionPipeline }): that constructor path fails in the published ES module build. The indexer sends a Buffer to the pipeline as a data URL labelled with the type its first bytes show (PNG, JPEG, WebP or GIF; anything else as PNG), so the pipeline skips preprocessing.

agent() accepts a rag field (including rag.multimodal.images) and does not read it.

See Multimodal RAG for the indexing design.

Installation​

npm install tesseract.js     # OCR tier, or ppu-paddle-ocr (with onnxruntime-node), which detection prefers
npm install sharp            # only for preprocessing

@huggingface/transformers is an optional dependency of @framers/agentos and installs with it unless optional dependencies are skipped. Transformers.js downloads model weights from the Hugging Face Hub the first time a tier loads them.

The cloud tier reads the provider's key from its environment variable: OPENAI_API_KEY, ANTHROPIC_API_KEY, GEMINI_API_KEY (with cloudProvider: 'gemini') or OPENROUTER_API_KEY, unless cloudApiKey is set.

The image goes to cloudProvider, and after it fails only to the providers cloudFallbackProviders lists, in order: no other vendor whose key is in the environment receives it. The tier fails (or, beside a local result, is listed in failedTiers) when cloudProvider and every listed provider fail. A listed provider uses its own key from the environment, and the tier's result names the provider and model that answered. Under the default cloudTask: 'describe', an empty answer is a failed tier: every image has a description. Under 'transcribe', an empty answer means the image shows no text, and it has confidence 0.

Limitations​

  • No OCR package. createVisionPipeline() then sets ocr: 'none': process() skips the OCR tier and goes on to the model and cloud tiers, while extractText() and a process(image, { tiers }) call whose list names only 'ocr' throw OCR is set to "none" but OCR tier was requested.
  • Layout labels. The Florence-2 tier gives every line as a text block; it does not label headings, tables, figures, lists or code.
  • CLIP text. The pipeline embeds images only; it has no method that embeds text into the CLIP space.