Multimodal RAG (Image + Audio)

Memory benchmarks (full N=500, gpt-4o and gpt-5-mini readers routed per question type): 85.6% on LongMemEval-S at $0.0090 per correct; with gpt-4o reading every question, the same retrieval scores 84.2%, level with Mastra Observational Memory (84.23%). 70.2% on LongMemEval-M on the 1.5M-token / 500-session haystack variant — the only open-source library on the public record above 65% on M with publicly reproducible methodology. The same text-first retrieval pipeline that produced these numbers is what the multimodal pattern below indexes against (derived captions, transcripts, OCR, document text) once you have a text representation. Benchmarks · Run JSONs · SOTA writeup

AgentOS’ core RAG APIs are text-first (EmbeddingManager + VectorStoreManager + RetrievalAugmentor). Multimodal support (image/audio) is implemented as a composable pattern on top:

1. Store the binary asset (optional) + metadata.
2. Derive a text representation (caption/transcript/OCR/document text extraction).
3. Index that text as a normal RAG document so the existing retrieval pipeline (vector, BM25, reranking, GraphRAG, etc.) can operate without any “special” multimodal database.
4. Optionally add modality-specific embeddings (image-to-image / audio-to-audio) as a fast path.

AgentOS supplies MultimodalIndexer, MultimodalAggregator and MultimodalMemoryBridge. The HTTP routes below come from @framers/agentos-ext-http-api; the asset table, the derivation steps and the environment variables on this page belong to the reference host backend that implements the router's ragService, and AgentOS reads none of those variables.

Retrieval runs on derived text: AgentOS has no visual late-interaction retriever and no page-native document retrieval. Document ingestion indexes extracted text into standard RAG collections through MultimodalIndexer.indexText(...), so derived document text is retrievable through the normal text pipeline rather than only being stored as memory traces.

⊕ Click to expand

Why This Design

• Works by default: If you can derive text, you can retrieve multimodal assets immediately using the standard RAG pipeline.
• Optional offline: Image/audio embedding retrieval is install-on-demand and can be enabled per deployment.
• No vendor lock-in: The same abstractions work with SqlVectorStore, HnswlibVectorStore, or QdrantVectorStore.

Architecture

Ingest (image/audio/document)

Key idea: the derived text is the canonical retrieval surface. Modality embeddings (when enabled) are an acceleration path, not a requirement. Documents are assets in the same model and are retrieved through their text.

UnifiedRetriever runs its multimodal source only for the plan's non-text modalities, so document and PDF text is retrieved through the standard text RAG collections, not through that source.

Query

Query-by-image and query-by-audio support a unified retrievalMode contract:

• auto (default): text-first retrieval, with native modality retrieval added when available
• text: derived text only
• native: modality-native embeddings only
• hybrid: fuse text + native retrieval when both are available

Text queries over /multimodal/query can search any combination of image, audio, and document assets.

Data Model (Reference Backend)

The reference backend stores multimodal asset metadata in a dedicated SQL table (name depends on the configured RAG table prefix):

• media_assets.asset_id (string) is the stable identifier.
• media_assets.modality is image, audio, or document.
• media_assets.collection_id is the “base” collection that the derived text is indexed into.
• store_payload controls whether raw bytes are persisted.
• metadata_json, tags_json, source_url, mime_type, original_file_name are stored for filtering/display.

The derived text representation is indexed as a normal RAG document:

• documentId = assetId
• collectionId = media_images, media_audio, or media_documents by default (configurable)
• chunks are generated from textRepresentation (usually 1 chunk unless you provide long text)

When offline embeddings are enabled, the reference backend also writes into embedding collections derived from the base collection:

• image embeddings: ${baseCollectionId}${suffix} (default suffix _img)
• audio embeddings: ${baseCollectionId}${suffix} (default suffix _aud)

This keeps modality embeddings separate from text embeddings, while still reusing the same vector-store provider.

HTTP API Surface

The host-agnostic Express router lives in @framers/agentos-ext-http-api — specifically src/rag/rag.routes.ts:

import express from 'express';
import { createAgentOSRagRouter } from '@framers/agentos-ext-http-api';

app.use(
  '/api/agentos/rag',
  createAgentOSRagRouter({
    isEnabled: () => true,
    ragService, // host-provided implementation
  }),
);

It mounts multimodal routes under /multimodal/*:

• POST /multimodal/images/ingest (multipart field: image)
• POST /multimodal/audio/ingest (multipart field: audio)
• POST /multimodal/documents/ingest (multipart field: document)
• POST /multimodal/query (search derived text)
• POST /multimodal/images/query (query-by-image)
• POST /multimodal/audio/query (query-by-audio)
• GET /multimodal/assets/:assetId
• GET /multimodal/assets/:assetId/content (only if payload is stored)
• DELETE /multimodal/assets/:assetId

See the @framers/agentos-ext-http-api package for request/response examples and deployment notes — the reference backend mounts the routes wired here (createAgentOSRagRouter).

Offline Embeddings (Optional)

Offline embeddings are disabled by default and are install-on-demand:

• Image embeddings: require Transformers.js (@huggingface/transformers preferred; @xenova/transformers supported).
• Audio embeddings: requires Transformers.js and WAV decoding support via wavefile (Node-only in the reference backend).

When offline embeddings are not enabled (or deps are missing), the system falls back to:

• query-by-image: caption the query image, then run text retrieval
• query-by-audio: transcribe the query audio, then run text retrieval
• document ingest: parse PDF/DOCX/TXT/MD/CSV/JSON/XML into derived text, then run normal text retrieval

Both query endpoints accept:

• textRepresentation to bypass captioning/transcription
• retrievalMode=auto|text|native|hybrid to control the planner

auto keeps derived text as the canonical retrieval layer and only adds native retrieval opportunistically.

Additional compatibility notes:

• Multipart query fields such as modalities and collectionIds may be sent as comma-separated strings (image,audio, docs,media_images) by higher-level clients.
• Document assets can be searched through /multimodal/query with modalities:["document"] or mixed alongside image/audio assets.
• Document parsing in the reference backend supports PDF, DOCX, TXT, Markdown, CSV, JSON, and XML.
• PDFs that contain no embedded text need a page-image OCR/vision pipeline; the reference backend surfaces that as an explicit extraction error instead of silently indexing nothing.
• Ollama can be used for image captioning when the selected model supports vision input and the caller sends image bytes as an inline data: URL; the Ollama provider sends only inline data: images and drops remote image URLs.
• Audio embedding retrieval is WAV-only in the Node reference backend. Non-WAV audio is retrieved via the transcript-first path.

Configuration (Reference Backend)

These env vars control the multimodal behavior of the reference backend; AgentOS does not read them:

• AGENTOS_RAG_MEDIA_STORE_PAYLOAD=true|false (default false)
• AGENTOS_RAG_MEDIA_IMAGE_COLLECTION_ID (default media_images)
• AGENTOS_RAG_MEDIA_AUDIO_COLLECTION_ID (default media_audio)
• AGENTOS_RAG_MEDIA_DOCUMENT_COLLECTION_ID (default media_documents)
• AGENTOS_RAG_MEDIA_IMAGE_EMBEDDINGS_ENABLED=true|false (default false)
• AGENTOS_RAG_MEDIA_IMAGE_EMBED_MODEL (default Xenova/clip-vit-base-patch32)
• AGENTOS_RAG_MEDIA_IMAGE_EMBED_COLLECTION_SUFFIX (default _img)
• AGENTOS_RAG_MEDIA_AUDIO_EMBEDDINGS_ENABLED=true|false (default false)
• AGENTOS_RAG_MEDIA_AUDIO_EMBED_MODEL (default Xenova/clap-htsat-unfused)
• AGENTOS_RAG_MEDIA_AUDIO_EMBED_COLLECTION_SUFFIX (default _aud)
• AGENTOS_RAG_MEDIA_*_EMBED_CACHE_DIR (optional; recommended for servers to persist model downloads)

Extending To Video

The recommended approach is the same pattern:

1. Persist video metadata and optional bytes.
2. Derive one or more text representations (e.g. transcript, scene captions, frame OCR).
3. Index derived text into a media_videos collection.
4. (Optional) add a video embedding collection for query-by-video.

This keeps the base retrieval system consistent while still allowing richer modality-specific paths.

Source Files

Symbol	Repo	Path
MultimodalIndexer	framerslab/agentos	src/cognition/rag/multimodal/MultimodalIndexer.ts
MultimodalAggregator	framerslab/agentos	src/cognition/memory/io/ingestion/MultimodalAggregator.ts
UnifiedRetriever	framerslab/agentos	src/cognition/rag/unified/UnifiedRetriever.ts
EmbeddingManager	framerslab/agentos	src/cognition/rag/EmbeddingManager.ts
VectorStoreManager	framerslab/agentos	src/cognition/rag/VectorStoreManager.ts
RetrievalAugmentor	framerslab/agentos	src/cognition/rag/RetrievalAugmentor.ts
SqlVectorStore	framerslab/agentos	src/cognition/rag/vector_stores/SqlVectorStore.ts
HnswlibVectorStore	framerslab/agentos	src/cognition/rag/vector_stores/HnswlibVectorStore.ts
QdrantVectorStore	framerslab/agentos	src/cognition/rag/vector_stores/QdrantVectorStore.ts
Vector stores tree	framerslab/agentos	src/cognition/rag/vector_stores/
Multimodal tree (Aggregator + Indexer + types)	framerslab/agentos	src/cognition/rag/multimodal/
createAgentOSRagRouter	@framers/agentos-ext-http-api (npm)	src/rag/rag.routes.ts
HTTP API package	@framers/agentos-ext-http-api (npm)	(root)

---

References

Retrieval-augmented generation foundations

• Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W.-t., Rocktäschel, T., Riedel, S., & Kiela, D. (2020). Retrieval-augmented generation for knowledge-intensive NLP tasks. NeurIPS 2020. — Original RAG paper.
• Karpukhin, V., Oguz, B., Min, S., Lewis, P., Wu, L., Edunov, S., Chen, D., & Yih, W.-t. (2020). Dense passage retrieval for open-domain question answering. EMNLP 2020. — Bi-encoder dense retrieval (the cosine-similarity layer in this pipeline).

Hybrid retrieval (dense + sparse + reranker)

• Robertson, S., & Zaragoza, H. (2009). The probabilistic relevance framework: BM25 and beyond. Foundations and Trends in Information Retrieval, 3(4), 333–389. — BM25 reference (the sparse arm of hybrid retrieval).
• Cormack, G. V., Clarke, C. L. A., & Buettcher, S. (2009). Reciprocal rank fusion outperforms Condorcet and individual rank learning methods. SIGIR 2009. — RRF for fusing dense + sparse rankings.
• Nogueira, R., & Cho, K. (2019). Passage re-ranking with BERT. arXiv preprint. — Cross-encoder reranking principle behind the Cohere / Transformers.js rerank stage.

Hypothetical document expansion

• Gao, L., Ma, X., Lin, J., & Callan, J. (2022). Precise zero-shot dense retrieval without relevance labels. arXiv preprint. — HyDE retrieval, the foundation of the hypothetical-document expansion path.
• Lei, F., et al. (2025). Never come up empty: Adaptive HyDE retrieval for improving LLM developer support. arXiv preprint. — Adaptive HyDE thresholding on a 3M-post Stack Overflow corpus; informs the adaptive thresholding in MemoryHydeRetriever.

Graph-augmented retrieval

• Edge, D., Trinh, H., Cheng, N., Bradley, J., Chao, A., Mody, A., Truitt, S., & Larson, J. (2024). From local to global: A graph RAG approach to query-focused summarization. arXiv preprint. — Microsoft GraphRAG; community detection + summarization for multi-hop reasoning.
• Blondel, V. D., Guillaume, J.-L., Lambiotte, R., & Lefebvre, E. (2008). Fast unfolding of communities in large networks. Journal of Statistical Mechanics: Theory and Experiment, 10, P10008. — Louvain algorithm used by GraphRAGEngine for community detection.

Multimodal embeddings

• Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., & Sutskever, I. (2021). Learning transferable visual models from natural language supervision. ICML 2021. — CLIP, the foundation for image-text joint embeddings used in vision retrieval.
• Wu, Y., Chen, K., Zhang, T., Hui, Y., Berg-Kirkpatrick, T., & Dubnov, S. (2023). Large-scale contrastive language-audio pretraining with feature fusion and keyword-to-caption augmentation. ICASSP 2023. — CLAP audio-text embeddings — referenced as Xenova/clap-htsat-unfused for the audio retrieval path.

Vector indexing

• Malkov, Y. A., & Yashunin, D. A. (2020). Efficient and robust approximate nearest neighbor search using hierarchical navigable small world graphs. IEEE Transactions on Pattern Analysis and Machine Intelligence, 42(4), 824–836. — HNSW algorithm behind the HnswlibVectorStore backend.

Implementation references

• src/cognition/rag/ — vector stores, embeddings, fusion, reranking, GraphRAG
• src/cognition/memory/retrieval/hyde/MemoryHydeRetriever.ts — HyDE for memory-specific recall
• src/cognition/memory/retrieval/graph/graphrag/GraphRAGEngine.ts — Microsoft GraphRAG-style implementation