Skip to main content

Class: BM25Index

Defined in: packages/agentos/src/cognition/rag/search/BM25Index.ts:110

BM25 sparse keyword index for hybrid retrieval.

Dense embeddings excel at semantic similarity but miss exact keyword matches (e.g., error codes, function names, product IDs). BM25 catches these by scoring documents based on term frequency, inverse document frequency, and document length normalization.

It is a LexicalIndex with AgentOS's tokenizer: the configured tokenize or pipeline, else the built-in one, whose stop words are natural's English list when that package loads and ENGLISH_STOP_WORDS otherwise, read on the first tokenization, not when the module is imported. toJSON() saves it and BM25Index.fromJSON restores it.

Examples​

const index = new BM25Index({ k1: 1.5, b: 0.75 });

index.addDocuments([
{ id: 'doc-1', text: 'TypeScript compiler error TS2304' },
{ id: 'doc-2', text: 'JavaScript runtime TypeError explanation' },
{ id: 'doc-3', text: 'Fix error TS2304 by adding type declarations' },
]);

const results = index.search('error TS2304', 5);
// results[0].id === 'doc-3' (exact match on "error" + "TS2304")
// results[1].id === 'doc-1' (exact match on "error" + "TS2304")
const hybrid = new HybridSearcher(vectorStore, embeddingManager, bm25Index, {
denseWeight: 0.7,
sparseWeight: 0.3,
});
const results = await hybrid.search('What does error TS2304 mean?');

Extends​

Constructors​

Constructor​

new BM25Index(config?): BM25Index

Defined in: packages/agentos/src/cognition/rag/search/BM25Index.ts:125

Creates a new BM25 index.

Parameters​

config?​

BM25Config

Optional BM25 tuning parameters and tokenizer.

Returns​

BM25Index

Example​

// Use defaults (k1=1.2, b=0.75)
const index = new BM25Index();

// Custom parameters for short documents
const shortDocIndex = new BM25Index({ k1: 1.5, b: 0.5 });

Overrides​

LexicalIndex.constructor

Properties​

avgDocLength​

protected avgDocLength: number

Defined in: packages/agentos/src/cognition/rag/search/LexicalIndex.ts:172

Average document length across the index (in tokens).

Inherited from​

LexicalIndex.avgDocLength


b​

protected b: number

Defined in: packages/agentos/src/cognition/rag/search/LexicalIndex.ts:164

Document length normalization (0 = none, 1 = full).

Inherited from​

LexicalIndex.b


documents​

protected documents: Map<string, BM25Document>

Defined in: packages/agentos/src/cognition/rag/search/LexicalIndex.ts:166

Map of document ID to internal document representation.

Inherited from​

LexicalIndex.documents


idf​

protected idf: Map<string, number>

Defined in: packages/agentos/src/cognition/rag/search/LexicalIndex.ts:170

Each term's IDF, recomputed after documents are added or removed.

Inherited from​

LexicalIndex.idf


idfDirty​

protected idfDirty: boolean

Defined in: packages/agentos/src/cognition/rag/search/LexicalIndex.ts:174

Whether the IDF cache needs recomputation.

Inherited from​

LexicalIndex.idfDirty


invertedIndex​

protected invertedIndex: Map<string, Map<string, number>>

Defined in: packages/agentos/src/cognition/rag/search/LexicalIndex.ts:168

Each term's documents with the term's count in each: term -> { docId -> termFrequency }.

Inherited from​

LexicalIndex.invertedIndex


k1​

protected k1: number

Defined in: packages/agentos/src/cognition/rag/search/LexicalIndex.ts:162

Term saturation parameter (typical range: 1.2-2.0).

Inherited from​

LexicalIndex.k1


tokenizeText()​

protected readonly tokenizeText: (text) => string[]

Defined in: packages/agentos/src/cognition/rag/search/LexicalIndex.ts:176

The tokenizer this index was made with.

Parameters​

text​

string

Returns​

string[]

Inherited from​

LexicalIndex.tokenizeText

Methods​

addDocument()​

addDocument(id, text, metadata?): void

Defined in: packages/agentos/src/cognition/rag/search/LexicalIndex.ts:210

Adds a document, replacing one of the same id.

Parameters​

id​

string

text​

string

metadata?​

Record<string, unknown>

Returns​

void

Inherited from​

LexicalIndex.addDocument


addDocuments()​

addDocuments(docs): void

Defined in: packages/agentos/src/cognition/rag/search/LexicalIndex.ts:230

Adds several documents.

Parameters​

docs​

object[]

Returns​

void

Inherited from​

LexicalIndex.addDocuments


getStats()​

getStats(): BM25Stats

Defined in: packages/agentos/src/cognition/rag/search/LexicalIndex.ts:308

The index's size.

Returns​

BM25Stats

Inherited from​

LexicalIndex.getStats


recomputeIdf()​

protected recomputeIdf(): void

Defined in: packages/agentos/src/cognition/rag/search/LexicalIndex.ts:191

Recomputes every term's IDF (Robertson-Walker) and the average document length.

Returns​

void

Inherited from​

LexicalIndex.recomputeIdf


removeDocument()​

removeDocument(id): boolean

Defined in: packages/agentos/src/cognition/rag/search/LexicalIndex.ts:296

Removes a document; true when it was there.

Parameters​

id​

string

Returns​

boolean

Inherited from​

LexicalIndex.removeDocument


restore()​

protected restore(json): void

Defined in: packages/agentos/src/cognition/rag/search/LexicalIndex.ts:329

Fills this index from a saved one, or throws an Error naming what keeps it from being one toJSON writes.

Parameters​

json​

LexicalIndexJSON

Returns​

void

Inherited from​

LexicalIndex.restore


search(query, topK?, options?): BM25Result[]

Defined in: packages/agentos/src/cognition/rag/search/LexicalIndex.ts:252

Searches. With no options it answers exactly what BM25Index has always answered.

Parameters​

query​

string

The query text, read with the index's tokenizer.

topK?​

number = 10

The most results answered.

options?​

LexicalSearchOptions = {}

match and prefix; see LexicalSearchOptions.

Returns​

BM25Result[]

The matching documents, highest score first.

Default​

10

Inherited from​

LexicalIndex.search


toJSON()​

toJSON(): LexicalIndexJSON

Defined in: packages/agentos/src/cognition/rag/search/LexicalIndex.ts:314

The index as plain JSON.

Returns​

LexicalIndexJSON

Inherited from​

LexicalIndex.toJSON


fromJSON()​

static fromJSON(json, config?): BM25Index

Defined in: packages/agentos/src/cognition/rag/search/BM25Index.ts:138

A saved index restored; give the configuration it was made with. Its k1 and b come from the saved index, and its tokenizer from config, which must read text as the one the index was made with.

Parameters​

json​

LexicalIndexJSON

config?​

BM25Config

Returns​

BM25Index

Overrides​

LexicalIndex.fromJSON