Class: BM25Index
Defined in: packages/agentos/src/cognition/rag/search/BM25Index.ts:110
BM25 sparse keyword index for hybrid retrieval.
Dense embeddings excel at semantic similarity but miss exact keyword matches (e.g., error codes, function names, product IDs). BM25 catches these by scoring documents based on term frequency, inverse document frequency, and document length normalization.
It is a LexicalIndex with AgentOS's tokenizer: the configured
tokenize or pipeline, else the built-in one, whose stop words are
natural's English list when that package loads and ENGLISH_STOP_WORDS
otherwise, read on the first tokenization, not when the module is imported.
toJSON() saves it and BM25Index.fromJSON restores it.
Examples
const index = new BM25Index({ k1: 1.5, b: 0.75 });
index.addDocuments([
{ id: 'doc-1', text: 'TypeScript compiler error TS2304' },
{ id: 'doc-2', text: 'JavaScript runtime TypeError explanation' },
{ id: 'doc-3', text: 'Fix error TS2304 by adding type declarations' },
]);
const results = index.search('error TS2304', 5);
// results[0].id === 'doc-3' (exact match on "error" + "TS2304")
// results[1].id === 'doc-1' (exact match on "error" + "TS2304")
const hybrid = new HybridSearcher(vectorStore, embeddingManager, bm25Index, {
denseWeight: 0.7,
sparseWeight: 0.3,
});
const results = await hybrid.search('What does error TS2304 mean?');
Extends
Constructors
Constructor
new BM25Index(
config?):BM25Index
Defined in: packages/agentos/src/cognition/rag/search/BM25Index.ts:125
Creates a new BM25 index.
Parameters
config?
Optional BM25 tuning parameters and tokenizer.
Returns
BM25Index
Example
// Use defaults (k1=1.2, b=0.75)
const index = new BM25Index();
// Custom parameters for short documents
const shortDocIndex = new BM25Index({ k1: 1.5, b: 0.5 });
Overrides
Properties
avgDocLength
protectedavgDocLength:number
Defined in: packages/agentos/src/cognition/rag/search/LexicalIndex.ts:172
Average document length across the index (in tokens).
Inherited from
b
protectedb:number
Defined in: packages/agentos/src/cognition/rag/search/LexicalIndex.ts:164
Document length normalization (0 = none, 1 = full).
Inherited from
documents
protecteddocuments:Map<string,BM25Document>
Defined in: packages/agentos/src/cognition/rag/search/LexicalIndex.ts:166
Map of document ID to internal document representation.
Inherited from
idf
protectedidf:Map<string,number>
Defined in: packages/agentos/src/cognition/rag/search/LexicalIndex.ts:170
Each term's IDF, recomputed after documents are added or removed.
Inherited from
idfDirty
protectedidfDirty:boolean
Defined in: packages/agentos/src/cognition/rag/search/LexicalIndex.ts:174
Whether the IDF cache needs recomputation.
Inherited from
invertedIndex
protectedinvertedIndex:Map<string,Map<string,number>>
Defined in: packages/agentos/src/cognition/rag/search/LexicalIndex.ts:168
Each term's documents with the term's count in each: term -> { docId -> termFrequency }.
Inherited from
k1
protectedk1:number
Defined in: packages/agentos/src/cognition/rag/search/LexicalIndex.ts:162
Term saturation parameter (typical range: 1.2-2.0).
Inherited from
tokenizeText()
protectedreadonlytokenizeText: (text) =>string[]
Defined in: packages/agentos/src/cognition/rag/search/LexicalIndex.ts:176
The tokenizer this index was made with.
Parameters
text
string
Returns
string[]
Inherited from
Methods
addDocument()
addDocument(
id,text,metadata?):void
Defined in: packages/agentos/src/cognition/rag/search/LexicalIndex.ts:210
Adds a document, replacing one of the same id.
Parameters
id
string
text
string
metadata?
Record<string, unknown>
Returns
void
Inherited from
addDocuments()
addDocuments(
docs):void
Defined in: packages/agentos/src/cognition/rag/search/LexicalIndex.ts:230
Adds several documents.
Parameters
docs
object[]
Returns
void
Inherited from
getStats()
getStats():
BM25Stats
Defined in: packages/agentos/src/cognition/rag/search/LexicalIndex.ts:308
The index's size.
Returns
Inherited from
recomputeIdf()
protectedrecomputeIdf():void
Defined in: packages/agentos/src/cognition/rag/search/LexicalIndex.ts:191
Recomputes every term's IDF (Robertson-Walker) and the average document length.
Returns
void
Inherited from
removeDocument()
removeDocument(
id):boolean
Defined in: packages/agentos/src/cognition/rag/search/LexicalIndex.ts:296
Removes a document; true when it was there.
Parameters
id
string
Returns
boolean
Inherited from
restore()
protectedrestore(json):void
Defined in: packages/agentos/src/cognition/rag/search/LexicalIndex.ts:329
Fills this index from a saved one, or throws an Error naming what keeps it from being one toJSON writes.
Parameters
json
Returns
void
Inherited from
search()
search(
query,topK?,options?):BM25Result[]
Defined in: packages/agentos/src/cognition/rag/search/LexicalIndex.ts:252
Searches. With no options it answers exactly what BM25Index has always answered.
Parameters
query
string
The query text, read with the index's tokenizer.
topK?
number = 10
The most results answered.
options?
LexicalSearchOptions = {}
match and prefix; see LexicalSearchOptions.
Returns
The matching documents, highest score first.
Default
10
Inherited from
toJSON()
toJSON():
LexicalIndexJSON
Defined in: packages/agentos/src/cognition/rag/search/LexicalIndex.ts:314
The index as plain JSON.
Returns
Inherited from
fromJSON()
staticfromJSON(json,config?):BM25Index
Defined in: packages/agentos/src/cognition/rag/search/BM25Index.ts:138
A saved index restored; give the configuration it was made with. Its k1 and b come from the saved index, and
its tokenizer from config, which must read text as the one the index was made with.
Parameters
json
config?
Returns
BM25Index