Skip to main content

Class: LexicalIndex

Defined in: packages/agentos/src/cognition/rag/search/LexicalIndex.ts:160

The BM25 engine over a caller's tokenizer. It imports nothing, so a page can load it; it ranks documents holding any query word (or only those holding every one), matches a stored word whole or by a query word it begins with, and is saved with LexicalIndex.toJSON and restored with LexicalIndex.fromJSON.

A document's score is the sum, over the query's words, of IDF(t) * (tf(t,D) * (k1 + 1)) / (tf(t,D) + k1 * (1 - b + b * |D| / avgdl)), with the Robertson-Walker IDF(t) = log((N - n(t) + 0.5) / (n(t) + 0.5) + 1).

Example​

const index = new LexicalIndex({ tokenize: lexicalTokens });
index.addDocument('s1#0', 'The chapters on the budget review');
index.search('chap budg', 10, { prefix: true, match: 'all' }); // the passage s1#0

const saved = JSON.stringify(index.toJSON());
const restored = LexicalIndex.fromJSON(JSON.parse(saved), { tokenize: lexicalTokens });

Extended by​

Constructors​

Constructor​

new LexicalIndex(config): LexicalIndex

Defined in: packages/agentos/src/cognition/rag/search/LexicalIndex.ts:179

Makes an empty index with the caller's tokenizer and, unless given, k1 1.2 and b 0.75.

Parameters​

config​

LexicalIndexConfig

Returns​

LexicalIndex

Properties​

avgDocLength​

protected avgDocLength: number

Defined in: packages/agentos/src/cognition/rag/search/LexicalIndex.ts:172

Average document length across the index (in tokens).

b​

protected b: number

Defined in: packages/agentos/src/cognition/rag/search/LexicalIndex.ts:164

Document length normalization (0 = none, 1 = full).

documents​

protected documents: Map<string, BM25Document>

Defined in: packages/agentos/src/cognition/rag/search/LexicalIndex.ts:166

Map of document ID to internal document representation.

idf​

protected idf: Map<string, number>

Defined in: packages/agentos/src/cognition/rag/search/LexicalIndex.ts:170

Each term's IDF, recomputed after documents are added or removed.

idfDirty​

protected idfDirty: boolean

Defined in: packages/agentos/src/cognition/rag/search/LexicalIndex.ts:174

Whether the IDF cache needs recomputation.

invertedIndex​

protected invertedIndex: Map<string, Map<string, number>>

Defined in: packages/agentos/src/cognition/rag/search/LexicalIndex.ts:168

Each term's documents with the term's count in each: term -> { docId -> termFrequency }.

k1​

protected k1: number

Defined in: packages/agentos/src/cognition/rag/search/LexicalIndex.ts:162

Term saturation parameter (typical range: 1.2-2.0).

tokenizeText()​

protected readonly tokenizeText: (text) => string[]

Defined in: packages/agentos/src/cognition/rag/search/LexicalIndex.ts:176

The tokenizer this index was made with.

Parameters​

text​

string

Returns​

string[]

Methods​

addDocument()​

addDocument(id, text, metadata?): void

Defined in: packages/agentos/src/cognition/rag/search/LexicalIndex.ts:210

Adds a document, replacing one of the same id.

Parameters​

id​

string

text​

string

metadata?​

Record<string, unknown>

Returns​

void

addDocuments()​

addDocuments(docs): void

Defined in: packages/agentos/src/cognition/rag/search/LexicalIndex.ts:230

Adds several documents.

Parameters​

docs​

object[]

Returns​

void

getStats()​

getStats(): BM25Stats

Defined in: packages/agentos/src/cognition/rag/search/LexicalIndex.ts:308

The index's size.

Returns​

BM25Stats

recomputeIdf()​

protected recomputeIdf(): void

Defined in: packages/agentos/src/cognition/rag/search/LexicalIndex.ts:191

Recomputes every term's IDF (Robertson-Walker) and the average document length.

Returns​

void

removeDocument()​

removeDocument(id): boolean

Defined in: packages/agentos/src/cognition/rag/search/LexicalIndex.ts:296

Removes a document; true when it was there.

Parameters​

id​

string

Returns​

boolean

restore()​

protected restore(json): void

Defined in: packages/agentos/src/cognition/rag/search/LexicalIndex.ts:329

Fills this index from a saved one, or throws an Error naming what keeps it from being one toJSON writes.

Parameters​

json​

LexicalIndexJSON

Returns​

void

search(query, topK?, options?): BM25Result[]

Defined in: packages/agentos/src/cognition/rag/search/LexicalIndex.ts:252

Searches. With no options it answers exactly what BM25Index has always answered.

Parameters​

query​

string

The query text, read with the index's tokenizer.

topK?​

number = 10

The most results answered.

options?​

LexicalSearchOptions = {}

match and prefix; see LexicalSearchOptions.

Returns​

BM25Result[]

The matching documents, highest score first.

Default​

10

toJSON()​

toJSON(): LexicalIndexJSON

Defined in: packages/agentos/src/cognition/rag/search/LexicalIndex.ts:314

The index as plain JSON.

Returns​

LexicalIndexJSON

fromJSON()​

static fromJSON(json, config): LexicalIndex

Defined in: packages/agentos/src/cognition/rag/search/LexicalIndex.ts:349

A saved index restored, with the tokenizer it was made with.

Parameters​

json​

LexicalIndexJSON

config​

Pick<LexicalIndexConfig, "tokenize">

Returns​

LexicalIndex

Throws​

When the saved index is of another version or is not one LexicalIndex.toJSON could have written; the message names what is wrong.