Skip to main content

Class: FuzzyIndex

Defined in: packages/agentos/src/cognition/rag/search/FuzzyIndex.ts:491

A ranked search over a caller's documents. Each document is a set of named fields, a text or a list of short texts each, and each field has the weight the index was made with. A query is read in NFC and cut into words by the tokenizer, and each word, folded to lower case with its accents dropped, matches a word of a document that is:

  • the same word, which counts 1;
  • a word it begins, which counts 0.75;
  • one edit away (a letter added, dropped or changed, or two letters side by side swapped: withinOneEdit), where both words have editMinLength letters or more, which counts 0.5.

A document's score is sum over the query's words q of (max over its fields f and their words t of weight(f) * kind(q, t)): each query word's best match, its kind times the weight of the field it is in, summed over the query's words, each counted once. With match: 'all' (the default) every query word must match somewhere in the document; with 'any', at least one. Hits come highest score first; a tie keeps the order the documents were added in, a document added again under its id counting as added last.

A hit holds each field of the document with its matched words marked, and up to passages sentences or lines of the passage fields that hold a match: those that hold the most query words, then the earliest, shown in the order of the text, each clipped to about passageWidth characters around its first match. A document matched only in another field shows the opening of its text, unmarked. A mark is a whole word at UTF-16 offsets into the text as the index holds it, in NFC; the marks of a text are in order and never overlap.

A search reads the words near each query word's, not every word the index holds: it looks a query word up in two orders of the index's words, by how they begin and by how they end. It reads the words that begin with the query word's first half, the query word itself and the words it begins among them, the words that end with its second half, and the one word with the two letters on either side of its middle swapped; a query word under editMinLength letters reads only the words it begins. The first search after a document is added brings the two orders up to date. The fields of the hits a search answers are then read again to mark them.

Pure and synchronous: it reads and writes nothing outside itself and imports nothing but the library's word rule, so a page can hold it. It is saved with FuzzyIndex.toJSON and restored with FuzzyIndex.fromJSON.

Example​

const index = new FuzzyIndex({ fields: { title: 100, keywords: 10, body: 1 } });
index.addDocument({
  id: 'n1',
  fields: { title: 'Enclosure samples', keywords: ['factory'], body: 'The samples leave the factory on Friday.' },
});
const [hit] = index.search('firday'); // "Friday", one swap away in the body: 0.5 times 1
hit.passages; // [{ text: 'The samples leave the factory on Friday.', marks: [{ start: 33, end: 39 }] }]

const restored = FuzzyIndex.fromJSON(JSON.parse(JSON.stringify(index.toJSON())));

Constructors​

Constructor​

new FuzzyIndex(config): FuzzyIndex

Defined in: packages/agentos/src/cognition/rag/search/FuzzyIndex.ts:530

Makes an empty index.

Parameters​

config​

FuzzyIndexConfig

Returns​

FuzzyIndex

Throws​

When fields names no field or gives a weight that is not a finite number above 0, when passageFields names a field fields does not or one twice, or when editMinLength, passages or passageWidth is not a whole number of at least 1, 0 and 1.

Throws​

When tokenize is given and is not a function.

Methods​

addDocument()​

addDocument(document): void

Defined in: packages/agentos/src/cognition/rag/search/FuzzyIndex.ts:586

Adds a document, replacing one of the same id, which then counts as added last. Fields the index was not made with are left out, and a field given as undefined is taken as not given. A refused document leaves the index as it was.

Parameters​

document​

FuzzyDocument

Returns​

void

Throws​

When the id is not a text, fields is not an object, or a field is not a text or a list of texts.

Throws​

When the id is empty.

Throws​

When the tokenizer answers a word that is not a stretch of its text.

addDocuments()​

addDocuments(documents): void

Defined in: packages/agentos/src/cognition/rag/search/FuzzyIndex.ts:626

Adds several documents, in order. A document that is refused throws, and those before it stay added.

Parameters​

documents​

Iterable<FuzzyDocument>

Returns​

void

removeDocument()​

removeDocument(id): boolean

Defined in: packages/agentos/src/cognition/rag/search/FuzzyIndex.ts:631

Removes a document; true when it was there.

Parameters​

id​

string

Returns​

boolean

search(query, options?): FuzzyHit[]

Defined in: packages/agentos/src/cognition/rag/search/FuzzyIndex.ts:713

Searches. A query with no word in it, such as an empty one or punctuation alone, finds nothing.

Parameters​

query​

string

The query text.

options?​

FuzzySearchOptions = {}

topK and match; see FuzzySearchOptions.

Returns​

FuzzyHit[]

The documents found, highest score first, a tie in the order they were added.

Throws​

When the query is not a text.

Throws​

When topK is not a whole number of 0 or more or Infinity, or match is not 'all' or 'any'.

toJSON()​

toJSON(): FuzzyIndexJSON

Defined in: packages/agentos/src/cognition/rag/search/FuzzyIndex.ts:822

The index as plain JSON: its settings and each document's id and fields, in the order they were added.

Returns​

FuzzyIndexJSON

fromJSON()​

static fromJSON(json, options?): FuzzyIndex

Defined in: packages/agentos/src/cognition/rag/search/FuzzyIndex.ts:844

A saved index restored: its settings and documents from the JSON, its tokenizer from options (the default unless given). A saved index holds texts, so the restore reads them again with that tokenizer.

Parameters​

json​

FuzzyIndexJSON

options?​
tokenize?​

FuzzyTokenizer

Returns​

FuzzyIndex

Throws​

When the saved index is of another version or is not one FuzzyIndex.toJSON could have written; the message names what is wrong.