Skip to main content

Text Processing Pipeline

A pipeline turns raw text into the tokens an index stores and a query is matched with. A tokenizer splits the text, and each stage after it changes, drops or annotates the tokens.

import { createCodePipeline, createRagPipeline } from '@framers/agentos/cognition/nlp';

const rag = createRagPipeline();
rag.processToStrings('Running runs'); // ['run', 'run']
rag.getStageNames(); // ['StandardTokenizer', 'LowercaseNormalizer', 'StopWordFilter', 'PorterStemmer']

const code = createCodePipeline();
code.processToStrings('getUserName XMLParser path.to.module');
// ['get', 'user', 'name', 'xml', 'parser', 'path', 'module']

The module is @framers/agentos/cognition/nlp; @framers/agentos/nlp is the same module under a shorter name. The package root does not export the pipeline classes. The module also exports the text statistics classes.

The pipeline​

TextProcessingPipeline holds one tokenizer and an ordered list of stages.

MemberWhat it does
new TextProcessingPipeline(tokenizer)A pipeline with the tokenizer and no stages.
add(stage)Appends a stage and answers the pipeline, so calls chain.
process(text)Runs the tokenizer, then each stage in the order added, and answers the tokens.
processToStrings(text)process(text) mapped to each token's text: the form an index stores.
getStageNames()The tokenizer's name, then each stage's.

Each token is a Token:

FieldValue
textThe token as the stages left it.
originalThe text the tokenizer cut, before any stage ran; for CodeTokenizer, the whole identifier the part came from.
positionWhere the token starts in the input, in UTF-16 code units.
stemSet by PorterStemmer.
lemmaSet by WordNetLemmatizer.

rag.process('Running runs') answers [{ text: 'run', original: 'Running', position: 0, stem: 'run' }, { text: 'run', original: 'runs', position: 8, stem: 'run' }].

Tokenizers​

ClassWhat it cuts
StandardTokenizer(minLength = 2)Runs of letters, digits and underscores in any script (/[\p{L}\p{N}_]+/u), case kept. A run shorter than minLength is dropped.
CodeTokenizer(minLength = 2)Identifiers: it takes runs of letters, digits, underscores and dots, then splits them at dots, at underscores, between an ASCII lower-case letter or digit and an ASCII capital (getUserName to get, User, Name), and before the last capital of a run of capitals followed by a lower-case letter (XMLParser to XML, Parser). Case is kept; original is the whole identifier. A part shorter than minLength is dropped.

A tokenizer is any object with a name and tokenize(text): Token[] (ITokenizer).

Stages​

ClassWhat it does
LowercaseNormalizerLower-cases text.
AccentStripperDecomposes text (Unicode NFD) and removes the combining marks U+0300 to U+036F: café becomes cafe.
StopWordFilter(stopWords?)Drops each token whose text is in the set, compared as written: put it after LowercaseNormalizer. With no set it uses getNaturalStopWords().
PorterStemmerReplaces text with its Porter stem from the natural package and sets stem: running to run, foxes to fox, connected to connect. natural's stemmer lower-cases tokens of three or more characters.
NoOpStemmerPasses tokens through. createCodePipeline() ends with it, so identifiers are not stemmed.
WordNetLemmatizerReplaces text with the stem from natural's Lancaster stemmer and sets lemma to it. It does not look words up in WordNet, so better does not become good.
NoOpLemmatizerPasses tokens through.

PorterStemmer and WordNetLemmatizer load natural on their first process() call; await stage.initialize() loads it ahead of time. When natural cannot be loaded they pass tokens through unchanged.

A stage is any object with a name and process(tokens): Token[] (ITextProcessor):

import {
  ENGLISH_STOP_WORDS,
  LowercaseNormalizer,
  StandardTokenizer,
  StopWordFilter,
  TextProcessingPipeline,
  type ITextProcessor,
} from '@framers/agentos/cognition/nlp';

const dropNumbers: ITextProcessor = {
  name: 'DropNumbers',
  process: (tokens) => tokens.filter((token) => !/^\d+$/.test(token.text)),
};

const pipeline = new TextProcessingPipeline(new StandardTokenizer(3))
  .add(new LowercaseNormalizer())
  .add(new StopWordFilter(ENGLISH_STOP_WORDS))
  .add(dropNumbers);

pipeline.processToStrings('The 2024 budget grew by 120 percent'); // ['budget', 'grew', 'percent']

Stop words​

ListWordsUsed by
getNaturalStopWords()natural's English list, 170 entries: 132 words, the 26 single letters, the 10 digits, $ and _. When natural cannot be loaded it answers ENGLISH_STOP_WORDS.new StopWordFilter(), createProsePipeline(), createRagPipeline() and the built-in tokenizer of BM25Index
ENGLISH_STOP_WORDS129 common English words.Pass it to StopWordFilter.
CODE_STOP_WORDS88 English function words. It leaves out class, function, import, const, let, var, return and this, which carry meaning in code.createCodePipeline()

natural's list holds neither no nor not, so the default filter keeps negations; ENGLISH_STOP_WORDS drops no, nor and not. The text statistics classes keep a list of their own.

Presets​

FunctionStages
createProsePipeline()StandardTokenizer, LowercaseNormalizer, AccentStripper, StopWordFilter (natural's list), PorterStemmer
createRagPipeline()StandardTokenizer, LowercaseNormalizer, StopWordFilter (natural's list), PorterStemmer
createCodePipeline()CodeTokenizer, LowercaseNormalizer, StopWordFilter (CODE_STOP_WORDS), NoOpStemmer

Each call builds a new pipeline. createProsePipeline() turns The café serves naïve résumé writers into tokens that include cafe, naiv and resum, and not the.

In a search index​

BM25Index tokenizes with a pipeline given as pipeline, so a query for run finds a passage with running:

import { createRagPipeline } from '@framers/agentos/cognition/nlp';
import { BM25Index } from '@framers/agentos/cognition/rag';

const index = new BM25Index({ pipeline: createRagPipeline() });
index.addDocument('a', 'The developers are running integration tests');
index.addDocument('b', 'She runs the application every morning');
index.search('run').map((hit) => hit.id); // both 'a' and 'b'

A saved index holds stemmed terms, so it is restored with the same pipeline: see Lexical and Hybrid Search.

Where it runs​

@framers/agentos/cognition/nlp runs in Node. StopWordFilter, PorterStemmer and WordNetLemmatizer load the CommonJS natural package, a dependency of @framers/agentos, through createRequire from node:module, and the module also loads the text statistics classes, which import fs/promises and natural. The pipeline classes call no model and need no key.

A page in a browser tokenizes with lexicalTokens from @framers/agentos/cognition/library/browser: words of letters, digits and combining marks in any script, read in NFC and in lower case, nothing stemmed (Words and snippets).

Examples​