Function: lexicalTokens()

lexicalTokens(text): string[]

Defined in: packages/agentos/src/cognition/library/tokens.ts:37

The words of a text: read in NFC and in lower case, each a letter or a digit and the letters, digits and marks after it, in order.

Parameters

text

string

Returns

string[]