Class: StatisticalUtilityAI
Defined in: packages/agentos/src/cognition/nlp/ai_utilities/StatisticalUtilityAI.ts:315
Interface
IUtilityAI Defines the contract for a comprehensive Utility AI service.
Implements
Constructors
Constructor
new StatisticalUtilityAI(
utilityId?):StatisticalUtilityAI
Defined in: packages/agentos/src/cognition/nlp/ai_utilities/StatisticalUtilityAI.ts:363
Parameters
utilityId?
string
Returns
StatisticalUtilityAI
Properties
utilityId
readonlyutilityId:string
Defined in: packages/agentos/src/cognition/nlp/ai_utilities/StatisticalUtilityAI.ts:316
Implementation of
Methods
analyzeSentiment()
analyzeSentiment(
text,options?):Promise<SentimentResult>
Defined in: packages/agentos/src/cognition/nlp/ai_utilities/StatisticalUtilityAI.ts:847
Scores a text's sentiment with a word lexicon, with no model call.
The lexicon is the first one natural ships for the language (options.language, else
sentimentConfig.defaultLexiconLanguage, else defaultLanguage): AFINN for English, Spanish and Portuguese,
pattern for French, German, Italian and Dutch, senticon for Catalan, Basque and Galician. The entries of
sentimentConfig.lexiconPath join it in the default lexicon language. options.lexiconNameOrPath picks another
of natural's lexicons by name ('afinn', 'afinnFinancialMarketNews', 'pattern', 'senticon'), or a file
that is then the lexicon alone. Another language scores 0 unless lexiconPath gives it words.
The text is read as words (letters, marks and digits, an apostrophe inside a word kept, lower case) in clauses
that end at . , ; : ! ? or …. At each word, a phrase the lexicon lists that starts there (can't stand,
not good, well-being; the longest when several do) takes its value; else a negation word of the language (in
English not, no, never, neither and any word ending in n't) flips the value of the lexicon entries that
start among the next three words of its clause; else a word the lexicon lists takes its value. A word scores only
in a form the lexicon lists: nothing is stemmed. score is the sum of the values put on AFINN's scale (pattern
and senticon values times 5) and mapped into -1 to 1 as VADER maps its compound score: s / sqrt(s * s + 15).
polarity is positive from 0.05, negative from -0.05, else neutral; intensity is the score's size;
comparative is the sum over the number of words, as natural computes it. positiveTokens and
negativeTokens hold each scored word or phrase with its signed value, and neutralTokens the other words.
Parameters
text
string
options?
Returns
Promise<SentimentResult>
Throws
NOT_SUPPORTED when lexiconNameOrPath names a lexicon natural has no version of in the
language, and RESOURCE_NOT_FOUND when it names a file that cannot be read.
Implementation of
calculateReadability()
calculateReadability(
text,options):Promise<ReadabilityResult>
Defined in: packages/agentos/src/cognition/nlp/ai_utilities/StatisticalUtilityAI.ts:1017
Parameters
text
string
options
Returns
Promise<ReadabilityResult>
Implementation of
IUtilityAI.calculateReadability
calculateSimilarity()
calculateSimilarity(
text1,text2,options?):Promise<number>
Defined in: packages/agentos/src/cognition/nlp/ai_utilities/StatisticalUtilityAI.ts:762
A similarity from 0 to 1. 'jaccard' and 'cosine_tfidf' (the default) compare the texts' words: the class's
words in lower case, the stop words of language dropped (removeStopWords, on unless false) and Porter-stemmed
for the language (stem, on unless false). The corpusForIDF texts are read the same way. 'levenshtein'
compares the raw texts.
Parameters
text1
string
text2
string
options?
Returns
Promise<number>
Implementation of
IUtilityAI.calculateSimilarity
checkHealth()
checkHealth():
Promise<{dependencies?:object[];details?:any;isHealthy:boolean; }>
Defined in: packages/agentos/src/cognition/nlp/ai_utilities/StatisticalUtilityAI.ts:1074
Returns
Promise<{ dependencies?: object[]; details?: any; isHealthy: boolean; }>
Implementation of
classifyText()
classifyText(
textToClassify,options):Promise<ClassificationResult>
Defined in: packages/agentos/src/cognition/nlp/ai_utilities/StatisticalUtilityAI.ts:648
Classifies a text. With method: 'naive_bayes' and a model trained or loaded under modelId (else
classifierConfig.naiveBayes.defaultModelId, else 'default_bayes_classifier', the id trainModel() uses), the
model reads the text as it read its examples (the class's words in any script, lower case, natural's English
stop words left out, each stemmed with the stemmer the model learned with; language is not read) and answers
its classes, best first. Otherwise it logs a warning and scores each candidate class by the words of its name
(split at white space) found in the lower-cased text, inside a longer word too.
Parameters
textToClassify
string
options
Returns
Promise<ClassificationResult>
Implementation of
detectLanguage()
detectLanguage(
text,options?):Promise<LanguageDetectionResult[]>
Defined in: packages/agentos/src/cognition/nlp/ai_utilities/StatisticalUtilityAI.ts:924
Detect the language of a text with the package's 21 language profiles.
The script that holds more than half of the text's letters decides first: Hangul is Korean, Han with kana that make up a tenth of those letters or more is Japanese and with fewer Chinese, and Arabic, Devanagari, Bengali and Cyrillic text is Arabic, Hindi, Bengali, Russian or Ukrainian when its letters show which; that language comes back alone. Latin-script text is ranked against 13 trigram profiles (English, Spanish, French, German, Portuguese, Italian, Dutch, Turkish, Vietnamese, Polish, Swedish, Danish, Norwegian) by Cavnar and Trenkle's out-of-place distance.
A text shorter than 10 characters after trimming, and one the profiles do not decide, answers
[{ language: defaultLanguage, confidence: 0.1 }], never 'und'.
Parameters
text
string
The input text to analyse.
options?
maxCandidates caps the ranked list (3 unless given). method is not read.
Returns
Promise<LanguageDetectionResult[]>
Ranked { language, confidence } pairs: ISO 639-1 codes ('en', 'fr'), confidences from 0 to 1.
Implementation of
extractKeywords()
extractKeywords(
textToAnalyze,options?):Promise<string[]>
Defined in: packages/agentos/src/cognition/nlp/ai_utilities/StatisticalUtilityAI.ts:719
The text's most frequent words, most frequent first, a tie in the order the words first appear: 5 unless
maxKeywords says otherwise. The words are the class's (any script, lower case), less natural's English stop
words and the class's stop words for language (else defaultLanguage): the built-in English list, or the list
customStopWordsPaths gave the language. A TF-IDF over the one text would rank them the same, since every word
of a single text has the same inverse document frequency.
Parameters
textToAnalyze
string
options?
Returns
Promise<string[]>
Implementation of
generateNGrams()
generateNGrams(
tokens,options):Promise<Record<number,string[][]>>
Defined in: packages/agentos/src/cognition/nlp/ai_utilities/StatisticalUtilityAI.ts:996
Parameters
tokens
string[]
options
Returns
Promise<Record<number, string[][]>>
Implementation of
initialize()
initialize(
config):Promise<void>
Defined in: packages/agentos/src/cognition/nlp/ai_utilities/StatisticalUtilityAI.ts:375
Parameters
config
Returns
Promise<void>
Implementation of
loadTrainedModel()
loadTrainedModel(
modelId,modelType?,storagePath?):Promise<{message?:string;success:boolean; }>
Defined in: packages/agentos/src/cognition/nlp/ai_utilities/StatisticalUtilityAI.ts:1154
Parameters
modelId
string
modelType?
string
storagePath?
string
Returns
Promise<{ message?: string; success: boolean; }>
Implementation of
normalizeText()
normalizeText(
text,options?):Promise<string>
Defined in: packages/agentos/src/cognition/nlp/ai_utilities/StatisticalUtilityAI.ts:945
Parameters
text
string
options?
Returns
Promise<string>
Implementation of
parseJsonSafe()
parseJsonSafe<
T>(jsonString,options?):Promise<T|null>
Defined in: packages/agentos/src/cognition/nlp/ai_utilities/StatisticalUtilityAI.ts:1202
The JSON the text holds, read as LLMUtilityAI reads it (parseReplyJson: the text as it is, else inside the first
Markdown code fence it holds, backticks inside a sentence opening none, with a language tag or none and with words
before or after it, up to the first later line that starts with as many backticks as opened the fence or more),
then a check against targetSchema with Ajv when given, as LLMUtilityAI checks it: the standard formats checked, a
keyword or a format Ajv does not know passed over, and a value coerced to the type its schema names where it can
(targetSchemaChecks). Answers null when the text is empty or neither it nor a text inside a fence parses, when
the value fails the schema, and when Ajv cannot compile the schema, the schema takes the draft-07 meta-schema's
$id or it is marked $async. It repairs nothing: a HybridUtilityAI that holds an LLM-backed class sends a text
this answers null for on to it.
Type Parameters
T
T = any
Parameters
jsonString
string
options?
Returns
Promise<T | null>
Implementation of
saveTrainedModel()
saveTrainedModel(
modelId,modelType?,storagePath?):Promise<{message?:string;pathOrStoreId?:string;success:boolean; }>
Defined in: packages/agentos/src/cognition/nlp/ai_utilities/StatisticalUtilityAI.ts:1126
Parameters
modelId
string
modelType?
string
storagePath?
string
Returns
Promise<{ message?: string; pathOrStoreId?: string; success: boolean; }>
Implementation of
shutdown()
shutdown():
Promise<void>
Defined in: packages/agentos/src/cognition/nlp/ai_utilities/StatisticalUtilityAI.ts:1235
Returns
Promise<void>
Implementation of
stemTokens()
stemTokens(
tokens,options?):Promise<string[]>
Defined in: packages/agentos/src/cognition/nlp/ai_utilities/StatisticalUtilityAI.ts:750
Parameters
tokens
string[]
options?
Returns
Promise<string[]>
Implementation of
summarize()
summarize(
textToSummarize,options?):Promise<string>
Defined in: packages/agentos/src/cognition/nlp/ai_utilities/StatisticalUtilityAI.ts:613
The first sentences of the text, joined by spaces.
How many: 3 unless desiredLength says otherwise. 'short' is 2, 'medium' 30 percent of the sentences and at
least 3, 'long' half of them and at least 4. A number of 1 or less is a fraction of the sentences, rounded, so
0 asks for none; a larger number is a count; a number that is not a number (NaN) keeps the default. At least
1 sentence and at most all of them.
method: 'first_n_sentences' does this, and so does the default, 'extractive_sentence_rank', after a warning.
Parameters
textToSummarize
string
options?
Returns
Promise<string>
Throws
NOT_SUPPORTED for any other method.
Implementation of
tokenize()
tokenize(
text,options?):Promise<string[]>
Defined in: packages/agentos/src/cognition/nlp/ai_utilities/StatisticalUtilityAI.ts:741
Words by the class's word rule (a letter, digit or underscore, then letters, marks, digits and underscores, in any
script), lower-cased unless toLowerCase: false. A word never holds punctuation, so removePunctuation changes
nothing. With type: 'sentence', the sentences from natural's sentence tokenizer, case kept unless
toLowerCase: true.
Parameters
text
string
options?
Returns
Promise<string[]>
Implementation of
trainModel()
trainModel(
trainingData,modelType,trainingOptions?):Promise<{message?:string;modelId?:string;success:boolean; }>
Defined in: packages/agentos/src/cognition/nlp/ai_utilities/StatisticalUtilityAI.ts:1090
Parameters
trainingData
object[]
modelType
string
trainingOptions?
alpha?
number
modelId?
string
stemmer?
"porter" | "lancaster"
Returns
Promise<{ message?: string; modelId?: string; success: boolean; }>