Interface: TokenizationOptions

Defined in: packages/agentos/src/cognition/nlp/ai_utilities/IUtilityAI.ts:67

Properties

language?

optional language: string

Defined in: packages/agentos/src/cognition/nlp/ai_utilities/IUtilityAI.ts:73

---

modelId?

optional modelId: string

Defined in: packages/agentos/src/cognition/nlp/ai_utilities/IUtilityAI.ts:74

---

removePunctuation?

optional removePunctuation: boolean

Defined in: packages/agentos/src/cognition/nlp/ai_utilities/IUtilityAI.ts:72

LLMUtilityAI: default true for words, false for sentences. StatisticalUtilityAI's words never hold punctuation.

---

toLowerCase?

optional toLowerCase: boolean

Defined in: packages/agentos/src/cognition/nlp/ai_utilities/IUtilityAI.ts:70

Default true. StatisticalUtilityAI keeps a sentence's case unless it is true.

---

type?

optional type: "word" | "sentence" | "subword_bpe"

Defined in: packages/agentos/src/cognition/nlp/ai_utilities/IUtilityAI.ts:68