The result of tokenizing the input string.

interface TextTokenizeResult {
    token_count: number;
    tokens?: string[];
}

Properties

Properties

token_count: number

The number of tokens in the input string.

tokens?: string[]

The input string broken up into the tokens, if requested.