Contents

NLTokenizer

A tokenizer that segments natural language text into semantic units.

Declaration

class NLTokenizer

Mentioned in

Overview

NLTokenizer creates individual units from natural language text. Define the desired unit (word, sentence, paragraph, or document as declared in the NLTokenUnit) for tokenization, and then assign a string to tokenize. The enumerateTokensInRange:usingBlock: method provides the ranges of the tokens in the string based on the tokenization unit.

For more information, see Tokenizing natural language text.

Topics

Creating a tokenizer

Configuring a tokenizer

Enumerating the tokens

See Also

Tokenization