Contents

NLTokenUnit.word

An individual word.

Declaration

case word

Mentioned in

Discussion

Use this linguistic unit to tokenize text into individual words, like in the following example:

let text = "This is a sentence containing several words. 😀"

let tokenizer = NLTokenizer(unit: .word)
tokenizer.string = text

let range = text.startIndex..<text.endIndex

let tokenArray = tokenizer.tokens(for: range)
print("Number of tokens: \(tokenArray.count)")

tokenizer.enumerateTokens(in: range) { tokenRange, _ in
    print(text[tokenRange])
    return true
}

For more information, see Tokenizing natural language text.

See Also

Constants