Tokenization Modifiers
Tokenization options are used with Cfstringtokenizercreate(_:_:_:_:_:) to specify how the string should be tokenized
Overview
You use the tokenization unit options with CFStringTokenizerCreate(_:_:_:_:_:) to specify how a string should be tokenized.
You use the modifiers together with a tokenization unit to modify the way the string is tokenized.
You use the attribute specifiers to tell the tokenizer to prepare the specified attribute when it tokenizes the given string. You can retrieve the attribute value by calling CFStringTokenizerCopyCurrentTokenAttribute(_:_:) with one of the attribute options.
The locale sensitivity of the tokenization unit options may change in a future release.