---
title: Tokenization Modifiers
framework: corefoundation
role: collectionGroup
role_heading: API Collection
path: corefoundation/1588024-tokenization-modifiers
---

# Tokenization Modifiers

Tokenization options are used with CFStringTokenizerCreate(_:_:_:_:_:) to specify how the string should be tokenized

## Overview

Overview You use the tokenization unit options with CFStringTokenizerCreate(_:_:_:_:_:) to specify how a string should be tokenized. You use the modifiers together with a tokenization unit to modify the way the string is tokenized. You use the attribute specifiers to tell the tokenizer to prepare the specified attribute when it tokenizes the given string. You can retrieve the attribute value by calling CFStringTokenizerCopyCurrentTokenAttribute(_:_:) with one of the attribute options. The locale sensitivity of the tokenization unit options may change in a future release.

## Topics

### Constants

- [kCFStringTokenizerUnitWord](corefoundation/kcfstringtokenizerunitword.md)
- [kCFStringTokenizerUnitSentence](corefoundation/kcfstringtokenizerunitsentence.md)
- [kCFStringTokenizerUnitParagraph](corefoundation/kcfstringtokenizerunitparagraph.md)
- [kCFStringTokenizerUnitLineBreak](corefoundation/kcfstringtokenizerunitlinebreak.md)
- [kCFStringTokenizerUnitWordBoundary](corefoundation/kcfstringtokenizerunitwordboundary.md)
- [kCFStringTokenizerAttributeLatinTranscription](corefoundation/kcfstringtokenizerattributelatintranscription.md)
- [kCFStringTokenizerAttributeLanguage](corefoundation/kcfstringtokenizerattributelanguage.md)

## See Also

### Constants

- [CFStringTokenizerTokenType](corefoundation/cfstringtokenizertokentype.md)