---
title: CFStringTokenizer
framework: corefoundation
role: symbol
role_heading: Class
path: corefoundation/cfstringtokenizer
---

# CFStringTokenizer

## Declaration

```swift
class CFStringTokenizer
```

## Overview

Overview CFStringTokenizer allows you to tokenize strings into words, sentences or paragraphs in a language-neutral way. It supports languages such as Japanese and Chinese that do not delimit words by spaces, as well as de-compounding German compounds. You can obtain Latin transcription for tokens. It also provides language identification API. You can use a CFStringTokenizer to break a string into tokens (sub-strings) on the basis of words, sentences, or paragraphs. When you create a tokenizer, you can supply options to further modify the tokenization—see Tokenization Modifiers. In addition, with CFStringTokenizer: You can de-compound German compounds You can identify the language used in a string (using CFStringTokenizerCopyBestStringLanguage(_:_:)) You can obtain Latin transcription for tokens To find a token that includes the character specified by character index and set it as the current token, you call CFStringTokenizerGoToTokenAtIndex(_:_:). To advance to the next token and set it as the current token, you call CFStringTokenizerAdvanceToNextToken(_:). To get the range of current token, you call CFStringTokenizerGetCurrentTokenRange(_:). You can use         CFStringTokenizerCopyCurrentTokenAttribute(_:_:) to get the attribute of the current token. If the current token is a compound, you can call CFStringTokenizerGetCurrentSubTokens(_:_:_:_:) to retrieve the subtokens or derived subtokens contained in the compound token. To guess the language of a string, you call CFStringTokenizerCopyBestStringLanguage(_:_:).

## Topics

### Creating a Tokenizer

- [CFStringTokenizerCreate(_:_:_:_:_:)](corefoundation/cfstringtokenizercreate(_:_:_:_:_:).md)

### Setting the String

- [CFStringTokenizerSetString(_:_:_:)](corefoundation/cfstringtokenizersetstring(_:_:_:).md)

### Changing the Location

- [CFStringTokenizerAdvanceToNextToken(_:)](corefoundation/cfstringtokenizeradvancetonexttoken(_:).md)
- [CFStringTokenizerGoToTokenAtIndex(_:_:)](corefoundation/cfstringtokenizergototokenatindex(_:_:).md)

### Getting Information About the Current Token

- [CFStringTokenizerCopyCurrentTokenAttribute(_:_:)](corefoundation/cfstringtokenizercopycurrenttokenattribute(_:_:).md)
- [CFStringTokenizerGetCurrentTokenRange(_:)](corefoundation/cfstringtokenizergetcurrenttokenrange(_:).md)
- [CFStringTokenizerGetCurrentSubTokens(_:_:_:_:)](corefoundation/cfstringtokenizergetcurrentsubtokens(_:_:_:_:).md)

### Identifying a Language

- [CFStringTokenizerCopyBestStringLanguage(_:_:)](corefoundation/cfstringtokenizercopybeststringlanguage(_:_:).md)

### Getting the CFStringTokenizer Type ID

- [CFStringTokenizerGetTypeID()](corefoundation/cfstringtokenizergettypeid().md)

### Constants

- [Tokenization Modifiers](corefoundation/1588024-tokenization-modifiers.md)
- [CFStringTokenizerTokenType](corefoundation/cfstringtokenizertokentype.md)

## Relationships

### Conforms To

- [Equatable](swift/equatable.md)
- [Hashable](swift/hashable.md)

## See Also

### Related Documentation

- [String Programming Guide for Core Foundation](apple-archive/documentation/CoreFoundation/Conceptual/CFStrings/introCFStrings.html.md)

### Opaque Types

- [CFAllocator](corefoundation/cfallocator.md)
- [CFArray](corefoundation/cfarray.md)
- [CFAttributedString](corefoundation/cfattributedstring.md)
- [CFBag](corefoundation/cfbag.md)
- [CFBinaryHeap](corefoundation/cfbinaryheap.md)
- [CFBitVector](corefoundation/cfbitvector.md)
- [CFBoolean](corefoundation/cfboolean.md)
- [CFBundle](corefoundation/cfbundle.md)
- [CFCalendar](corefoundation/cfcalendar.md)
- [CFCharacterSet](corefoundation/cfcharacterset.md)
- [CFData](corefoundation/cfdata.md)
- [CFDate](corefoundation/cfdate.md)
- [CFDateFormatter](corefoundation/cfdateformatter.md)
- [CFDictionary](corefoundation/cfdictionary.md)
- [CFError](corefoundation/cferror.md)