enumerateTokenVectors(in:using:)
Iterates over the embedding vectors corresponding to the subword tokens within the specified range of the input string.
Declaration
@nonobjc func enumerateTokenVectors(in range: Range<String.Index>, using block: ([Double], Range<String.Index>) -> Bool)Parameters
- range:
The range in the string to enumerate.
- block:
A block that contains each token’s embedding vector and its corresponding character range in the string.
Discussion
Use this method to access the individual (subword) token embeddings. You can apply pooling or combination techniques to aggregate these subword vectors into a single representation for a word, phrase, or entire input.
Common pooling techniques include:
Mean pooling to take the average of subword vectors.
Max pooling for finding the element-wise maximum across tokens.
Use the embeddings of the first or last subword tokens to represent the entire input.