Contents

MLCMultiheadAttentionLayer

A multihead, scaled dot-product attention layer that attends to one or more entries in the input key-value pairs.

Declaration

class MLCMultiheadAttentionLayer

Overview

The dimensions of projections are as follows:

Query: (1, headCount, keyDimension/headCount, modelDimension)
Key: (1, headCount, keyDimension/headCount, modelDimension)
Value: (1, headCount, valueDimension/headCount, modelDimension)
Output: (1, 1, modelDimension, valueDimension)

``

Topics

Creating Multi-Head Attention Layers

Inspecting Multi-Head Attention Layers

See Also

Activation Layers