4 ms·
The tokens are the structure over which the attention mechanism is permutation equivariant. This structure permeates the forward pass, its important at every la
by QuadmasterXLII 3y ago
The tokens are the structure over which the attention mechanism is permutation equivariant. This structure permeates the forward pass, its important at every layer and will be until we find something better than attention.