4 ms·
In self attention each token creates both "query" and "key" vectors. These are different vectors, so one type of token can look up data for different types of t
by johntb86 4y ago
In self attention each token creates both "query" and "key" vectors. These are different vectors, so one type of token can look up data for different types of tokens. "Apple" can generated a verb query to look for a verb in the sentence, while also generating a noun key that other words can look for.
And with multi-head attention, a single token can get data from different types of tokens.
The "self" in self-attention just means that it's looking at other parts of the same string, rather than looking at weights generated from a different string.