3 ms·
Sorry I meant the omission can be anything in the context, not anything in the world.. lol. That is still hard. You only have so many attention heads looking f
by XenophileJKO 1y ago
Sorry I meant the omission can be anything in the context, not anything in the world.. lol.
That is still hard. You only have so many attention heads looking for things.. you can't pay attention to EVERYTHING.. which is what is required to find the omission.
- yorwba 1y agoTo pay attention to everything, set the query vector to 0. Then all attention scores will be equal and the attention output is the average of the value vectors.