3 ms·
> squish the positional encodings for the keys down to position zero I might be misunderstanding, but wouldn't this turn your instructions into a bag of words?
by bhickey 2y ago
> squish the positional encodings for the keys down to position zero
I might be misunderstanding, but wouldn't this turn your instructions into a bag of words?
- valine 2y agoNo, and that’s because we are talking about relative positions. Every query can have its own set of keys. From the perspective of token 100 token 3 would be squished down, but from the perspective of token 3 it is still at position 3 and can see tokens 0,1,2 without them being squished.