4 ms·
Doesn't position become irrelevant after some distance in context window? I mean for long data table it's often doesn't matter how it's sorted. For meaningful t
by two_in_one 3y ago
Doesn't position become irrelevant after some distance in context window? I mean for long data table it's often doesn't matter how it's sorted. For meaningful text, text in between changes the meaning :). Transformers don't capture this. And just position may be too simplistic. RNNs (mentioned in other comments) with proper design can be better solution(?)
- HarHarVeryFunny 3y agoPositional encoding is relative, so it helps where word order is important at a short-range distance. This relative positioning will work around any absolute position in the context. The main way that long contexts are useful is via the key query "attention" mechanism, since the key may be matched at any position in the context. This allows something at the end of the context to refer to something at the beginning (or anyplace else).