Positional encoding is relative, so it helps where word order is important at a short-range distance. This relative positioning will work around any absolute po

Positional encoding is relative, so it helps where word order is important at a short-range distance. This relative positioning will work around any absolute position in the context. The main way that long contexts are useful is via the key query "attention" mechanism, since the key may be matched at any position in the context. This allows something at the end of the context to refer to something at the beginning (or anyplace else).