Wouldn't the important embedding be of the token at position A and the token at position B rather than the positions themselves? Since there's a lot fewer token

Wouldn't the important embedding be of the token at position A and the token at position B rather than the positions themselves? Since there's a lot fewer tokens than positions you're fairly likely to get the two tokens to hash together at least a few times. edit: I'd image the position itself (ie: word number in text rather than token) could be embedded using sin-cosine or by breaking it up into chapter/paragraph/word. Seems more meaningful and efficient than word number in text. That would prevent this issue on that side of things.