You might be misunderstanding the use of sin-cos in early positional embeddings. The waves are shifted such that each position gets a unique positional encoding

You might be misunderstanding the use of sin-cos in early positional embeddings. The waves are shifted such that each position gets a unique positional encoding. These encodings are not relative in distance to other positions, they are at best ordinal. They are meant to provide the same flexibility for the model as absolute positional encoding that are learned (so, position=2 always gets the same encoding) - which is what they achieve. The matter of learning about distances between positions is left up to the later stages of the model. This differs from the much more modern approach of relative positional embeddings, for instance Alibi or Rotary Embeddings. These I think fit your intuition much better, as they seek to encode relative distances between tokens correctly.