One aspect of the specific positional embedding used there is that it explicitly encodes a signal that the very first attention layer can directly use for both

One aspect of the specific positional embedding used there is that it explicitly encodes a signal that the very first attention layer can directly use for both relative and absolute position - i.e. that there can be a trivial set of wights for "pay attention only to the token two tokens to the right from the target token" and also a trivial set of weights saying "pay attention to the first token in the sequence" and also "pay attention to this aspect of all the words weighed by the distance from the target token". As by default the transformer architecture is effectively position-blind, having the flexibility to learn all these different types of relations is important; and many possible simple, clean, efficient position encodings make it easy to represent some relations but very difficult for others, perhaps theoretically possible but needing extra layers and/or hard to learn by gradient descent.