This is a great point to illustrate the importance of the token representation of position, though I didn't find ChatGPT had any problem with the second example

This is a great point to illustrate the importance of the token representation of position, though I didn't find ChatGPT had any problem with the second example you showed with extra words though - it was correct. I'm not sure the cases you reference work in practise how you describe exactly really nowadays, even though they did in the past (say, 2019) - I agree the wraparound effect could occur like that, though the way the position is encoded in the vector is usually built to work so that it will wrap around at different levels as such (the frequencies w) which include padding tokens as non-zero values (implementation dependent), so in essence for every sentence ending at 8 tokens, the model has many other encodings that indicate the loop, not just the one wraparound encoding. This should avoid that problem and the reason for this relates to spans of tokens and is a problem considered when devising training for earlier transformer models like BART and the MASK/NSP tasks. But in practise, position is learned these days, it does not use sin/cosine anymore. There is a technical difference in positional embedding vs positional encodings (and the trend since 2020 to learn position) which is quite interesting.