I see, so [1, 2, 3, 4, 5, 6, 7, 8] is more similar to [5, 3, 1, 7, 8, 2, 4, 6] than with a linear layer? That's what you mean by permutation equivariant? I und

I see, so [1, 2, 3, 4, 5, 6, 7, 8] is more similar to [5, 3, 1, 7, 8, 2, 4, 6] than with a linear layer? That's what you mean by permutation equivariant? I understand each context input is embedded with its position, but I suppose the transformer can learn to ignore the position and just look at the context as an unordered set?