3 ms·
I haven’t read the article in depth, but transformer style self-attention is usually permutation equivariant, meaning that if you change the order of inputs the
by bjornsing 5y ago
I haven’t read the article in depth, but transformer style self-attention is usually permutation equivariant, meaning that if you change the order of inputs then you get the same change in order of outputs, but the outputs themselves stay the same.
As I understand it all outputs are then added together / averaged into one vector. That should make the model ”hard” permutation invariant.
- visarga 5y agoyes, there are a bunch of permutation invariant operations - such as sum, max and pairwise interactions
- sdenton4 5y agoMost actual implementations use a position encoding, though, which breaks permutation invariance. This is a good inductive bias... word order matters, for example.
- bjornsing 5y agoSure. But it would be very weird to use a position embedding here, when the whole point is permutation invariance.
- Buttons840 5y agoI must not understand what transformers are. I thought they were a type of RNN where order of input matters? Any suggestions on how to learn about transformers? Or a few sentences to get me started on the right path?