4 ms·
Is this "hard" permutation-invariant? For example, does shuffling the order of the inputs really have zero effect on the output? Or is it just "soft" permutatio
by Buttons840 5y ago
Is this "hard" permutation-invariant? For example, does shuffling the order of the inputs really have zero effect on the output? Or is it just "soft" permutation-invariant? Looking at their architecture diagram, the "attention" network receives input from the "sensory neurons" in a certain order, right? If so, I suspect order does matter, although they attempt to minimize the importance of the order.
- bjornsing 5y agoI haven’t read the article in depth, but transformer style self-attention is usually permutation equivariant, meaning that if you change the order of inputs then you get the same change in order of outputs, but the outputs themselves stay the same. As I understand it all outputs are then added together / averaged into one vector. That should make the model ”hard” permutation invariant.
- visarga 5y agoyes, there are a bunch of permutation invariant operations - such as sum, max and pairwise interactions
- sdenton4 5y agoMost actual implementations use a position encoding, though, which breaks permutation invariance. This is a good inductive bias... word order matters, for example.
- bjornsing 5y agoSure. But it would be very weird to use a position embedding here, when the whole point is permutation invariance.
- Buttons840 5y agoI must not understand what transformers are. I thought they were a type of RNN where order of input matters? Any suggestions on how to learn about transformers? Or a few sentences to get me started on the right path?