4 ms·
The big difference is that the transformer is (approximately) permutation equivariant, which makes a massive difference in generalization and training speed.
by QuadmasterXLII 4y ago
The big difference is that the transformer is (approximately) permutation equivariant, which makes a massive difference in generalization and training speed.
- Buttons840 4y agoI see, so [1, 2, 3, 4, 5, 6, 7, 8] is more similar to [5, 3, 1, 7, 8, 2, 4, 6] than with a linear layer? That's what you mean by permutation equivariant? I understand each context input is embedded with its position, but I suppose the transformer can learn to ignore the position and just look at the context as an unordered set?
- eachro 4y agoWhat makes it approximately permutation equivariant (vs entirely)? As I understand things, if the order is jumbled, the attention matrix does get its rows and cols permuted in the way you'd expect so I'd have thought they'd be entirely permutation equivariant.
- YetAnotherNick 4y agoInputs have position encoding in them.