2 ms·
Most actual implementations use a position encoding, though, which breaks permutation invariance. This is a good inductive bias... word order matters, for examp
by sdenton4 5y ago
Most actual implementations use a position encoding, though, which breaks permutation invariance. This is a good inductive bias... word order matters, for example.
- bjornsing 5y agoSure. But it would be very weird to use a position embedding here, when the whole point is permutation invariance.