3 ms·
Why do we add the positional encoding to the original vector instead of appending it? Doesn't adding positional data to the embedding destroy information?
by fancyfredbot 3y ago
Why do we add the positional encoding to the original vector instead of appending it? Doesn't adding positional data to the embedding destroy information?
- razodactyl 3y agoThe hope is that the neural network will learn to undo the positional information and understand that it's an augmentation used to tell the neural network where the word occurred in the sequence. Sequence being a misleading term since the input to the network is actually a set of tokens meaning no positional information without the extra information added.