3 ms·
Can somebody explain to me the sinus wave positional encoding thing? The naïve approach would be to just add number indices to the tokens, wouldn’t it?
by leobg 4y ago
Can somebody explain to me the sinus wave positional encoding thing? The naïve approach would be to just add number indices to the tokens, wouldn’t it?
- tipsytoad 4y agoI'm no expert, but I think it's so that the model can learn the relative position wrt other tokens. They use indices for models like vision transformers with a fixed number of patches but for variable length context I think it's more beneficial to use encodings that can also capture the relative distance.
- ftxbro 4y agoAccording to https://kazemnejad.com/blog/transformer_architecture_positional_encoding/ https://kazemnejad.com/blog/transformer_architecture_positio... they explain it as a clever way to satisfy the following criteria: - It should output a unique encoding for each time-step (word’s position in a sentence) - Distance between any two time-steps should be consistent across sentences with different lengths. - Our model should generalize to longer sentences without any efforts. Its values should be bounded. - It must be deterministic. Your example contradicts the 'values should be bounded' criterion as it generalizes to longer sentences.