4 ms·
Positional embeddings used in transformers are actually suspiciously similar to grid cells. They allow for a very similar kind of coordinate transform.
by phreeza 2y ago
Positional embeddings used in transformers are actually suspiciously similar to grid cells. They allow for a very similar kind of coordinate transform.
- soraki_soladead 2y agoYou might enjoy this paper[0] which shows that recurrent position encodings recover grid cell representations and maps to path integration found in a popular model of the hippocampus. This isn't terribly surprising since RNNs have shown this before[1, 2] but its an interesting connection. [0] https://arxiv.org/abs/2112.04035 https://arxiv.org/abs/2112.04035 [1] https://arxiv.org/abs/1803.07770 https://arxiv.org/abs/1803.07770 [2] https://www.nature.com/articles/s41586-018-0102-6 https://www.nature.com/articles/s41586-018-0102-6