I recall paragraph2Vec. It was the earliest way to experiment with position embeddings that I recall reading. At first when I read it, it felt kind of crazy/wei

I recall paragraph2Vec. It was the earliest way to experiment with position embeddings that I recall reading. At first when I read it, it felt kind of crazy/weird that it worked: You feed in the paragraph ID(yes just an integer of the para where the word was) via its own layer along with the word to CBOW/SkipGram set up. Then throw away that part after training. Then, during inference, you attach a new randmly initialized layer for where that old layer was and "re-train" just that part before generating the embedding for words.