Very interesting. I was not aware, for example, of the embedding vector lookup. The transformers I have worked with typically used a simple one-hot token repres

Very interesting. I was not aware, for example, of the embedding vector lookup. The transformers I have worked with typically used a simple one-hot token representation, but they were domain-specific and not trained on natural language. How are these embeddings trained?