This looks really good and will add to my reading list. Bear with me on this question as I am still learning... From a quick skim I couldn't find it. It might

This looks really good and will add to my reading list. Bear with me on this question as I am still learning... From a quick skim I couldn't find it. It might be there? But with a transformer, at which layer do you consider the vector to be the embedding? Or are there multiple choices? For example you have a positional and token embedding - so they are of course embeddings, but simple ones. King - Man + Woman = Queen type thing. But I imagine you might want to consider the output of the 1st ? 2nd? Nth? transformer on the decoder side as an embedding of sorts. For example the sentence "attention is all you" has gone through a couple of layers you now have a vector that understands those words pretty well. But if you go too far through I imagine you get closer to the "prediction" and so lose the meaning and just get something closer to "need". But if possible to get, that sentence embedding (by which I mean N tokens up to context size) might be useful for search indexing of long text.