I read this comment yesterday and keep thinking about it. That final token really must "comprehend" everything leading up to it, right? In which case longer con

I read this comment yesterday and keep thinking about it. That final token really must "comprehend" everything leading up to it, right? In which case longer context lengths are just trying to pack more meaning into that embedding state. Which means the embedding model must do a lot of the lifting to be able to accurately represent meaning across long contexts so well. Now I want to know more about how those models are derived.