3 ms·

Thanks for the comment, very insightful. As for LLM embeddings being inferior: the quality of embeddings really is dependent on the usecase. Embedding is a los

Thanks for the comment, very insightful. As for LLM embeddings being inferior: the quality of embeddings really is dependent on the usecase. Embedding is a lossy form of compression. In more than half the cases I've seen, a 32-dimension GLoVe embedding preforms just as accurately as the 4000 dimension LLM embeddings some people default to using. The 32d embedding performs better even, if you consider system performance (throughput, costs, etc.)

zwaps 2y ago
Exactly, because Glove embeddings generate directly a linear space. This is, if I remember the paper, explicitly their objective. This corresponds to a single symmetrical relationship between two tokens, so a rather simple network embeddable into few dimensions without much loss. And it has been shown time and time again that Glove embeddings work really well. The underlying relational structure is linear, as mentioned, so they are for example amendable to linear operations, aggregation etc. By contrast, LLM embeddings are quite lossy as they somehow need to be created from an underlying non linear space. For instance to represent a sentence from its tokens, it does not work ti just sum or mean. You need to train an extra aggregation model on top with new data that builds a good embedding space from the LLM activtions. However, there is a limit to what Glove ans Word2Vec can represent. Simple things like polysemic words are already a limit etc. On a sequence level, especially for longer passages, LLM based sentence embeddings give you much more information if you meed it (insofar that relational semantics of a bag of words approach would not be sufficient)