4 ms·
So the question is: do embeddings form a linear space? I.e. does scaling and addition make any sense?
by waynecochran 3y ago
So the question is: do embeddings form a linear space? I.e. does scaling and addition make any sense?
- Der_Einzige 3y agoYes, that’s what the glove paper showed. Linear substructure in PCA space.
- RockyMcNuts 3y agodoes that apply for ada-002 embeddings, if you don't use GloVe? I would think it only applies if you create embeddings using a linear model?
- deleted 3y ago[deleted]
- nightski 3y agoI'm not a mathematician but on their site it says - "Cosine similarity and Euclidean distance will result in the identical rankings" so I am assuming that would have to be the case?
- yorwba 3y agoWhen you have unit vectors, cosine similarity and euclidean distance always result in identical rankings, even when those unit vectors are assigned at random and have no semantic structure at all. That's because the euclidean distance |a - b|² = ⟨a - b, a - b⟩ = ⟨a, a⟩ - 2⟨a, b⟩ + ⟨b, b⟩ = |a|² - 2|a||b|cos ɸ + |b|² for unit vectors with |a| = |b| = 1 only depends on the cosine of the angle ɸ between the two vectors.
- kridsdale3 3y agoPythagoras strikes again.
- binarymax 3y agoNot always, another comment mentions it is true for GloVe, but that doesn't mean it is true for all model's vector spaces.
- samus 3y agoIn a "good" embedding, they do.
- tonyarkles 3y agoWhen I was first reading about Word2vec I thought it was absolutely wonderful that the vector delta (king - queen) was similar to the vector delta (man - woman). That captures relationships in such a fascinating way!
- esafak 3y agoIt is not obvious to me why word2vec's training objective yields this. word2vec ensures similarity of related words, but why can you then perform linear algebra on unrelated words?
- blackbear_ 3y agoThe training objective does not have much to do with it, the bigger reason is that these neural networks themselves use linear algebra (dot products/projections and whatnot) to manipulate embeddings.