9 ms·

Is Cosine-Similarity of Embeddings Really About Similarity?

210 points by Jimmc414 3y ago

latency-guy2 3y ago
Not reading the paper, cosine similarity has little to no semantic understanding of sentences. E.g. the following triple 1: "Yes, this is a demonstration" 2: "Yes, this isn't a demonstration" 3: "Here is an example" <1, 2>, Has "higher" cosine similarity than <1, 3>, structurally equivalent except for one token/word, <1, 2> semantically means the opposite of each other depending on what you're targeting in that sentence. While <1, 3> means effectively the same thing. If this paper is about persuading people about efficacy with regards to semantic understanding, OK, but that was always known. If its about something with relation to vectors and the underlying operations, then I'll be interested.
- soarerz 3y ago
  What is the cheapest way to capture similarity if not via dot product then?
  - deleted 3y ago
    [deleted]
  - gajus 3y ago
    Interested to know as well
  - latency-guy2 3y ago
    I don't have an answer for this really outside of silly ones like "strict equality check", but I assert that no one else does either, at least today and right now, and its an inherent limitation due to the nature of embeddings and the space it desires to be (cheap, fast, good enough similarity for your use case). You're probably best off using the commercial suggestion, and if its dot product, go for it. I am no expert in this area and my interest wanes every day.
  - scotty79 3y ago
    Instead of sums of multiplications you could for example use sum of squares of differences. Means squared error instead of dot product, it's not cheaper but it's close If you want to go cheaper you could use sum of abs of differences.
    - soarerz 3y ago
      This is effectively "the same" as dot product. For a lot of embeddings we have today, norm of any embedding vector is roughly of same size, so the angle between two vectors is roughly same size as length of difference that you are saying, and can be expressed in terms of 1 - dot product after scaling
- Tostino 3y ago
  That is entirely dependant on the model for the embeddings. You can fine tune for pretty much any outcome you want.
  - _t89y 3y ago
    You can't fine-tune for understanding or reasoning. You can't "get better performance" on understanding. You're either equipped for it or you're not.
- superkuh 3y ago
  That might be true for one-hot vectors but it's not true for learned embedding through the lens of attention. That said, I only made to page 3/9 of the paper before the mark-up for the math went over my head.
  - _t89y 3y ago
    It is true. And if you want to say anything about meaning this isn't even the right math.
  - latency-guy2 3y ago
    If we're talking about adding dimensionality and relying on the kernel, sure, but I only get so much incorrect by going from 1 to M dims. I can't of course be certain in all cases, but dimensions are typically (past experience, and using knowledge from word2vec experiments from years ago) derivative of higher dimensions. The kernel still operates on the same concept by applying a norm along with whatever weightings to each dim. Semantic understanding is still not there in my opinion, we might feign it by increasing specificity, but only so much. Largest contributor will likely still the determining factor rather than the series of smaller, more specific dimensions. I tested this using similar sentences as my original comment and failing in more scenarios than passing. I of course am biased since it may be given I did not select the right dimensions or measures.
- vidarh 3y ago
  Whether your not the cosine similarity of either pair is higher depends on the mapping you create from the strings to the embedding vector. That mapping can be whichever function you choose, and your result will be entirely dependent on that. If you choose a straight linear mapping of tokens to a number, then you'd be right. Extending that, if you choose any mapping which does not do a more extensive remapping from raw syntactic structure to some sort of semantic representation, you'd be right. But hence why we increasingly use models to create embeddings instead of simpler approaches before applying a similarity metric, whether cosine similarity or other. Put another way, there is no inherent reason why you couldn't have a model where the embeddings for 1 and 3 are identical even, and so it is meaningless to talk about the cosine similarity of your sentences without setting out your assumptions about how you will created embeddings from them.
  - _t89y 3y ago
    It is meaningless to talk about cosine similarity of sentences, or words, at all. Choose whatever mapping you want. You'll still be in Firth Mode.
    - _t89y 3y ago
      Uh oh. LOL. Got some angry Firthers out there.
    - vidarh 3y ago
      It's meaningful to talk about cosine similarity for anything that you can quantify in ways such that the cosine similarity reflects a measure you care about. Same applies for any function. If it works, it's meaningful to talk about it whether or not it has a reasonable interpretation beyond that.
  - latency-guy2 3y ago
    > meaningless to talk about the cosine similarity of your sentences without setting out your assumptions about how you will created embeddings from them. I agree, but from generics POV, you have to settle on a few things to compare between models. If you can't, then benchmarks are useless too outside of extremely narrow measures. I only address structure in the parent, and sure, it can be too generic of a statement by only touching on structure. But I would almost assert structure is still an important feature, and I would almost assert that it is required or otherwise a dominant feature when you want to deliver a product for general use. I don't think I get too much more incorrect going beyond a few dimensions given this.
- deleted 3y ago
  [deleted]
- _t89y 3y ago
  No understanding. Embeddings are a semantically vacuous representation and similarity is a semantically vacuous interpretation.
neoncontrails 3y ago
> In the following, we show that [taking cosine similarity between two features in a learned embedding] can lead to arbitrary results, and they may not even be unique. Was uniqueness ever a guarantee? It's a distance metric. It's reasonable to assume that two features can be equidistant to the ideal solution to a linear system of equations. Maybe I'm missing something.
- pletnes 3y ago
  I sure hope noone claimed that. You’re doing potentially huge dimensionality reduction, uniqueness would be like saying you cannot have md5 collisions.
  - JohnKemeny 3y ago
    Johnson and Lindenstrauss disagree. https://en.wikipedia.org/wiki/Johnson%E2%80%93Lindenstrauss_lemma https://en.wikipedia.org/wiki/Johnson%E2%80%93Lindenstrauss_...
    - yorwba 3y ago
      If you have 1000 points and want to preserve their squared distances to within an error of 1%, the Johnson-Lindenstrauss construction suggests an embedding dimension of 8(ln 1000)/(0.01²) > 552620. If your points start out in a lower-dimensional space than that to begin with, it's obviously pointless. The crossover point where the number of dimensions falls below the number of points is at 1113868. If you're willing to tolerate 10% error, it's at 7094.
- nerdponx 3y ago
  It's not even a distance metric, it doesn't obey the triangle inequality (hence the not-technically-meaningful name "similarity", like "collection" as opposed to "set").
  - deleted 3y ago
    [deleted]
SubiculumCode 3y ago
Distance metrics are an interesting topic. The field of ecology has a ton of them. For example see vegdist the Dissimilarity Indices for Community Ecologists function in the Vegan package in R: https://rdrr.io/cran/vegan/man/vegdist.html https://rdrr.io/cran/vegan/man/vegdist.html which includes, among others the "canberra", "clark", "bray", "kulczynski", "gower", "altGower", "morisita", "horn", "mountford", "raup", "chao", "cao", "mahalanobis", "chord", "hellinger", "aitchison", or "robust.aitchison". Generic distance metrics can often be replaced with context-specific ones for better utility; it makes me wonder whether that insight could be useful in deep learning.
- hackerlight 3y ago
  What are good distance metrics applied to latent embeddings as part of a diversity loss function to prevent model collapse?
  - SubiculumCode 3y ago
    hell if I know!! Sorry. I've used the vegan package for some analyses, but I've mostly used Manhattan and cosine metrics. I just wanted to bring up the idea that there are a lot of metrics out there that may not be generally appreciated. Claude Opus says "There are a few good distance metrics commonly used with latent embeddings to promote diversity and prevent model collapse: 1. Euclidean Distance (L2 Distance) 2. Cosine distance 3. Kullback-Leibler (KL) Divergence: KL divergence quantifies how much one probability distribution differs from another. It can be used to measure the difference between the distributions of latent embeddings. Minimizing KL divergence as a diversity loss would encourage the embedding distribution to be more uniform. 4. Maximum Mean Discrepancy (MMD): MMD measures the difference between two distributions by comparing their moments (mean, variance, etc.) in a reproducing kernel Hilbert space. It's useful for comparing high-dimensional distributions like those of embeddings. MMD loss promotes diversity by penalizing embeddings that are too clustered together. 5. Gaussian Annulus Loss: This loss function encourages embeddings to lie within an annulus (ring) in the latent space defined by two Gaussian distributions. It promotes uniformity in the embedding norms while allowing angular diversity. This can be effective at preventing collapse to a single point. But I haven't checked for hallucinations.
LifeIsBio 3y ago
The paper kinda leaves you hanging on the "alternatives" front, even though they have a section dedicated to it. In addition to the _quality_ of any proposed alternative(s), computational speed also has to be a consideration. I've run into multiple situations where you want to measure similarities on the order of millions/billions of times. Especially for realtime applications (like RAG?) speed may even out weight quality.
_t89y 3y ago
It's definitely not about semantics or language. As far as language is concerned similarity metrics are semantically vacuous and quantifying semantic similarity is a bogus enterprise.
- blackbear_ 3y ago
  Can you elaborate?
  - _t89y 3y ago
    Modeling language in a latent space is useful for certain kinds of analyses and certain aspects of language. It has its place as an empirical tool. That place is not the nuts and bolts of language itself. There are more suitable formalisms for this than directional magnitudes and BPE tiktokens.
    - blackbear_ 3y ago
      What are those formalisms?
    - throwaway2562 3y ago
      Intuitively I agree with you, despite the unexpected ‘success’ of current approaches. What formalisms do you suggest?
      - _t89y 3y ago
        The Lambek calculus. Categorial grammars. Meanings are proofs. Not clusters of directional magnitudes in space.
        jll29 3y ago
        Curiously, the upcoming third edition of Jurafsky and Martin [1], one of the two standard text books for NLP, places Context-Free Grammars, Combinatory Categorial Grammars, and logical meaning representations in its appendices on the companion Web site, no longer in the text book itself. Unthinkable only a few years ago. [1] https://web.stanford.edu/~jurafsky/slp3/ https://web.stanford.edu/~jurafsky/slp3/
greesil 3y ago
It's about keeping stuff on the unit hypersphere
mo_42 3y ago
I quickly read through the paper. One thing to note is that they use the Frobenius norm (at least I suppose this from the index F) for the matrix factorization. That is for their learning algorithm. Then, they use the cosine-similarity to evaluate. A metric that wasn't used in the algorithm. This is a long-standing question for me. Theoretically, I should use the CS in my optimization and then also in the evaluation. But I haven't tested this empirically. For example, there is sperical K-meams that clusters the data on the unit sphere.
- deleted 3y ago
  [deleted]
- nerdponx 3y ago
  I think that's kind of the point of the paper. The model is based on un-normalized dot products, and wasn't deliberately designed to produce meaningful cosine similarities. They are showing that, in that case, cosine similarities might be arbitrary and not as useful as people might assume or hope.
visarga 3y ago
Embeddings result from computing what word can appear in a given context, so words that would appear in the same spot will have higher cosine score between themselves. But it doesn't differentiate further, so you can have "beautiful" and "ugly" embed very close to each other even though they are opposites - they tend to appear in similar places. Another limitation of embeddings and cosine-similarity is that that they can't tell you "how similar" - is it equivalence or just relatedness? They make a mess of equivalent, antonymous and related things.
- mo_42 3y ago
  Only if those two words appear in the same contexts with the same frequency. In natural language this is probably not the case. There are things typically considered beautiful and others as ugly.
  - singularity2001 3y ago
    one key insights is that opposites do have a lot in common, they are often opposites in exactly one of n feature dimensions. For example black and white are both (arguably) colors, are related to (lack of) light, have representations in various formats (RGB, …), appear in the same grammatical position of ordered adjectives …
- _t89y 3y ago
  They make a mess of language. They are not a suitable representation. They are suitable for their efficiency in information retrieval systems and for sometimes crudely capturing semantic attributes in a way that is unreliable and uninterpretable. It ends there. Here's to ten more years of word2vec.
- minimaxir 3y ago
  For word2vec-esque word embeddings, yes. For modern embedding models which effectively mean-pool the last hidden state of LLMs (and therefore make use of its optimizations such as attention tricks), embeddings can be much more robust to different contexts both local and global.
h_koko 3y ago
What are better alternatives to cosine similarity?
- minimaxir 3y ago
  There aren't any alternatives: cosine similarity is effectively an extension of Euclidian distance, which is the mathematical correct way for finding the distance between vectors. You may not want to use cosine similarity as your only metric for rankings, however, and you may want to experiment with how you construct the embeddings.
  - rocqua 3y ago
    Euclidean distance is only 'mathematically correct' if you care about rotations in your space being distance preserving. I don't think that is really the case in these spaces. There is a whole branch of mathematics called dedicated to other ways to measure distance.
    - 0xdeadbeefbabe 3y ago
      How wouldn't distances be preserved under rotation. I'm asking for a friend of course.
      - rocqua 3y ago
        In the Manhattan distance for example if you rotate your coordinate grid, distances will change.
- _t89y 3y ago
  For describing semantics in natural language? Pretty much anything else.
  - singularity2001 3y ago
    like?
apstroll 3y ago
Cosine Similarity is very much about similarity, but it's quite fickle and indirect. Given a function f(l, r) that measures, say, the logprobability of observing both l and r, and that the function takes the form f(l, r) = <L(l), R(r)>, i.e. the dot product between embeddings of l and r, then cosine similarity of x and y, i.e. normalized dot product of L(x) and L(y) is very closely related to the correlation of f(x, Z) and f(y, Z) when we let Z vary.
sweezyjeezy 3y ago
> While cosine similarity is invariant under such rotations R, one of the key insights in this paper is that the first (but not the second) objective is also invariant to rescalings of the columns of A and B Ha interesting I wrote a blog post where I pointed this out a few years ago [1], and how we got around it for item-item similarity at an old job (essentially an implicit re-projection to original space as noted in section 3). https://swarbrickjones.wordpress.com/2016/11/24/note-on-an-item-item-recommender-system-using-matrix-factorisation/ https://swarbrickjones.wordpress.com/2016/11/24/note-on-an-i...
jsn_5 3y ago
All models are wrong, some are useful.
mikewarot 3y ago
Why would anyone expect cosine-similarity to be a useful metric? In the real word, the arbitrary absolute position of an object in the universe (if it could be measured) isn't that important, it's the directions and distances to nearby objects that matters most. It's my understanding that the delta between two word embeddings, gives a direction, and the magic is from using those directions to get to new words. The oft cited example is King-Man+Woman = Queen [1] When did this view fall from favor? [1] https://www.technologyreview.com/2015/09/17/166211/king-man-woman-queen-the-marvelous-mathematics-of-computational-linguistics/ https://www.technologyreview.com/2015/09/17/166211/king-man-...
- necroforest 3y ago
  cosine similarity is (isomorphic to) "distances to nearby objects". and not all embeddings are word embeddings.
  - VHRanger 3y ago
    It's isomorphic when vectors are normalized, otherwise it's angle distance, not position distance
    - jameshart 3y ago
      It’s a mistake to think of vectors as coordinates of objects in space, though. You can visualize them like that, but that’s not what they are. The vectors are the objects. A vector is just a list of n numbers. Embedded into a n dimensional space, a vector is a distance in a direction. It isn’t ’the point you get to by going that distance in that direction from the origin of that space’. You don’t need as space to have an origin for the embedding to make sense - for ‘cosine similarity’ to make sense. Cosine similarity is just ‘how similar is the direction these vectors point in’. The geometric intuition of ‘angle between’ actually does a disservice here when we are talking about high dimensional vectors. We’re talking about things that are much more similar to functions than spatial vectors, and while you can readily talk about the ‘normalized dot product’ of two functions it’s much less reasonable to talk about the ‘cosine similarity’ between them - it just turns out that mathematically those are equivalent.
montebicyclelo 3y ago
Hmm, typically the models where people use cosine similarity on embeddings have been deliberately trained such that the cosine similarity is meaningful. It looks like this paper is looking at examples where the models have not been deliberately trained for the cosine similarities, and hence in these situations it would indeed be unreasonable to assume cosine similarities to be a good idea.. (but that's kind of a given?) For example, here's the loss from the CLIP paper [1], which ensures cosine similarities are meaningful: # joint multimodal embedding [n, d_e] I_e = l2_normalize(np.dot(I_f, W_i), axis=1) T_e = l2_normalize(np.dot(T_f, W_t), axis=1) # scaled pairwise cosine similarities [n, n] logits = np.dot(I_e, T_e.T) * np.exp(t) # symmetric loss function labels = np.arange(n) loss_i = cross_entropy_loss(logits, labels, axis=0) loss_t = cross_entropy_loss(logits, labels, axis=1) loss = (loss_i + loss_t)/2 And Sentence Transformers [2] using CosineSimilarityLoss: train_loss = losses.CosineSimilarityLoss(model) # Tune the model model.fit(train_objectives=[(train_dataloader, train_loss)], epochs=1, warmup_steps=100) [1] https://arxiv.org/pdf/2103.00020.pdf https://arxiv.org/pdf/2103.00020.pdf [2] https://www.sbert.net/docs/training/overview.html https://www.sbert.net/docs/training/overview.html
- 0xdeadbeefbabe 3y ago
  Right, the embedding is contrived to give euclidean distance and cosine similarity the usual meaning.
- malshe 3y ago
  Thanks for these examples.
shiandow 3y ago
A well kept secret of linear algebra is that having an inner product at all isn't as self-evident as it might seem. Euclidean distance might seem like some canonical notion of distance, but it doesn't have to be meaningful, especially if the choice of coordinates has no geometrical meaning.
- 0xdeadbeefbabe 3y ago
  It seems easiest to blame the embedding and not the cosine similarity.
toisanji 3y ago
We found similar results when working on our paper for creating LLM agents with metacognition and explicitly called this out in the paper: https://replicantlife.com https://replicantlife.com
cfgauss2718 3y ago
Distance measures are only as good as the Pseudo-Riemannian metric they (implicitly)implement. If the manifold hypothesis is believed, then these metrics should be local because the manifold curvature is a local property. You would be mistaken to use an ordinary dot product to compare straight lines on a map of the globe, because those lines aren’t actually straight - they do not account for the rich information in the curvature tensor. Using the wrong inner product is akin to the flat-Earth fallacy.
- ashvardanian 3y ago
  That sounds like a good point. Any research you know about extracting the curvature tensor from the a transformer-like model?
- trhway 3y ago
  We implicitly train for minimizing of the distance in that style of metric - by using functions continuous and differentiable on classic manifolds (where continuity and differentiability is defined using the classic local maps into Euclidian space). I think if we were training using functions continuous and differentiable in say p-adic metric space (which looks extremely jagged/fractallian/non-continuos when embedded into Euclidian) then we'd have something like p-adic version of cosine (or other L-something metric) for similarity
- groceryheist 3y ago
  This is exactly right and is one (among many) reasons that reliance on cosine similarities in my field (computational social science) is so problematic. The curvature of the manifold must be accounted for in measuring distances. Other measures based on optimal transport are more theoretically sound, but are computationally expensive.
- seanhunter 3y ago
  I'm not sure I understand the underlying maths well enough to opine on your point but I can say for certain that no embedding space that I've ever seen used for any kind of ML is uniform in the sense that a Euclidian distance around one point means the same thing as the same Euclidian distance around another point. I'm not even sure that it would be possible to make an embedding that was uniform in that way because it would mean that we had a universal measure of similarity between concepts (which can obviously be enormously different). The other potential issue is for all the embeddings that I have seen the resulting space once you have embedded some documents is sort of "clumpy" and very sparse overall. So you have very large areas with basically nothing at all I think because semantically there are many dimensions which only make sense for subsets of concepts so you end up with big voids where really the embedding space is totally unreachable so distance doesn't have any meaning at all. In spite of all that there are a few similarity measures which work well enough to be useful for many practical purposes and cosine similarity is one of them. I don't think anyone thinks it's perfect.
yantrarora 3y ago
[dead]

210 points by Jimmc414 3y ago

Not reading the paper, cosine similarity has little to no semantic understanding of sentences. E.g. the following triple 1: "Yes, this is a demonstration" 2: "Yes, this isn't a demonstration" 3: "Here is an example" <1, 2>, Has "higher" cosine similarity than <1, 3>, structurally equivalent except for one token/word, <1, 2> semantically means the opposite of each other depending on what you're targeting in that sentence. While <1, 3> means effectively the same thing. If this paper is about persuading people about efficacy with regards to semantic understanding, OK, but that was always known. If its about something with relation to vectors and the underlying operations, then I'll be interested.

soarerz 3y ago
What is the cheapest way to capture similarity if not via dot product then?
- deleted 3y ago
  [deleted]
- gajus 3y ago
  Interested to know as well
- latency-guy2 3y ago
  I don't have an answer for this really outside of silly ones like "strict equality check", but I assert that no one else does either, at least today and right now, and its an inherent limitation due to the nature of embeddings and the space it desires to be (cheap, fast, good enough similarity for your use case). You're probably best off using the commercial suggestion, and if its dot product, go for it. I am no expert in this area and my interest wanes every day.
- scotty79 3y ago
  Instead of sums of multiplications you could for example use sum of squares of differences. Means squared error instead of dot product, it's not cheaper but it's close If you want to go cheaper you could use sum of abs of differences.
  - soarerz 3y ago
    This is effectively "the same" as dot product. For a lot of embeddings we have today, norm of any embedding vector is roughly of same size, so the angle between two vectors is roughly same size as length of difference that you are saying, and can be expressed in terms of 1 - dot product after scaling
Tostino 3y ago
That is entirely dependant on the model for the embeddings. You can fine tune for pretty much any outcome you want.
- _t89y 3y ago
  You can't fine-tune for understanding or reasoning. You can't "get better performance" on understanding. You're either equipped for it or you're not.
superkuh 3y ago
That might be true for one-hot vectors but it's not true for learned embedding through the lens of attention. That said, I only made to page 3/9 of the paper before the mark-up for the math went over my head.
- _t89y 3y ago
  It is true. And if you want to say anything about meaning this isn't even the right math.
- latency-guy2 3y ago
  If we're talking about adding dimensionality and relying on the kernel, sure, but I only get so much incorrect by going from 1 to M dims. I can't of course be certain in all cases, but dimensions are typically (past experience, and using knowledge from word2vec experiments from years ago) derivative of higher dimensions. The kernel still operates on the same concept by applying a norm along with whatever weightings to each dim. Semantic understanding is still not there in my opinion, we might feign it by increasing specificity, but only so much. Largest contributor will likely still the determining factor rather than the series of smaller, more specific dimensions. I tested this using similar sentences as my original comment and failing in more scenarios than passing. I of course am biased since it may be given I did not select the right dimensions or measures.
vidarh 3y ago
Whether your not the cosine similarity of either pair is higher depends on the mapping you create from the strings to the embedding vector. That mapping can be whichever function you choose, and your result will be entirely dependent on that. If you choose a straight linear mapping of tokens to a number, then you'd be right. Extending that, if you choose any mapping which does not do a more extensive remapping from raw syntactic structure to some sort of semantic representation, you'd be right. But hence why we increasingly use models to create embeddings instead of simpler approaches before applying a similarity metric, whether cosine similarity or other. Put another way, there is no inherent reason why you couldn't have a model where the embeddings for 1 and 3 are identical even, and so it is meaningless to talk about the cosine similarity of your sentences without setting out your assumptions about how you will created embeddings from them.
- _t89y 3y ago
  It is meaningless to talk about cosine similarity of sentences, or words, at all. Choose whatever mapping you want. You'll still be in Firth Mode.
  - _t89y 3y ago
    Uh oh. LOL. Got some angry Firthers out there.
  - vidarh 3y ago
    It's meaningful to talk about cosine similarity for anything that you can quantify in ways such that the cosine similarity reflects a measure you care about. Same applies for any function. If it works, it's meaningful to talk about it whether or not it has a reasonable interpretation beyond that.
- latency-guy2 3y ago
  > meaningless to talk about the cosine similarity of your sentences without setting out your assumptions about how you will created embeddings from them. I agree, but from generics POV, you have to settle on a few things to compare between models. If you can't, then benchmarks are useless too outside of extremely narrow measures. I only address structure in the parent, and sure, it can be too generic of a statement by only touching on structure. But I would almost assert structure is still an important feature, and I would almost assert that it is required or otherwise a dominant feature when you want to deliver a product for general use. I don't think I get too much more incorrect going beyond a few dimensions given this.
deleted 3y ago
[deleted]
_t89y 3y ago
No understanding. Embeddings are a semantically vacuous representation and similarity is a semantically vacuous interpretation.

neoncontrails 3y ago

> In the following, we show that [taking cosine similarity between two features in a learned embedding] can lead to arbitrary results, and they may not even be unique. Was uniqueness ever a guarantee? It's a distance metric. It's reasonable to assume that two features can be equidistant to the ideal solution to a linear system of equations. Maybe I'm missing something.

pletnes 3y ago
I sure hope noone claimed that. You’re doing potentially huge dimensionality reduction, uniqueness would be like saying you cannot have md5 collisions.
- JohnKemeny 3y ago
  Johnson and Lindenstrauss disagree. https://en.wikipedia.org/wiki/Johnson%E2%80%93Lindenstrauss_lemma https://en.wikipedia.org/wiki/Johnson%E2%80%93Lindenstrauss_...
  - yorwba 3y ago
    If you have 1000 points and want to preserve their squared distances to within an error of 1%, the Johnson-Lindenstrauss construction suggests an embedding dimension of 8(ln 1000)/(0.01²) > 552620. If your points start out in a lower-dimensional space than that to begin with, it's obviously pointless. The crossover point where the number of dimensions falls below the number of points is at 1113868. If you're willing to tolerate 10% error, it's at 7094.
nerdponx 3y ago
It's not even a distance metric, it doesn't obey the triangle inequality (hence the not-technically-meaningful name "similarity", like "collection" as opposed to "set").
- deleted 3y ago
  [deleted]

SubiculumCode 3y ago

Distance metrics are an interesting topic. The field of ecology has a ton of them. For example see vegdist the Dissimilarity Indices for Community Ecologists function in the Vegan package in R: https://rdrr.io/cran/vegan/man/vegdist.html https://rdrr.io/cran/vegan/man/vegdist.html which includes, among others the "canberra", "clark", "bray", "kulczynski", "gower", "altGower", "morisita", "horn", "mountford", "raup", "chao", "cao", "mahalanobis", "chord", "hellinger", "aitchison", or "robust.aitchison". Generic distance metrics can often be replaced with context-specific ones for better utility; it makes me wonder whether that insight could be useful in deep learning.

hackerlight 3y ago
What are good distance metrics applied to latent embeddings as part of a diversity loss function to prevent model collapse?
- SubiculumCode 3y ago
  hell if I know!! Sorry. I've used the vegan package for some analyses, but I've mostly used Manhattan and cosine metrics. I just wanted to bring up the idea that there are a lot of metrics out there that may not be generally appreciated. Claude Opus says "There are a few good distance metrics commonly used with latent embeddings to promote diversity and prevent model collapse: 1. Euclidean Distance (L2 Distance) 2. Cosine distance 3. Kullback-Leibler (KL) Divergence: KL divergence quantifies how much one probability distribution differs from another. It can be used to measure the difference between the distributions of latent embeddings. Minimizing KL divergence as a diversity loss would encourage the embedding distribution to be more uniform. 4. Maximum Mean Discrepancy (MMD): MMD measures the difference between two distributions by comparing their moments (mean, variance, etc.) in a reproducing kernel Hilbert space. It's useful for comparing high-dimensional distributions like those of embeddings. MMD loss promotes diversity by penalizing embeddings that are too clustered together. 5. Gaussian Annulus Loss: This loss function encourages embeddings to lie within an annulus (ring) in the latent space defined by two Gaussian distributions. It promotes uniformity in the embedding norms while allowing angular diversity. This can be effective at preventing collapse to a single point. But I haven't checked for hallucinations.

LifeIsBio 3y ago

The paper kinda leaves you hanging on the "alternatives" front, even though they have a section dedicated to it. In addition to the _quality_ of any proposed alternative(s), computational speed also has to be a consideration. I've run into multiple situations where you want to measure similarities on the order of millions/billions of times. Especially for realtime applications (like RAG?) speed may even out weight quality.

_t89y 3y ago

It's definitely not about semantics or language. As far as language is concerned similarity metrics are semantically vacuous and quantifying semantic similarity is a bogus enterprise.

blackbear_ 3y ago
Can you elaborate?
- _t89y 3y ago
  Modeling language in a latent space is useful for certain kinds of analyses and certain aspects of language. It has its place as an empirical tool. That place is not the nuts and bolts of language itself. There are more suitable formalisms for this than directional magnitudes and BPE tiktokens.
  - blackbear_ 3y ago
    What are those formalisms?
  - throwaway2562 3y ago
    Intuitively I agree with you, despite the unexpected ‘success’ of current approaches. What formalisms do you suggest?
    - _t89y 3y ago
      The Lambek calculus. Categorial grammars. Meanings are proofs. Not clusters of directional magnitudes in space.
      - jll29 3y ago
        Curiously, the upcoming third edition of Jurafsky and Martin [1], one of the two standard text books for NLP, places Context-Free Grammars, Combinatory Categorial Grammars, and logical meaning representations in its appendices on the companion Web site, no longer in the text book itself. Unthinkable only a few years ago. [1] https://web.stanford.edu/~jurafsky/slp3/ https://web.stanford.edu/~jurafsky/slp3/

greesil 3y ago

It's about keeping stuff on the unit hypersphere

mo_42 3y ago

I quickly read through the paper. One thing to note is that they use the Frobenius norm (at least I suppose this from the index F) for the matrix factorization. That is for their learning algorithm. Then, they use the cosine-similarity to evaluate. A metric that wasn't used in the algorithm. This is a long-standing question for me. Theoretically, I should use the CS in my optimization and then also in the evaluation. But I haven't tested this empirically. For example, there is sperical K-meams that clusters the data on the unit sphere.

deleted 3y ago
[deleted]
nerdponx 3y ago
I think that's kind of the point of the paper. The model is based on un-normalized dot products, and wasn't deliberately designed to produce meaningful cosine similarities. They are showing that, in that case, cosine similarities might be arbitrary and not as useful as people might assume or hope.

visarga 3y ago

Embeddings result from computing what word can appear in a given context, so words that would appear in the same spot will have higher cosine score between themselves. But it doesn't differentiate further, so you can have "beautiful" and "ugly" embed very close to each other even though they are opposites - they tend to appear in similar places. Another limitation of embeddings and cosine-similarity is that that they can't tell you "how similar" - is it equivalence or just relatedness? They make a mess of equivalent, antonymous and related things.

mo_42 3y ago
Only if those two words appear in the same contexts with the same frequency. In natural language this is probably not the case. There are things typically considered beautiful and others as ugly.
- singularity2001 3y ago
  one key insights is that opposites do have a lot in common, they are often opposites in exactly one of n feature dimensions. For example black and white are both (arguably) colors, are related to (lack of) light, have representations in various formats (RGB, …), appear in the same grammatical position of ordered adjectives …
_t89y 3y ago
They make a mess of language. They are not a suitable representation. They are suitable for their efficiency in information retrieval systems and for sometimes crudely capturing semantic attributes in a way that is unreliable and uninterpretable. It ends there. Here's to ten more years of word2vec.
minimaxir 3y ago
For word2vec-esque word embeddings, yes. For modern embedding models which effectively mean-pool the last hidden state of LLMs (and therefore make use of its optimizations such as attention tricks), embeddings can be much more robust to different contexts both local and global.

h_koko 3y ago

What are better alternatives to cosine similarity?

minimaxir 3y ago
There aren't any alternatives: cosine similarity is effectively an extension of Euclidian distance, which is the mathematical correct way for finding the distance between vectors. You may not want to use cosine similarity as your only metric for rankings, however, and you may want to experiment with how you construct the embeddings.
- rocqua 3y ago
  Euclidean distance is only 'mathematically correct' if you care about rotations in your space being distance preserving. I don't think that is really the case in these spaces. There is a whole branch of mathematics called dedicated to other ways to measure distance.
  - 0xdeadbeefbabe 3y ago
    How wouldn't distances be preserved under rotation. I'm asking for a friend of course.
    - rocqua 3y ago
      In the Manhattan distance for example if you rotate your coordinate grid, distances will change.
_t89y 3y ago
For describing semantics in natural language? Pretty much anything else.
- singularity2001 3y ago
  like?

apstroll 3y ago

Cosine Similarity is very much about similarity, but it's quite fickle and indirect. Given a function f(l, r) that measures, say, the logprobability of observing both l and r, and that the function takes the form f(l, r) = <L(l), R(r)>, i.e. the dot product between embeddings of l and r, then cosine similarity of x and y, i.e. normalized dot product of L(x) and L(y) is very closely related to the correlation of f(x, Z) and f(y, Z) when we let Z vary.

sweezyjeezy 3y ago

> While cosine similarity is invariant under such rotations R, one of the key insights in this paper is that the first (but not the second) objective is also invariant to rescalings of the columns of A and B Ha interesting I wrote a blog post where I pointed this out a few years ago [1], and how we got around it for item-item similarity at an old job (essentially an implicit re-projection to original space as noted in section 3). https://swarbrickjones.wordpress.com/2016/11/24/note-on-an-item-item-recommender-system-using-matrix-factorisation/ https://swarbrickjones.wordpress.com/2016/11/24/note-on-an-i...

jsn_5 3y ago

All models are wrong, some are useful.

mikewarot 3y ago

Why would anyone expect cosine-similarity to be a useful metric? In the real word, the arbitrary absolute position of an object in the universe (if it could be measured) isn't that important, it's the directions and distances to nearby objects that matters most. It's my understanding that the delta between two word embeddings, gives a direction, and the magic is from using those directions to get to new words. The oft cited example is King-Man+Woman = Queen [1] When did this view fall from favor? [1] https://www.technologyreview.com/2015/09/17/166211/king-man-woman-queen-the-marvelous-mathematics-of-computational-linguistics/ https://www.technologyreview.com/2015/09/17/166211/king-man-...

necroforest 3y ago
cosine similarity is (isomorphic to) "distances to nearby objects". and not all embeddings are word embeddings.
- VHRanger 3y ago
  It's isomorphic when vectors are normalized, otherwise it's angle distance, not position distance
  - jameshart 3y ago
    It’s a mistake to think of vectors as coordinates of objects in space, though. You can visualize them like that, but that’s not what they are. The vectors are the objects. A vector is just a list of n numbers. Embedded into a n dimensional space, a vector is a distance in a direction. It isn’t ’the point you get to by going that distance in that direction from the origin of that space’. You don’t need as space to have an origin for the embedding to make sense - for ‘cosine similarity’ to make sense. Cosine similarity is just ‘how similar is the direction these vectors point in’. The geometric intuition of ‘angle between’ actually does a disservice here when we are talking about high dimensional vectors. We’re talking about things that are much more similar to functions than spatial vectors, and while you can readily talk about the ‘normalized dot product’ of two functions it’s much less reasonable to talk about the ‘cosine similarity’ between them - it just turns out that mathematically those are equivalent.

montebicyclelo 3y ago

Hmm, typically the models where people use cosine similarity on embeddings have been deliberately trained such that the cosine similarity is meaningful. It looks like this paper is looking at examples where the models have not been deliberately trained for the cosine similarities, and hence in these situations it would indeed be unreasonable to assume cosine similarities to be a good idea.. (but that's kind of a given?) For example, here's the loss from the CLIP paper [1], which ensures cosine similarities are meaningful: # joint multimodal embedding [n, d_e] I_e = l2_normalize(np.dot(I_f, W_i), axis=1) T_e = l2_normalize(np.dot(T_f, W_t), axis=1) # scaled pairwise cosine similarities [n, n] logits = np.dot(I_e, T_e.T) * np.exp(t) # symmetric loss function labels = np.arange(n) loss_i = cross_entropy_loss(logits, labels, axis=0) loss_t = cross_entropy_loss(logits, labels, axis=1) loss = (loss_i + loss_t)/2 And Sentence Transformers [2] using CosineSimilarityLoss: train_loss = losses.CosineSimilarityLoss(model) # Tune the model model.fit(train_objectives=[(train_dataloader, train_loss)], epochs=1, warmup_steps=100) [1] https://arxiv.org/pdf/2103.00020.pdf https://arxiv.org/pdf/2103.00020.pdf [2] https://www.sbert.net/docs/training/overview.html https://www.sbert.net/docs/training/overview.html

0xdeadbeefbabe 3y ago
Right, the embedding is contrived to give euclidean distance and cosine similarity the usual meaning.
malshe 3y ago
Thanks for these examples.

shiandow 3y ago

A well kept secret of linear algebra is that having an inner product at all isn't as self-evident as it might seem. Euclidean distance might seem like some canonical notion of distance, but it doesn't have to be meaningful, especially if the choice of coordinates has no geometrical meaning.

0xdeadbeefbabe 3y ago
It seems easiest to blame the embedding and not the cosine similarity.

toisanji 3y ago

We found similar results when working on our paper for creating LLM agents with metacognition and explicitly called this out in the paper: https://replicantlife.com https://replicantlife.com

cfgauss2718 3y ago

Distance measures are only as good as the Pseudo-Riemannian metric they (implicitly)implement. If the manifold hypothesis is believed, then these metrics should be local because the manifold curvature is a local property. You would be mistaken to use an ordinary dot product to compare straight lines on a map of the globe, because those lines aren’t actually straight - they do not account for the rich information in the curvature tensor. Using the wrong inner product is akin to the flat-Earth fallacy.

ashvardanian 3y ago
That sounds like a good point. Any research you know about extracting the curvature tensor from the a transformer-like model?
trhway 3y ago
We implicitly train for minimizing of the distance in that style of metric - by using functions continuous and differentiable on classic manifolds (where continuity and differentiability is defined using the classic local maps into Euclidian space). I think if we were training using functions continuous and differentiable in say p-adic metric space (which looks extremely jagged/fractallian/non-continuos when embedded into Euclidian) then we'd have something like p-adic version of cosine (or other L-something metric) for similarity
groceryheist 3y ago
This is exactly right and is one (among many) reasons that reliance on cosine similarities in my field (computational social science) is so problematic. The curvature of the manifold must be accounted for in measuring distances. Other measures based on optimal transport are more theoretically sound, but are computationally expensive.
seanhunter 3y ago
I'm not sure I understand the underlying maths well enough to opine on your point but I can say for certain that no embedding space that I've ever seen used for any kind of ML is uniform in the sense that a Euclidian distance around one point means the same thing as the same Euclidian distance around another point. I'm not even sure that it would be possible to make an embedding that was uniform in that way because it would mean that we had a universal measure of similarity between concepts (which can obviously be enormously different). The other potential issue is for all the embeddings that I have seen the resulting space once you have embedded some documents is sort of "clumpy" and very sparse overall. So you have very large areas with basically nothing at all I think because semantically there are many dimensions which only make sense for subsets of concepts so you end up with big voids where really the embedding space is totally unreachable so distance doesn't have any meaning at all. In spite of all that there are a few similarity measures which work well enough to be useful for many practical purposes and cosine similarity is one of them. I don't think anyone thinks it's perfect.

yantrarora 3y ago

[dead]