4 ms·

Distance measures are only as good as the Pseudo-Riemannian metric they (implicitly)implement. If the manifold hypothesis is believed, then these metrics should

Distance measures are only as good as the Pseudo-Riemannian metric they (implicitly)implement. If the manifold hypothesis is believed, then these metrics should be local because the manifold curvature is a local property. You would be mistaken to use an ordinary dot product to compare straight lines on a map of the globe, because those lines aren’t actually straight - they do not account for the rich information in the curvature tensor. Using the wrong inner product is akin to the flat-Earth fallacy.

ashvardanian 3y ago
That sounds like a good point. Any research you know about extracting the curvature tensor from the a transformer-like model?
trhway 3y ago
We implicitly train for minimizing of the distance in that style of metric - by using functions continuous and differentiable on classic manifolds (where continuity and differentiability is defined using the classic local maps into Euclidian space). I think if we were training using functions continuous and differentiable in say p-adic metric space (which looks extremely jagged/fractallian/non-continuos when embedded into Euclidian) then we'd have something like p-adic version of cosine (or other L-something metric) for similarity
groceryheist 3y ago
This is exactly right and is one (among many) reasons that reliance on cosine similarities in my field (computational social science) is so problematic. The curvature of the manifold must be accounted for in measuring distances. Other measures based on optimal transport are more theoretically sound, but are computationally expensive.
seanhunter 3y ago
I'm not sure I understand the underlying maths well enough to opine on your point but I can say for certain that no embedding space that I've ever seen used for any kind of ML is uniform in the sense that a Euclidian distance around one point means the same thing as the same Euclidian distance around another point. I'm not even sure that it would be possible to make an embedding that was uniform in that way because it would mean that we had a universal measure of similarity between concepts (which can obviously be enormously different). The other potential issue is for all the embeddings that I have seen the resulting space once you have embedded some documents is sort of "clumpy" and very sparse overall. So you have very large areas with basically nothing at all I think because semantically there are many dimensions which only make sense for subsets of concepts so you end up with big voids where really the embedding space is totally unreachable so distance doesn't have any meaning at all. In spite of all that there are a few similarity measures which work well enough to be useful for many practical purposes and cosine similarity is one of them. I don't think anyone thinks it's perfect.

ashvardanian 3y ago

That sounds like a good point. Any research you know about extracting the curvature tensor from the a transformer-like model?

trhway 3y ago

We implicitly train for minimizing of the distance in that style of metric - by using functions continuous and differentiable on classic manifolds (where continuity and differentiability is defined using the classic local maps into Euclidian space). I think if we were training using functions continuous and differentiable in say p-adic metric space (which looks extremely jagged/fractallian/non-continuos when embedded into Euclidian) then we'd have something like p-adic version of cosine (or other L-something metric) for similarity

groceryheist 3y ago

This is exactly right and is one (among many) reasons that reliance on cosine similarities in my field (computational social science) is so problematic. The curvature of the manifold must be accounted for in measuring distances. Other measures based on optimal transport are more theoretically sound, but are computationally expensive.

seanhunter 3y ago

I'm not sure I understand the underlying maths well enough to opine on your point but I can say for certain that no embedding space that I've ever seen used for any kind of ML is uniform in the sense that a Euclidian distance around one point means the same thing as the same Euclidian distance around another point. I'm not even sure that it would be possible to make an embedding that was uniform in that way because it would mean that we had a universal measure of similarity between concepts (which can obviously be enormously different). The other potential issue is for all the embeddings that I have seen the resulting space once you have embedded some documents is sort of "clumpy" and very sparse overall. So you have very large areas with basically nothing at all I think because semantically there are many dimensions which only make sense for subsets of concepts so you end up with big voids where really the embedding space is totally unreachable so distance doesn't have any meaning at all. In spite of all that there are a few similarity measures which work well enough to be useful for many practical purposes and cosine similarity is one of them. I don't think anyone thinks it's perfect.