4 ms·

Tangential: How do you guys decide which similarity metric to use? Euclidean seems the most intuitive but I'm sure others also have use cases. This situation is

Tangential: How do you guys decide which similarity metric to use? Euclidean seems the most intuitive but I'm sure others also have use cases. This situation is similar to the similarity metrics used to compare two distributions (KL for insurance, tho technically it's not a metric). Is there some art to it or is there really a systematic way to choose metrics for different tasks?

ashvardanian 3y ago
My intuition: L2 in low dims, Cosine in high dims. Hamming on similar size sets, Jaccard on others. Jensen-Shannon (symmetric KL) when dealing with histograms and probability distribution.
- SubiculumCode 3y ago
  Manhattan is better than L2 at high dimensions..but why cos?
  - SubiculumCode 3y ago
    https://stats.stackexchange.com/questions/341535/curse-of-dimensionality-does-cosine-similarity-work-better-and-if-so-why https://stats.stackexchange.com/questions/341535/curse-of-di...
    - godelski 3y ago
      Cosine similarity is certainly not immune to the curse of dimensionality. In fact, it is explicitly prone to it. As dimensions increase, the likelihood that any two tensors are orthogonal rapidly increases. This is easy to reason out if you start from 2 dimensions, count the number of orthogonal vectors, and then move to 3D, and so on.
godelski 3y ago
This is actually a really underappreciated question! (and also really interesting!) There's a lot of nuance to this because the truth is that distance becomes less meaningful as you increase dimensions. You can find some papers comparing Lp distances with different p values (p=2 == Euclidean == L2). But as dimension increases, the distance to the furthest points decreases (making it harder to differentiate near points from far points). Cosine similarity is a commonly used one, but as dimensions increase the likelihood that any two tensors are orthogonal rapidly increases. This might seem counterintuitive because the probability is really low in 2 or 3 dims as you only have 2 or a plane in R3. So really the answer honestly tends to be ad hoc: "whatever works best". It's good to keep in mind that any intuition you have about geometry goes out the window as dimensions increase. It's always important to remember assumptions made, especially when focusing on empiricism. There are definitely some nuances that can point you in better directions (pun intended :) than random guessing, especially if you know a lot about your geometry, but it is messy and nuances can make big differences. I wish I had a better answer but I hope this is informative. Maybe some mathematician will show up and add more. I'm sure there's someone on HN that loves to talk about higher dimensional geometry and I'd love to hear those "rants."
- sroussey 3y ago
  Yeah, I’m working on a test framework to try the various permutations to see what works. And for me to get a feel for it.
- Loic 3y ago
  "whatever works best" because it also totally depends on your field of application. Here[0] is for example a similarity search using the Tanimoto[1] index. The "works best" is also, in many cases, subjective. This is also not easy to assess, you may need several people looking at the results, here molecules, to say if yes they are similar or not. A chemist will think differently than a biologist in this regard. [0]: https://www.chemeo.com/similar?smiles=Cc1c%28%5bN%2b%5d%28%3dO%29%5bO-%5d%29cc%28%5bN%2b%5d%28%3dO%29%5bO-%5d%29cc1%5bN%2b%5d%28%3dO%29%5bO-%5d https://www.chemeo.com/similar?smiles=Cc1c%28%5bN%2b%5d%28%3... [1]: https://jcheminf.biomedcentral.com/articles/10.1186/s13321-015-0069-3 https://jcheminf.biomedcentral.com/articles/10.1186/s13321-0...
  - tudorw 3y ago
    Not sure how relevant this is as not really my field, but possibly interesting! 'A feature selection method based on the Golden Jackal-Grey Wolf Hybrid Optimization Algorithm' https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0295579 https://journals.plos.org/plosone/article?id=10.1371/journal...

ashvardanian 3y ago

My intuition: L2 in low dims, Cosine in high dims. Hamming on similar size sets, Jaccard on others. Jensen-Shannon (symmetric KL) when dealing with histograms and probability distribution.

SubiculumCode 3y ago
Manhattan is better than L2 at high dimensions..but why cos?
- SubiculumCode 3y ago
  https://stats.stackexchange.com/questions/341535/curse-of-dimensionality-does-cosine-similarity-work-better-and-if-so-why https://stats.stackexchange.com/questions/341535/curse-of-di...
  - godelski 3y ago
    Cosine similarity is certainly not immune to the curse of dimensionality. In fact, it is explicitly prone to it. As dimensions increase, the likelihood that any two tensors are orthogonal rapidly increases. This is easy to reason out if you start from 2 dimensions, count the number of orthogonal vectors, and then move to 3D, and so on.

godelski 3y ago

This is actually a really underappreciated question! (and also really interesting!) There's a lot of nuance to this because the truth is that distance becomes less meaningful as you increase dimensions. You can find some papers comparing Lp distances with different p values (p=2 == Euclidean == L2). But as dimension increases, the distance to the furthest points decreases (making it harder to differentiate near points from far points). Cosine similarity is a commonly used one, but as dimensions increase the likelihood that any two tensors are orthogonal rapidly increases. This might seem counterintuitive because the probability is really low in 2 or 3 dims as you only have 2 or a plane in R3. So really the answer honestly tends to be ad hoc: "whatever works best". It's good to keep in mind that any intuition you have about geometry goes out the window as dimensions increase. It's always important to remember assumptions made, especially when focusing on empiricism. There are definitely some nuances that can point you in better directions (pun intended :) than random guessing, especially if you know a lot about your geometry, but it is messy and nuances can make big differences. I wish I had a better answer but I hope this is informative. Maybe some mathematician will show up and add more. I'm sure there's someone on HN that loves to talk about higher dimensional geometry and I'd love to hear those "rants."

sroussey 3y ago
Yeah, I’m working on a test framework to try the various permutations to see what works. And for me to get a feel for it.
Loic 3y ago
"whatever works best" because it also totally depends on your field of application. Here[0] is for example a similarity search using the Tanimoto[1] index. The "works best" is also, in many cases, subjective. This is also not easy to assess, you may need several people looking at the results, here molecules, to say if yes they are similar or not. A chemist will think differently than a biologist in this regard. [0]: https://www.chemeo.com/similar?smiles=Cc1c%28%5bN%2b%5d%28%3dO%29%5bO-%5d%29cc%28%5bN%2b%5d%28%3dO%29%5bO-%5d%29cc1%5bN%2b%5d%28%3dO%29%5bO-%5d https://www.chemeo.com/similar?smiles=Cc1c%28%5bN%2b%5d%28%3... [1]: https://jcheminf.biomedcentral.com/articles/10.1186/s13321-015-0069-3 https://jcheminf.biomedcentral.com/articles/10.1186/s13321-0...
- tudorw 3y ago
  Not sure how relevant this is as not really my field, but possibly interesting! 'A feature selection method based on the Golden Jackal-Grey Wolf Hybrid Optimization Algorithm' https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0295579 https://journals.plos.org/plosone/article?id=10.1371/journal...