5 ms·

Excellent article. I especially liked the explanation of Kissing Numbers and was surprised to learn how few dimensions we've found that have them defined. Addit

Excellent article. I especially liked the explanation of Kissing Numbers and was surprised to learn how few dimensions we've found that have them defined. Additionally, I think sometimes the people in some contexts that shake their fists at high-dimensional data forget that having lots of dimensions is not always inherently a curse. In my perspective, calling it the Curse of Dimensionality in an neural network context is to me sort of like calling it the Curse of the Storage Shed. Oh no, all of this gestures hands mathematical space in this place to put my stuff orthogonally, whatever shall I do. Oh, woe is me. That's not the Borsuk–Ulam theorem, hiding over there, that's just a fake shrub! I should never take a look at that. Oh, woe is me indeed. (extremely thickly-laid facetious remark concluded)

Der_Einzige 3y ago
Well it's definitely a curse for anyone that wants their embeddings/vectors to be meaningful outside of the most nearest of neighbors.
- tysam_and 3y ago
  I'm unfortunately less-experienced in embeddings than I'd want to be for this discussion. Something I do find interesting is that lots of people use a scaled cosine distance for vectors (the dot product) that is much more tolerant to high dimensions during training, and then a Euclidean distance metric which is not as friendly towards high dimensions during inference. I wish I had more classical statistical experience to dive into the nittier and grittier of the details here with appropriate aplomb.
  - extasia 3y ago
    I thought that they use the dot product during training[0] and then the cosine similarity during inference. Sometimes the vectors are pre-normalized for inference (thus reducing cosine sim. to just the dot product), is this what you're thinking of perhaps? [0]. Not at my desktop, but I read this in the Stanford NLP course in the past two weeks, specifically the section on the skip gram algorithm (word 2 vec)
    - IIAOPSW 3y ago
      if your vectors are normalized, dot and cosine are the same thing.
      - messe 3y ago
        The person you're replying to also said that.
    - tysam_and 3y ago
      That's why I said the scaled cosine distance, the dot product between two vectors is a scaled version of the cosine distance. This is practically every neural network layer that uses a MAD (multiply-add) computational motif, including 3x3s. Surprisingly few people know this. I believe you might have the reduction order a bit backwards -- the dot product reduces to cosine similarity and not the other way around as cosine similarity is the normalized dot product.
      - extasia 3y ago
        You're correct! It's dot product reduces to cosine sim, not the other way around. Could you clarify your second statement about Mad and 3x3s? Afraid I don't follow.
        tysam_and 3y ago
        Ah, of course. Sure. I can do that. Any operation in neural networks that multiplies two vectors together and then collects them with a single addition operation is a dot product. So 3x3 convs are technically a dot product, even though they have a spatial dimension. Same for the initial part of most transformers' attention layers, MLP layers, etc.... The overlap can be a bit tricky to deal with in terms of implications, but I've found it a very helpful formulation for squeezing out some performance boosts in previous implementations of neural networks that I've worked on. Hope this helps, feel free to let me know if you have any more questions/thoughts/etc, love! <3 :)) :D :fireworkds:
- extasia 3y ago
  Could you give an example of an embedding space that is only meaningful locally? The word vector idea of modifying a word by some operation e.g King->Queen ~= man->woman ~= uncle->aunt Which intuitively seems non-local to me.
  - jerf 3y ago
    Toss that intuition out. That's in some sense the major problem with higher dimensional spaces; our intuition about locality is built in a super-small-dimensional space. Compare the locality of King, man, and uncle to King, lemma, and titrate. The former set is quite local.
  - tysam_and 3y ago
    The parent commenter in this case is referencing the tendency for high dimensional spaces to very quickly go "off the rails" in that, if I understand correctly, most smooth spaces are locally flat, but as you get further and further away, the curvature can really screw with things. Even with that fixed, Euclidean and otherwise measures do not translate intuitively to high dimensions outside of much more local spaces.

Der_Einzige 3y ago

Well it's definitely a curse for anyone that wants their embeddings/vectors to be meaningful outside of the most nearest of neighbors.

tysam_and 3y ago
I'm unfortunately less-experienced in embeddings than I'd want to be for this discussion. Something I do find interesting is that lots of people use a scaled cosine distance for vectors (the dot product) that is much more tolerant to high dimensions during training, and then a Euclidean distance metric which is not as friendly towards high dimensions during inference. I wish I had more classical statistical experience to dive into the nittier and grittier of the details here with appropriate aplomb.
- extasia 3y ago
  I thought that they use the dot product during training[0] and then the cosine similarity during inference. Sometimes the vectors are pre-normalized for inference (thus reducing cosine sim. to just the dot product), is this what you're thinking of perhaps? [0]. Not at my desktop, but I read this in the Stanford NLP course in the past two weeks, specifically the section on the skip gram algorithm (word 2 vec)
  - IIAOPSW 3y ago
    if your vectors are normalized, dot and cosine are the same thing.
    - messe 3y ago
      The person you're replying to also said that.
  - tysam_and 3y ago
    That's why I said the scaled cosine distance, the dot product between two vectors is a scaled version of the cosine distance. This is practically every neural network layer that uses a MAD (multiply-add) computational motif, including 3x3s. Surprisingly few people know this. I believe you might have the reduction order a bit backwards -- the dot product reduces to cosine similarity and not the other way around as cosine similarity is the normalized dot product.
    - extasia 3y ago
      You're correct! It's dot product reduces to cosine sim, not the other way around. Could you clarify your second statement about Mad and 3x3s? Afraid I don't follow.
      - tysam_and 3y ago
        Ah, of course. Sure. I can do that. Any operation in neural networks that multiplies two vectors together and then collects them with a single addition operation is a dot product. So 3x3 convs are technically a dot product, even though they have a spatial dimension. Same for the initial part of most transformers' attention layers, MLP layers, etc.... The overlap can be a bit tricky to deal with in terms of implications, but I've found it a very helpful formulation for squeezing out some performance boosts in previous implementations of neural networks that I've worked on. Hope this helps, feel free to let me know if you have any more questions/thoughts/etc, love! <3 :)) :D :fireworkds:
extasia 3y ago
Could you give an example of an embedding space that is only meaningful locally? The word vector idea of modifying a word by some operation e.g King->Queen ~= man->woman ~= uncle->aunt Which intuitively seems non-local to me.
- jerf 3y ago
  Toss that intuition out. That's in some sense the major problem with higher dimensional spaces; our intuition about locality is built in a super-small-dimensional space. Compare the locality of King, man, and uncle to King, lemma, and titrate. The former set is quite local.
- tysam_and 3y ago
  The parent commenter in this case is referencing the tendency for high dimensional spaces to very quickly go "off the rails" in that, if I understand correctly, most smooth spaces are locally flat, but as you get further and further away, the curvature can really screw with things. Even with that fixed, Euclidean and otherwise measures do not translate intuitively to high dimensions outside of much more local spaces.