5 ms·

I wonder why it was chosen to represent everything, well, not unit vectors, but nothing higher than 1. Why should 2 and 4 both be [1 0 ...] instead of [2 0 ...

I wonder why it was chosen to represent everything, well, not unit vectors, but nothing higher than 1. Why should 2 and 4 both be [1 0 ...] instead of [2 0 ...], etc?

jerpint 4y ago
It might have to do with UMAP making dot products and assumptions about the inputs. If everything is 0s and 1s, the vectors will have a normal distribution of magnitudes (more or less). Otherwise the magnitudes will just explode and I don’t think UMAP will work.
- nerdponx 4y ago
  It's also very common in general to use this "one-hot encoding" in statistics and machine learning. In many cases using all 2s or all -10s would produce the exact same result in theory, but with more work by the optimizing algorithm, possibly with adverse results as described above. It's easy to reason about mathematically, too. If the input vector is all 1s and 0s, it's easy to read off the result of multiplying that vector with another vector. Norms of binary vectors and dot products between binary vectors are super-easy, and have a nice correspondence with counting the appearances of elements. It also corresponds to an array of Boolean values which is conceptually appealing, because that's basically how it's constructed. With a couple of specific exceptions, there's little reason not to use all 1s.
  - vanderZwan 4y ago
    That kind of sounds like normalisation in other contexts ("hey let's just say lightspeed is 1, simplifies the equations"), is that a fair analogy to make?
    - nerdponx 4y ago
      Normalization of that kind is also used in machine learning and statistics, but it's not quite the same thing. A related technique is "standardization", where are you subtract the mean and divide by the standard deviation, yielding unitless quantities of "standard deviations away from the mean". However this encoding technique is a bit more like choosing 1 and 0 to represent Boolean values in C: it's convenient, it's easy to reason about, it's mathematically simpler than any other option, and there's no compelling reason to choose anything else anyway. For binary variables in particular, you sometimes see people using -1 and 1 instead of 0 and 1, to get symmetry around 0. I think this was mostly only used for encoding the labels/outputs of SVM models, where it's mathematically appealing as representing two sides of a hyperplane. There are a handful of other schemes for encoding "categorical" or "nominal" data of this kind, used in certain statistical applications such as the design and analysis of experiments. These encoding schemes are called contrasts in the stats literature, because they emphasize the differences (the "constrasts") between categories.

I wonder why it was chosen to represent everything, well, not unit vectors, but nothing higher than 1. Why should 2 and 4 both be [1 0 ...] instead of [2 0 ...], etc?

jerpint 4y ago

It might have to do with UMAP making dot products and assumptions about the inputs. If everything is 0s and 1s, the vectors will have a normal distribution of magnitudes (more or less). Otherwise the magnitudes will just explode and I don’t think UMAP will work.

nerdponx 4y ago
It's also very common in general to use this "one-hot encoding" in statistics and machine learning. In many cases using all 2s or all -10s would produce the exact same result in theory, but with more work by the optimizing algorithm, possibly with adverse results as described above. It's easy to reason about mathematically, too. If the input vector is all 1s and 0s, it's easy to read off the result of multiplying that vector with another vector. Norms of binary vectors and dot products between binary vectors are super-easy, and have a nice correspondence with counting the appearances of elements. It also corresponds to an array of Boolean values which is conceptually appealing, because that's basically how it's constructed. With a couple of specific exceptions, there's little reason not to use all 1s.
- vanderZwan 4y ago
  That kind of sounds like normalisation in other contexts ("hey let's just say lightspeed is 1, simplifies the equations"), is that a fair analogy to make?
  - nerdponx 4y ago
    Normalization of that kind is also used in machine learning and statistics, but it's not quite the same thing. A related technique is "standardization", where are you subtract the mean and divide by the standard deviation, yielding unitless quantities of "standard deviations away from the mean". However this encoding technique is a bit more like choosing 1 and 0 to represent Boolean values in C: it's convenient, it's easy to reason about, it's mathematically simpler than any other option, and there's no compelling reason to choose anything else anyway. For binary variables in particular, you sometimes see people using -1 and 1 instead of 0 and 1, to get symmetry around 0. I think this was mostly only used for encoding the labels/outputs of SVM models, where it's mathematically appealing as representing two sides of a hyperplane. There are a handful of other schemes for encoding "categorical" or "nominal" data of this kind, used in certain statistical applications such as the design and analysis of experiments. These encoding schemes are called contrasts in the stats literature, because they emphasize the differences (the "constrasts") between categories.