3 ms·
I tend to think of one-hot encoding as not a vector space But it is a vector space. Technically the one-hot elements are the basis of the vector space, whose
by karma_pharmer 2y ago
I tend to think of one-hot encoding as not a vector space
But it is a vector space. Technically the one-hot elements are the basis of the vector space, whose elements are weighted lists of words.
And it is the vector space with the highest useful dimension for embedding that vocabulary. Seriously, what are you going to do with the extra axes if you're representing a 10,000-word vocabulary using 1,000,000-element vectors? Those extra axes simply go to waste. In the one-hot case the only thing wasted is the magnitudes.
This is why word2vec embeddings are considered lower-dimensional embeddings. They're lower-dimension than the most-naive-but-not-wasteful representation.
- epistasis 2y agoBut when does anyone actually use a one-hot encoding as a vector space? They are used as a lookup table into a different representation, and the one-hit encoding is there to simplify mathematical description, not to be used as a vector space. Or have you experienced systems where people use one-hit encodings? While I would agree that technically, in terms of raw dimensionality, a one-hit encoding is larger dimension than the embedding, that would be a "lower" dimensionality, not "low" dimensionality. Who uses "low" as an absolute term when referring to 300-2000 dimensions?