5 ms·

“Embedding” —> representation(?) I do not think that word means what I think it means.

“Embedding” —> representation(?) I do not think that word means what *I* think it means.

epistasis 2y ago
That is essentially correct. You take an object and "embed" it in a high-dimensional vector space to represent it. For a deep dive, I highly recommend Vicki Boykis's free materials: https://vickiboykis.com/what_are_embeddings/ https://vickiboykis.com/what_are_embeddings/
- coreyp_1 2y ago
  Quite frankly, it's stuff like this that makes me love the HN community. Thank you for the additional resource!
- mercurybee 2y ago
  It's more common to refer to embeddings as low-dimensional.
  - epistasis 2y ago
    Can you give an example of that? I've rarely seen embeddings with fewer than hundreds of dimensions. UMAP/T-SNE are dimensional reduction techniques that could maybe considered embeddings, but I haven't encountered that in anything that relates to word2vec or LLMs or much of the current AI fashion.
    - nmfisher 2y ago
      If your vocabulary size is 10000, then that's also your initial one-hot "dimension". Projecting down to 512 floats (or whatever) is, relatively speaking, low dimension.
      - epistasis 2y ago
        Ah, I see, I understand but I haven't encountered that terminology used that way. I tend to think of one-hot encoding as not a vector space, however, as the information is actually only lg2(10000) per token. However the embedding vector of 512 or however many floats includes lots of information from positionally related tokens, so it's quite a bit more than lg2(vocab).
        karma_pharmer 2y ago
        I tend to think of one-hot encoding as not a vector space But it is a vector space. Technically the one-hot elements are the basis of the vector space, whose elements are weighted lists of words. And it is the vector space with the highest useful dimension for embedding that vocabulary. Seriously, what are you going to do with the extra axes if you're representing a 10,000-word vocabulary using 1,000,000-element vectors? Those extra axes simply go to waste. In the one-hot case the only thing wasted is the magnitudes. This is why word2vec embeddings are considered lower-dimensional embeddings. They're lower-dimension than the most-naive-but-not-wasteful representation.
        epistasis 2y ago
        But when does anyone actually use a one-hot encoding as a vector space? They are used as a lookup table into a different representation, and the one-hit encoding is there to simplify mathematical description, not to be used as a vector space. Or have you experienced systems where people use one-hit encodings? While I would agree that technically, in terms of raw dimensionality, a one-hit encoding is larger dimension than the embedding, that would be a "lower" dimensionality, not "low" dimensionality. Who uses "low" as an absolute term when referring to 300-2000 dimensions?
        srean 2y ago
        Thought of letting this one go, but then changed my mind. > I tend to think > I havent encountered Egocentric views of things are far less notable and far less interesting than what those things actually are. It makes for tiring reading. Hope it does not come off as being too testy, my apologies if it does.
DISCURSIVE 2y ago
Yeah, we usually just say vector embeddings are the numerical representation of that piece of unstructured data. This glossary page put it together quite nicely. https://zilliz.com/glossary/vector-embeddings https://zilliz.com/glossary/vector-embeddings

“Embedding” —> representation(?) I do not think that word means what *I* think it means.

That is essentially correct. You take an object and "embed" it in a high-dimensional vector space to represent it. For a deep dive, I highly recommend Vicki Boykis's free materials: https://vickiboykis.com/what_are_embeddings/ https://vickiboykis.com/what_are_embeddings/

coreyp_1 2y ago
Quite frankly, it's stuff like this that makes me love the HN community. Thank you for the additional resource!
mercurybee 2y ago
It's more common to refer to embeddings as low-dimensional.
- epistasis 2y ago
  Can you give an example of that? I've rarely seen embeddings with fewer than hundreds of dimensions. UMAP/T-SNE are dimensional reduction techniques that could maybe considered embeddings, but I haven't encountered that in anything that relates to word2vec or LLMs or much of the current AI fashion.
  - nmfisher 2y ago
    If your vocabulary size is 10000, then that's also your initial one-hot "dimension". Projecting down to 512 floats (or whatever) is, relatively speaking, low dimension.
    - epistasis 2y ago
      Ah, I see, I understand but I haven't encountered that terminology used that way. I tend to think of one-hot encoding as not a vector space, however, as the information is actually only lg2(10000) per token. However the embedding vector of 512 or however many floats includes lots of information from positionally related tokens, so it's quite a bit more than lg2(vocab).
      - karma_pharmer 2y ago
        I tend to think of one-hot encoding as not a vector space But it is a vector space. Technically the one-hot elements are the basis of the vector space, whose elements are weighted lists of words. And it is the vector space with the highest useful dimension for embedding that vocabulary. Seriously, what are you going to do with the extra axes if you're representing a 10,000-word vocabulary using 1,000,000-element vectors? Those extra axes simply go to waste. In the one-hot case the only thing wasted is the magnitudes. This is why word2vec embeddings are considered lower-dimensional embeddings. They're lower-dimension than the most-naive-but-not-wasteful representation.
        epistasis 2y ago
        But when does anyone actually use a one-hot encoding as a vector space? They are used as a lookup table into a different representation, and the one-hit encoding is there to simplify mathematical description, not to be used as a vector space. Or have you experienced systems where people use one-hit encodings? While I would agree that technically, in terms of raw dimensionality, a one-hit encoding is larger dimension than the embedding, that would be a "lower" dimensionality, not "low" dimensionality. Who uses "low" as an absolute term when referring to 300-2000 dimensions?
      - srean 2y ago
        Thought of letting this one go, but then changed my mind. > I tend to think > I havent encountered Egocentric views of things are far less notable and far less interesting than what those things actually are. It makes for tiring reading. Hope it does not come off as being too testy, my apologies if it does.

DISCURSIVE 2y ago

Yeah, we usually just say vector embeddings are the numerical representation of that piece of unstructured data. This glossary page put it together quite nicely. https://zilliz.com/glossary/vector-embeddings https://zilliz.com/glossary/vector-embeddings