6 ms·

Those would really just be identifiers. I think the key property of embeddings is that the dimensions each individually mean/measure something, and therefore th

Those would really just be identifiers. I think the key property of embeddings is that the dimensions each individually mean/measure something, and therefore the dot product of two embeddings (similarity of direction of the vectors) is a meaningful similarity measure of the things being represented. The classic example is word embeddings such as word2vec, or GloVE, where due to the embeddings being meaningful in this way, one can see vector relationships such as "man - woman" = "king - queen".

IanCal 3y ago
They're not, I get why you think that though. They're making a vector for a text that's the term frequencies in the document. It's one step simpler than tfidf which is a great starting point.
- OmarShehata 3y ago
  Are you saying it's pure chance that operations like "man - woman" = "king - queen" (and many, many other similar relationships and analogies) work? If not please explain this comment to those of us ignorant in these matters :)
  - StrangeDoctor 3y ago
    It’s not pure chance that the above calculus shakes out, but it doesn’t have to be that way. If you are embedding on a word by word level then it can happen, if it’s a little smaller or larger than word by word it’s not immediately clear what the calculation is doing. But the main difference here is you get 1 embedding for the document in question, not an embedding per word like word2vec. So it’s something more like “document about OS/2 warp” - “wiki page for ibm” + “wiki page for Microsoft” = “document on windows 3.1”
  - napoleongl 3y ago
    3Blue1Brown has some other examples in his videos about transformers, most notably I think is that hitler-Germany+italy ~ Mussolini! https://www.3blue1brown.com/lessons/gpt https://www.3blue1brown.com/lessons/gpt
- HarHarVeryFunny 3y ago
  OK, sounds counter-intuitive, but I'll take your word for it! It seems odd since the basis of word similarity captured in this type of way is that word meanings are associated with local context, which doesn't seem related to these global occurrence counts. Perhaps it works because two words with similar occurrence counts are more likely to often appear close to each other than two words where one has a high count, and another a small count? But this wouldn't seem to work for small counts, and anyways the counts are just being added to the base index rather than making similar-count words closer in the embedding space. Do you have any explanation for why this captures any similarity in meaning?
  - IanCal 3y ago
    > rather than making similar-count words closer in the embedding space. Ah I think I see the confusion here. They are describing creating an embedding of a document or piece of text. At the base, the embedding of a single word would just be a single 1. There is absolutely no help with word similarity. The problem of multiple meanings isn't solved by this approach at all, at least not directly. Talking about the "gravity of a situation" in a political piece makes the text a bit more similar to physics discussions about gravity. But most of the words won't match as well, so your document vector is still more similar to other political pieces than physics. Going up the scale, here's a few basic starting points that were (are?) the backbone of many production text AI/ML systems. 1. Bag of words. Here your vector has a 1 for words that are present, and 0 for ones that aren't. 2. Bag of words with a count. A little better, now we've got the information that you said "gravity" fifty times not once. Normalise it so text length doesn't matter and everything fits into 0-1. 3. TF-IDF. It's not very useful to know that you said a common word a lot. Most texts do, what we care about is ones that say it more than you'd expect so we take into account how often the words appear in the entire corpus. These don't help with words, but given how simple they are they are shockingly useful. They have their stupid moments, although one benefit is that it's very easy to debug why they cause a problem.
thisiszilff 3y ago
> I think the key property of embeddings is that the dimensions each individually mean/measure something, and therefore the dot product of two embeddings (similarity of direction of the vectors) is a meaningful similarity measure of the things being represented. In this case each dimension is the presence of a word in a particular text. So when you take the dot product of two texts you are effectively counting the number of words the two texts have in common (subject to some normalization constants depending on how you normalize the embedding). Cosine similarity still works for even these super naive embeddings which makes it slightly easier to understand before getting into any mathy stuff. You are 100% right this won't give you the word embedding analogies like king - man = queen or stuff like that. This embedding has no concept of relationships between words.
- HarHarVeryFunny 3y ago
  But that doesn't seem to be what you are describing in terms of using incrementing indices and adding occurrence counts. If you want to create a bag of words text embedding then you set the number of embedding dimensions to the vocabulary size and the value of each dimension to the global count of the corresponding word.
  - thisiszilff 3y ago
    Heh -- my explanation isn't the clearest I realize, but yes, it is BoW. Eg fix your vocab of 50k words (or whatever) and enumerate it. Then to make an embedding for some piece of text 1. initialize an all zero vector of size 50k 2. for each word in the text, add one to the index of the corresponding word (per our enumeration). If the word isn't in the 50k words in your vocabulary, then discard it 3. (optionally), normalize the embedding to 1 (though you don't really need this and can leave it off for the toy example). initialize an embedding (for a single text) as an all zero vector of size 50k
    - p1esk 3y ago
      This is not the best way to understand where modern embeddings are coming from.

IanCal 3y ago

They're not, I get why you think that though. They're making a vector for a text that's the term frequencies in the document. It's one step simpler than tfidf which is a great starting point.

OmarShehata 3y ago
Are you saying it's pure chance that operations like "man - woman" = "king - queen" (and many, many other similar relationships and analogies) work? If not please explain this comment to those of us ignorant in these matters :)
- StrangeDoctor 3y ago
  It’s not pure chance that the above calculus shakes out, but it doesn’t have to be that way. If you are embedding on a word by word level then it can happen, if it’s a little smaller or larger than word by word it’s not immediately clear what the calculation is doing. But the main difference here is you get 1 embedding for the document in question, not an embedding per word like word2vec. So it’s something more like “document about OS/2 warp” - “wiki page for ibm” + “wiki page for Microsoft” = “document on windows 3.1”
- napoleongl 3y ago
  3Blue1Brown has some other examples in his videos about transformers, most notably I think is that hitler-Germany+italy ~ Mussolini! https://www.3blue1brown.com/lessons/gpt https://www.3blue1brown.com/lessons/gpt
HarHarVeryFunny 3y ago
OK, sounds counter-intuitive, but I'll take your word for it! It seems odd since the basis of word similarity captured in this type of way is that word meanings are associated with local context, which doesn't seem related to these global occurrence counts. Perhaps it works because two words with similar occurrence counts are more likely to often appear close to each other than two words where one has a high count, and another a small count? But this wouldn't seem to work for small counts, and anyways the counts are just being added to the base index rather than making similar-count words closer in the embedding space. Do you have any explanation for why this captures any similarity in meaning?
- IanCal 3y ago
  > rather than making similar-count words closer in the embedding space. Ah I think I see the confusion here. They are describing creating an embedding of a document or piece of text. At the base, the embedding of a single word would just be a single 1. There is absolutely no help with word similarity. The problem of multiple meanings isn't solved by this approach at all, at least not directly. Talking about the "gravity of a situation" in a political piece makes the text a bit more similar to physics discussions about gravity. But most of the words won't match as well, so your document vector is still more similar to other political pieces than physics. Going up the scale, here's a few basic starting points that were (are?) the backbone of many production text AI/ML systems. 1. Bag of words. Here your vector has a 1 for words that are present, and 0 for ones that aren't. 2. Bag of words with a count. A little better, now we've got the information that you said "gravity" fifty times not once. Normalise it so text length doesn't matter and everything fits into 0-1. 3. TF-IDF. It's not very useful to know that you said a common word a lot. Most texts do, what we care about is ones that say it more than you'd expect so we take into account how often the words appear in the entire corpus. These don't help with words, but given how simple they are they are shockingly useful. They have their stupid moments, although one benefit is that it's very easy to debug why they cause a problem.

thisiszilff 3y ago

> I think the key property of embeddings is that the dimensions each individually mean/measure something, and therefore the dot product of two embeddings (similarity of direction of the vectors) is a meaningful similarity measure of the things being represented. In this case each dimension is the presence of a word in a particular text. So when you take the dot product of two texts you are effectively counting the number of words the two texts have in common (subject to some normalization constants depending on how you normalize the embedding). Cosine similarity still works for even these super naive embeddings which makes it slightly easier to understand before getting into any mathy stuff. You are 100% right this won't give you the word embedding analogies like king - man = queen or stuff like that. This embedding has no concept of relationships between words.

HarHarVeryFunny 3y ago
But that doesn't seem to be what you are describing in terms of using incrementing indices and adding occurrence counts. If you want to create a bag of words text embedding then you set the number of embedding dimensions to the vocabulary size and the value of each dimension to the global count of the corresponding word.
- thisiszilff 3y ago
  Heh -- my explanation isn't the clearest I realize, but yes, it is BoW. Eg fix your vocab of 50k words (or whatever) and enumerate it. Then to make an embedding for some piece of text 1. initialize an all zero vector of size 50k 2. for each word in the text, add one to the index of the corresponding word (per our enumeration). If the word isn't in the 50k words in your vocabulary, then discard it 3. (optionally), normalize the embedding to 1 (though you don't really need this and can leave it off for the toy example). initialize an embedding (for a single text) as an all zero vector of size 50k
  - p1esk 3y ago
    This is not the best way to understand where modern embeddings are coming from.