3 ms·

The algorithm learns by looking at contextual words around a given word. If a word is infrequent, there isn't enough context to span the space of possible cont

by spott 3y ago

The algorithm learns by looking at contextual words around a given word. If a word is infrequent, there isn't enough context to span the space of possible contexts for the word, making it difficult for the algorithm to get a good idea about what the word means. It is also worth discussing the difference between the training corpus (the data that the word vectors are found from) and the inference corpus (the data that the word vectors are used on). If a word is uncommon in the inference corpus, but is common enough to be trained on the training corpus, then a vector is still generated and useful for the inference task.

nico 3y ago
Thank you for the thoughtful explanation. Follow up: Is the context considered “all appearances of the word”, or “text around the word”? Also, are all positions/“areas” of the corpus considered evenly important except for their frequency? In other words, ie position within the corpus considered at all? (eg. Usually titles are more relevant to a document then a random sentence or word in the middle of it)
- extasia 3y ago
  The context for one training step is the target word, plus a window of the `n` words that precede and follow that word. For example in the sentence: > Usually titles are more relevant to a document then a random sentence or word in the middle of it If 'document' is the current target word, we select ["relevant", "to", "a"] and ["then, "a", "random"] as our left and right contexts. These context words and our target word, ["document"] form our 'positive' examples that we want to move closer together in the embedding space. Position in the text is not used in this algorithm. You're right about titles being more relevant to the meaning of a document but the document isn't important: it's just a resource that we're using to learn which words occur together (and thus are semantically related). The algorithm steps through every single word as a target and thus will inevitably utilize words that were titles just as much as words in any other part of the text:) Hope that helps!
  - nico 3y ago
    > Hope that helps! Yes, very much Thank you for such a great explanation, super clear :)

by spott 3y ago

nico 3y ago

Thank you for the thoughtful explanation. Follow up: Is the context considered “all appearances of the word”, or “text around the word”? Also, are all positions/“areas” of the corpus considered evenly important except for their frequency? In other words, ie position within the corpus considered at all? (eg. Usually titles are more relevant to a document then a random sentence or word in the middle of it)

extasia 3y ago
The context for one training step is the target word, plus a window of the `n` words that precede and follow that word. For example in the sentence: > Usually titles are more relevant to a document then a random sentence or word in the middle of it If 'document' is the current target word, we select ["relevant", "to", "a"] and ["then, "a", "random"] as our left and right contexts. These context words and our target word, ["document"] form our 'positive' examples that we want to move closer together in the embedding space. Position in the text is not used in this algorithm. You're right about titles being more relevant to the meaning of a document but the document isn't important: it's just a resource that we're using to learn which words occur together (and thus are semantically related). The algorithm steps through every single word as a target and thus will inevitably utilize words that were titles just as much as words in any other part of the text:) Hope that helps!
- nico 3y ago
  > Hope that helps! Yes, very much Thank you for such a great explanation, super clear :)