3 ms·

Thank you for the thoughtful explanation. Follow up: Is the context considered “all appearances of the word”, or “text around the word”? Also, are all positio

• extasia 3y ago
  The context for one training step is the target word, plus a window of the `n` words that precede and follow that word. For example in the sentence: > Usually titles are more relevant to a document then a random sentence or word in the middle of it If 'document' is the current target word, we select ["relevant", "to", "a"] and ["then, "a", "random"] as our left and right contexts. These context words and our target word, ["document"] form our 'positive' examples that we want to move closer together in the embedding space. Position in the text is not used in this algorithm. You're right about titles being more relevant to the meaning of a document but the document isn't important: it's just a resource that we're using to learn which words occur together (and thus are semantically related). The algorithm steps through every single word as a target and thus will inevitably utilize words that were titles just as much as words in any other part of the text:) Hope that helps!
  • nico 3y ago
    > Hope that helps! Yes, very much Thank you for such a great explanation, super clear :)