3 ms·

Good question! This is the whole point of parsing sentences to deduce semantic meaning. He bent the fork and He could fork the Github repository Remember the

Good question! This is the whole point of parsing sentences to deduce semantic meaning. He bent the fork and He could fork the Github repository Remember the deductive reasoning puzzles when you were a kid? [0]There are some facts in these sentences and let's use some deductive reasoning to figure it out. He followed by bent means that He has to be the subject and bent has to be the main verb of the sentence. The the following has to belong to fork. It is never not that way with grammar. I can go a lot deeper here but I'm going to keep this short. We can train any machine to deduce that fork has to be a noun and that it is a direct object of the verb bent. Fork means one thing in this sentence. It is a tool to eat with. The modal could in the second sentence requires grammatically to have a verb following it. It not so much guessing what can fork mean, an action or an eating utensil, but rather, what is its function? He has to be a subject. If it was on the other side of the verb, it would be him. Because of the the, the Github repository has to be a noun phrase. By deduction fork has to be a verb. Therefore fork is an action of dividing into another part. Like I said before we can go deeper, for example, the capital G in Github means it is a proper noun so it can't function as a verb and the verb takes an object. Whether a verb takes an object or not can give semantic meaning to the verb too. This is an example of how the existence of the in a phrase allows machines to deduce meanings of the same word in different sentence. [0] http://www.prufrock.com/Assets/ClientScans/prodscanlg/193_sa5.jpg http://www.prufrock.com/Assets/ClientScans/prodscanlg/193_sa...

cscurmudgeon 10y ago
Thank you. I am aware of some of these approaches. Do you know how word2vec and other similar approaches handle this?
- gradys 10y ago
  Generally, they don't. word2vec and GloVe, the two most popular word vector models in the circles I run with, don't have any solution to this at all. As a result, when you do two dimensional visualizations of these vector spaces, words with multiple commonly used senses get positions in the middle of nowhere. In many situations, the downstream models that depend on these word vectors manage to perform well anyway, so for simplicity, we just live with this limitation. That said, there has been work on handling polysemy (the property of one word having multiple meanings) in word vector models. The simplest method I've heard of it is to do the word sense disambiguation out of band and then tag some kind of identifier onto the end of each word before you start training your word vector model. As an example, you could run a part of speech tagger and then tag the part of speech onto the end of the word. So in the above example, we'd get "fork_verb" and "fork_noun". Part of speech doesn't fully disambiguate a word, so this only gets you part of the way there, but at least it's easy. You can do something similar with named entities. You can replace the two words "Larry" and "Page", which the model would learn very generic vectors for with "Larry_Page" or "EntityID_192318" or whatever. There has also been work in automatically detecting different senses of a word. I've got at least one paper[1] in my notes that talks about this kind of thing. It does k-means clustering on the contexts that a particular word appears in, and learns a different representation for each cluster. [1] - Eric H. Huang , Richard Socher , Christopher D. Manning , Andrew Y. Ng, Improving word representations via global context and multiple word prototypes, Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics: Long Papers, July 08-14, 2012, Jeju Island, Korea

cscurmudgeon 10y ago

Thank you. I am aware of some of these approaches. Do you know how word2vec and other similar approaches handle this?

gradys 10y ago
Generally, they don't. word2vec and GloVe, the two most popular word vector models in the circles I run with, don't have any solution to this at all. As a result, when you do two dimensional visualizations of these vector spaces, words with multiple commonly used senses get positions in the middle of nowhere. In many situations, the downstream models that depend on these word vectors manage to perform well anyway, so for simplicity, we just live with this limitation. That said, there has been work on handling polysemy (the property of one word having multiple meanings) in word vector models. The simplest method I've heard of it is to do the word sense disambiguation out of band and then tag some kind of identifier onto the end of each word before you start training your word vector model. As an example, you could run a part of speech tagger and then tag the part of speech onto the end of the word. So in the above example, we'd get "fork_verb" and "fork_noun". Part of speech doesn't fully disambiguate a word, so this only gets you part of the way there, but at least it's easy. You can do something similar with named entities. You can replace the two words "Larry" and "Page", which the model would learn very generic vectors for with "Larry_Page" or "EntityID_192318" or whatever. There has also been work in automatically detecting different senses of a word. I've got at least one paper[1] in my notes that talks about this kind of thing. It does k-means clustering on the contexts that a particular word appears in, and learns a different representation for each cluster. [1] - Eric H. Huang , Richard Socher , Christopher D. Manning , Andrew Y. Ng, Improving word representations via global context and multiple word prototypes, Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics: Long Papers, July 08-14, 2012, Jeju Island, Korea