3 ms·

> One important thing to note is that Word2Vec does not consider the positional variable of the context words. There is a very interesting new approach of lear

> One important thing to note is that Word2Vec does not consider the positional variable of the context words. There is a very interesting new approach of learning Gaussian mixtures from words, which addresses this problem: https://github.com/benathi/word2gm https://github.com/benathi/word2gm It also solves another interesting problem, which is that Word2Vec doesn't always put "categories" close to their actual contents. Eg. I wrote a simple AI for playing Codenames with word vectors: https://github.com/thomasahle/codenames https://github.com/thomasahle/codenames But where a human use the clue "Countries 2" for "Germany" and "England", the Glove vectors (at least in my implementation) much prefers the clue "France 2", which is pretty confusing at first.

yorwba 9y ago
The Gaussian mixture representation is quite interesting, so thanks for the link. But it seems like they do not make use of the relative position of words in the text either. They simply maximize the margin of the expected likelihood for words that occur in the same context over those that do not. However, in most cases it's probably actually better to ignore word order, since that captures more semantic relatedness rather than the syntactic features that influence word order. It also lets you translate between languages that do not necessarily have similar grammar: "Word Translation Without Parallel Data" https://arxiv.org/abs/1710.04087 https://arxiv.org/abs/1710.04087
- Cybiote 9y ago
  Whether it's okay to ignore word order really depends on your goals. In my work, tracking word order to build a sentence representation is necessary to get composition rules correct. You can get away with ignoring order surprisingly often but when it's needed, it's vital. When aggregated across an entire corpus, order ends up mattering a lot, in an absolute sense (something looking only at accuracy numbers will hide). Compare the following: 1) You must state the benefit. 2) You must benefit the state. 3) Feed fish vs. Fish feed. Sometimes limits in inference: 1) A loves B. 2) Cats eat fish. 3) Cats are mammals. Sometimes the sequence preceding or following will be important: 1) Problems with learning 2) Learning with problems 3) It has a learning problem 4) A learning problem for it. Or consider an example from: https://www.researchgate.net/publication/2335962_How_Well_Can_Passage_Meaning_be_Derived_without_Using_Word_Order_A_Comparison_of_Latent_Semantic_Analysis_and_Humans https://www.researchgate.net/publication/2335962_How_Well_Ca... 1) It was not the sales manager who hit the bottle that day, but the office worker with the serious drinking problem. 2) That day the office manager, who was drinking, hit the problem sales worker with the bottle, but it was not serious. Order ends up mattering when you want to capture nuance or power higher order inference.
  - yorwba 9y ago
    Right, what I meant was not that order doesn't matter at all, but that order mostly doesn't matter for the meaning of individual words (except for some homonyms which can be disambiguated using syntactical information). When you want to handle whole sentences, obviously order becomes much more relevant, but for that you'd feed the word vectors into an LSTM or similar to handle the order-dependence.
  - neoncontrails 9y ago
    This is much more nuanced criticism of language vectors than Chomsky's impatient hatred of them. There are interviews worth watching in which he loathes them for all the reasons you mention. Thankfully, I suppose, they rarely disappoint me. :)

yorwba 9y ago

The Gaussian mixture representation is quite interesting, so thanks for the link. But it seems like they do not make use of the relative position of words in the text either. They simply maximize the margin of the expected likelihood for words that occur in the same context over those that do not. However, in most cases it's probably actually better to ignore word order, since that captures more semantic relatedness rather than the syntactic features that influence word order. It also lets you translate between languages that do not necessarily have similar grammar: "Word Translation Without Parallel Data" https://arxiv.org/abs/1710.04087 https://arxiv.org/abs/1710.04087

Cybiote 9y ago
Whether it's okay to ignore word order really depends on your goals. In my work, tracking word order to build a sentence representation is necessary to get composition rules correct. You can get away with ignoring order surprisingly often but when it's needed, it's vital. When aggregated across an entire corpus, order ends up mattering a lot, in an absolute sense (something looking only at accuracy numbers will hide). Compare the following: 1) You must state the benefit. 2) You must benefit the state. 3) Feed fish vs. Fish feed. Sometimes limits in inference: 1) A loves B. 2) Cats eat fish. 3) Cats are mammals. Sometimes the sequence preceding or following will be important: 1) Problems with learning 2) Learning with problems 3) It has a learning problem 4) A learning problem for it. Or consider an example from: https://www.researchgate.net/publication/2335962_How_Well_Can_Passage_Meaning_be_Derived_without_Using_Word_Order_A_Comparison_of_Latent_Semantic_Analysis_and_Humans https://www.researchgate.net/publication/2335962_How_Well_Ca... 1) It was not the sales manager who hit the bottle that day, but the office worker with the serious drinking problem. 2) That day the office manager, who was drinking, hit the problem sales worker with the bottle, but it was not serious. Order ends up mattering when you want to capture nuance or power higher order inference.
- yorwba 9y ago
  Right, what I meant was not that order doesn't matter at all, but that order mostly doesn't matter for the meaning of individual words (except for some homonyms which can be disambiguated using syntactical information). When you want to handle whole sentences, obviously order becomes much more relevant, but for that you'd feed the word vectors into an LSTM or similar to handle the order-dependence.
- neoncontrails 9y ago
  This is much more nuanced criticism of language vectors than Chomsky's impatient hatred of them. There are interviews worth watching in which he loathes them for all the reasons you mention. Thankfully, I suppose, they rarely disappoint me. :)