4 ms·

Wow, is this really state of the art? Joe did not buy a car today. He was in buying mood. But all cars were too expensive. Why didn't Joe buy

by TekMol 9y ago

Wow, is this really state of the art? Joe did not buy a car today. He was in buying mood. But all cars were too expensive. Why didn't Joe buy a car? Answer: buying mood I think I have seen similar systems for decades now. I thought we would be further along meanwhile. I have tried for 10 or 20 minutes now. But I can't find any evidence that it has much sense of syntax: Paul gives a coin to Joe. Who received a coin? Answer: Paul All it seems to do is to extract candidates for "who", "what", "where" etc. So it seems to figure out correctly that "Paul" is a potential answer for "Who". No matter how I rephrase the "Who" question, I always get "Paul" as the answer. "Who? Paul!", "Who is a martian? Paul!", "Who won the summer olympics? Paul", "Who got a coin from the other guy? Paul!" Same for "what" questions: Gold can not be carried in a bag. Silver can. What can be carried in a bag? Answer: Gold

senatorobama 9y ago
A key tenet of supervised learning is that you will only ever do as well as what's in your training set.
- TekMol 9y ago
  They state that it "achieved state-of-the-art accuracies on the SQuAD dataset (Wikipedia sentences) in early 2017" So I would assume it has bean heavily trained?
  - yorwba 9y ago
    If you look at example questions from the dev set[1], you'll realize that they all use the same words as the sentence containing the answer. Additionally, the topics aren't everyday stuff, but something you'd write a Wikipedia article about. So I guess the model just learns to find the sentence most similar to the question and then selects an answer based on a coarse categorization, which fails when it is presented with unseen situations. Your example works if you rephrase the question to be more similar to the text: Paul gives a coin to Joe. Whom does Paul give a coin to? Answer: Joe You can cut the question down to "gives to?" or "coin to?", because that's enough to single out the answer. But as soon as you use s̶y̶n̶o̶n̶y̶m̶s̶ (EDIT: related words) that are not recognized (like "receive"), you have no chance of getting a meaningful answer. [1] https://rajpurkar.github.io/SQuAD-explorer/explore/1.1/dev/ https://rajpurkar.github.io/SQuAD-explorer/explore/1.1/dev/
    - TekMol 9y ago
      Yes, that might be a more accurate description. It picks the "who" thingy from the most similar context. With zero further understanding as it seems: Paul gives no coin to Marray. Paul gives a coin to Joe. Who got something from Paul? Anser: Marray Paul gives no coin to Marray. Paul gives a coin to Joe. Who received a coin? Answer: Paul
    - adrianbg 9y ago
      The "Who did What" dataset seems much better in this respect: Passage: Britain’s decision on Thursday to drop extradition proceedings against Gen. Augusto Pinochet and allow him to return to Chile is understandably frustrating ... Jack Straw, the home secretary, said the 84-year-old former dictator’s ability to understand the charges against him and to direct his defense had been seriously impaired by a series of strokes. ... Chile’s president-elect, Ricardo Lagos, has wisely pledged to let justice run its course. But the outgoing government of President Eduardo Frei is pushing a constitutional reform that would allow Pinochet to step down from the Senate and retain parliamentary immunity from prosecution. ... Question: Sources close to the presidential palace said that Fujimori declined at the last moment to leave the country and instead he will send a high level delegation to the ceremony, at which Chilean President Eduardo Frei will pass the mandate to XXX. Choices: (1) Augusto Pinochet (2) Jack Straw (3) Ricardo Lagos https://arxiv.org/abs/1608.05457 https://arxiv.org/abs/1608.05457
  - rspeer 9y ago
    Heavily trained on SQuAD questions. There are lots of models out there that are very good at recognizing SQuAD questions, and reverse-engineering the predictable ways that the Turkers who wrote the questions pulled the information out of the paragraph -- allowing them to answer the question without ever understanding it. https://arxiv.org/abs/1707.07328 https://arxiv.org/abs/1707.07328
galenko 9y ago
Sadly, the NLP world is full of hot air. I've seen so many companies get funding for complete "written by a 12-year old" dogshit "industry leading IP", it's not even funny anymore. The hype has gone down and some are actually doing great work, but 90% of the people who say they do NLP/AI stuff don't even fundamentally understand what NLP/AI is.
- fnl 9y ago
  Sadly, I'd fully agree to this. Things are possible now that were not 10 years ago. But mostly, only performance increased on things we could do 10 years ago, while hardly any new abilities came along. Machine translation, linguistic parsing, etc. came a long way. But we still can't do satisfactory abstractive summarization or create a conversational agent for more than an extremely narrow domain. Yet, at least the things we can do can be done at levels that are "production ready".
- tanilama 9y ago
  I still hold hope. However, it seems naively exploit the function approximation capacity we have with deep learning can only go that far to understand our own language. Maybe we need to look back and start from beginning and ask ourself: How does human learn, exactly? How do we learn with so few examples? How do we jointly learn image/audio/video/language with only one brain?
  - amelius 9y ago
    Perhaps we should consider working only on techniques which improve as more computational power is thrown at it.
    - glup 9y ago
      Computation won't help if we don't have the right representations. Arguably computation can help us discover the right representations but the space of possible representations is very, very large.
msamwald 9y ago
According to the website they use the BiDAF model, which as a single model does not produce state-of-the-art results on the SQuAD benchmark. It is ranked 27th here: https://rajpurkar.github.io/SQuAD-explorer/ https://rajpurkar.github.io/SQuAD-explorer/
halflings 9y ago
The difference with new NN-based systems is that they are trained end-to-end, learn the syntax and some form of "reasoning". Check Memory Networks, by facebook, for example (two NNs, one for "reasoning" and one for storing long-term data, quite impressive). Now, it's still an area of active research... and I'm not sure what "state-of-the-art" means for this library, somebody said that they rank #27th in some commonly used dataset.
- reachtarunhere 9y ago
  I am working with Memory Networks as part of my thesis. If you actually read and implement the FB paper you realise that the system is not half great as the demo shows. It is as bad as the top comment here. Yes we have come far along but frankly the hype is too high.
rubyfan 9y ago
I can’t imagine that the human mind works even remotely how these NPL systems work. Grammars, tokenizing, matrices... there must be a better approach
make3 9y ago
Try the more complicated text sources. It is able to parse them and still answer questions reasonably well
glup 9y ago
All of the above require fairly complex world knowledge as well as an explicit representation of a scene. There is minimal leverage for lexical distributional statistics in these cases—arguably the one thing we have had major success in using (e.g. building vector space word representations, like Word2Vec; finding the highest probability parse tree for an utterance).

by TekMol 9y ago

senatorobama 9y ago

A key tenet of supervised learning is that you will only ever do as well as what's in your training set.

TekMol 9y ago
They state that it "achieved state-of-the-art accuracies on the SQuAD dataset (Wikipedia sentences) in early 2017" So I would assume it has bean heavily trained?
- yorwba 9y ago
  If you look at example questions from the dev set[1], you'll realize that they all use the same words as the sentence containing the answer. Additionally, the topics aren't everyday stuff, but something you'd write a Wikipedia article about. So I guess the model just learns to find the sentence most similar to the question and then selects an answer based on a coarse categorization, which fails when it is presented with unseen situations. Your example works if you rephrase the question to be more similar to the text: Paul gives a coin to Joe. Whom does Paul give a coin to? Answer: Joe You can cut the question down to "gives to?" or "coin to?", because that's enough to single out the answer. But as soon as you use s̶y̶n̶o̶n̶y̶m̶s̶ (EDIT: related words) that are not recognized (like "receive"), you have no chance of getting a meaningful answer. [1] https://rajpurkar.github.io/SQuAD-explorer/explore/1.1/dev/ https://rajpurkar.github.io/SQuAD-explorer/explore/1.1/dev/
  - TekMol 9y ago
    Yes, that might be a more accurate description. It picks the "who" thingy from the most similar context. With zero further understanding as it seems: Paul gives no coin to Marray. Paul gives a coin to Joe. Who got something from Paul? Anser: Marray Paul gives no coin to Marray. Paul gives a coin to Joe. Who received a coin? Answer: Paul
  - adrianbg 9y ago
    The "Who did What" dataset seems much better in this respect: Passage: Britain’s decision on Thursday to drop extradition proceedings against Gen. Augusto Pinochet and allow him to return to Chile is understandably frustrating ... Jack Straw, the home secretary, said the 84-year-old former dictator’s ability to understand the charges against him and to direct his defense had been seriously impaired by a series of strokes. ... Chile’s president-elect, Ricardo Lagos, has wisely pledged to let justice run its course. But the outgoing government of President Eduardo Frei is pushing a constitutional reform that would allow Pinochet to step down from the Senate and retain parliamentary immunity from prosecution. ... Question: Sources close to the presidential palace said that Fujimori declined at the last moment to leave the country and instead he will send a high level delegation to the ceremony, at which Chilean President Eduardo Frei will pass the mandate to XXX. Choices: (1) Augusto Pinochet (2) Jack Straw (3) Ricardo Lagos https://arxiv.org/abs/1608.05457 https://arxiv.org/abs/1608.05457
- rspeer 9y ago
  Heavily trained on SQuAD questions. There are lots of models out there that are very good at recognizing SQuAD questions, and reverse-engineering the predictable ways that the Turkers who wrote the questions pulled the information out of the paragraph -- allowing them to answer the question without ever understanding it. https://arxiv.org/abs/1707.07328 https://arxiv.org/abs/1707.07328

galenko 9y ago

Sadly, the NLP world is full of hot air. I've seen so many companies get funding for complete "written by a 12-year old" dogshit "industry leading IP", it's not even funny anymore. The hype has gone down and some are actually doing great work, but 90% of the people who say they do NLP/AI stuff don't even fundamentally understand what NLP/AI is.

fnl 9y ago
Sadly, I'd fully agree to this. Things are possible now that were not 10 years ago. But mostly, only performance increased on things we could do 10 years ago, while hardly any new abilities came along. Machine translation, linguistic parsing, etc. came a long way. But we still can't do satisfactory abstractive summarization or create a conversational agent for more than an extremely narrow domain. Yet, at least the things we can do can be done at levels that are "production ready".
tanilama 9y ago
I still hold hope. However, it seems naively exploit the function approximation capacity we have with deep learning can only go that far to understand our own language. Maybe we need to look back and start from beginning and ask ourself: How does human learn, exactly? How do we learn with so few examples? How do we jointly learn image/audio/video/language with only one brain?
- amelius 9y ago
  Perhaps we should consider working only on techniques which improve as more computational power is thrown at it.
  - glup 9y ago
    Computation won't help if we don't have the right representations. Arguably computation can help us discover the right representations but the space of possible representations is very, very large.

msamwald 9y ago

According to the website they use the BiDAF model, which as a single model does not produce state-of-the-art results on the SQuAD benchmark. It is ranked 27th here: https://rajpurkar.github.io/SQuAD-explorer/ https://rajpurkar.github.io/SQuAD-explorer/

halflings 9y ago

The difference with new NN-based systems is that they are trained end-to-end, learn the syntax and some form of "reasoning". Check Memory Networks, by facebook, for example (two NNs, one for "reasoning" and one for storing long-term data, quite impressive). Now, it's still an area of active research... and I'm not sure what "state-of-the-art" means for this library, somebody said that they rank #27th in some commonly used dataset.

reachtarunhere 9y ago
I am working with Memory Networks as part of my thesis. If you actually read and implement the FB paper you realise that the system is not half great as the demo shows. It is as bad as the top comment here. Yes we have come far along but frankly the hype is too high.

rubyfan 9y ago

I can’t imagine that the human mind works even remotely how these NPL systems work. Grammars, tokenizing, matrices... there must be a better approach

make3 9y ago

Try the more complicated text sources. It is able to parse them and still answer questions reasonably well

glup 9y ago

All of the above require fairly complex world knowledge as well as an explicit representation of a scene. There is minimal leverage for lexical distributional statistics in these cases—arguably the one thing we have had major success in using (e.g. building vector space word representations, like Word2Vec; finding the highest probability parse tree for an utterance).