5 ms·

A key tenet of supervised learning is that you will only ever do as well as what's in your training set.

TekMol 9y ago
They state that it "achieved state-of-the-art accuracies on the SQuAD dataset (Wikipedia sentences) in early 2017" So I would assume it has bean heavily trained?
- yorwba 9y ago
  If you look at example questions from the dev set[1], you'll realize that they all use the same words as the sentence containing the answer. Additionally, the topics aren't everyday stuff, but something you'd write a Wikipedia article about. So I guess the model just learns to find the sentence most similar to the question and then selects an answer based on a coarse categorization, which fails when it is presented with unseen situations. Your example works if you rephrase the question to be more similar to the text: Paul gives a coin to Joe. Whom does Paul give a coin to? Answer: Joe You can cut the question down to "gives to?" or "coin to?", because that's enough to single out the answer. But as soon as you use s̶y̶n̶o̶n̶y̶m̶s̶ (EDIT: related words) that are not recognized (like "receive"), you have no chance of getting a meaningful answer. [1] https://rajpurkar.github.io/SQuAD-explorer/explore/1.1/dev/ https://rajpurkar.github.io/SQuAD-explorer/explore/1.1/dev/
  - TekMol 9y ago
    Yes, that might be a more accurate description. It picks the "who" thingy from the most similar context. With zero further understanding as it seems: Paul gives no coin to Marray. Paul gives a coin to Joe. Who got something from Paul? Anser: Marray Paul gives no coin to Marray. Paul gives a coin to Joe. Who received a coin? Answer: Paul
  - adrianbg 9y ago
    The "Who did What" dataset seems much better in this respect: Passage: Britain’s decision on Thursday to drop extradition proceedings against Gen. Augusto Pinochet and allow him to return to Chile is understandably frustrating ... Jack Straw, the home secretary, said the 84-year-old former dictator’s ability to understand the charges against him and to direct his defense had been seriously impaired by a series of strokes. ... Chile’s president-elect, Ricardo Lagos, has wisely pledged to let justice run its course. But the outgoing government of President Eduardo Frei is pushing a constitutional reform that would allow Pinochet to step down from the Senate and retain parliamentary immunity from prosecution. ... Question: Sources close to the presidential palace said that Fujimori declined at the last moment to leave the country and instead he will send a high level delegation to the ceremony, at which Chilean President Eduardo Frei will pass the mandate to XXX. Choices: (1) Augusto Pinochet (2) Jack Straw (3) Ricardo Lagos https://arxiv.org/abs/1608.05457 https://arxiv.org/abs/1608.05457
- rspeer 9y ago
  Heavily trained on SQuAD questions. There are lots of models out there that are very good at recognizing SQuAD questions, and reverse-engineering the predictable ways that the Turkers who wrote the questions pulled the information out of the paragraph -- allowing them to answer the question without ever understanding it. https://arxiv.org/abs/1707.07328 https://arxiv.org/abs/1707.07328

They state that it "achieved state-of-the-art accuracies on the SQuAD dataset (Wikipedia sentences) in early 2017" So I would assume it has bean heavily trained?

yorwba 9y ago
If you look at example questions from the dev set[1], you'll realize that they all use the same words as the sentence containing the answer. Additionally, the topics aren't everyday stuff, but something you'd write a Wikipedia article about. So I guess the model just learns to find the sentence most similar to the question and then selects an answer based on a coarse categorization, which fails when it is presented with unseen situations. Your example works if you rephrase the question to be more similar to the text: Paul gives a coin to Joe. Whom does Paul give a coin to? Answer: Joe You can cut the question down to "gives to?" or "coin to?", because that's enough to single out the answer. But as soon as you use s̶y̶n̶o̶n̶y̶m̶s̶ (EDIT: related words) that are not recognized (like "receive"), you have no chance of getting a meaningful answer. [1] https://rajpurkar.github.io/SQuAD-explorer/explore/1.1/dev/ https://rajpurkar.github.io/SQuAD-explorer/explore/1.1/dev/
- TekMol 9y ago
  Yes, that might be a more accurate description. It picks the "who" thingy from the most similar context. With zero further understanding as it seems: Paul gives no coin to Marray. Paul gives a coin to Joe. Who got something from Paul? Anser: Marray Paul gives no coin to Marray. Paul gives a coin to Joe. Who received a coin? Answer: Paul
- adrianbg 9y ago
  The "Who did What" dataset seems much better in this respect: Passage: Britain’s decision on Thursday to drop extradition proceedings against Gen. Augusto Pinochet and allow him to return to Chile is understandably frustrating ... Jack Straw, the home secretary, said the 84-year-old former dictator’s ability to understand the charges against him and to direct his defense had been seriously impaired by a series of strokes. ... Chile’s president-elect, Ricardo Lagos, has wisely pledged to let justice run its course. But the outgoing government of President Eduardo Frei is pushing a constitutional reform that would allow Pinochet to step down from the Senate and retain parliamentary immunity from prosecution. ... Question: Sources close to the presidential palace said that Fujimori declined at the last moment to leave the country and instead he will send a high level delegation to the ceremony, at which Chilean President Eduardo Frei will pass the mandate to XXX. Choices: (1) Augusto Pinochet (2) Jack Straw (3) Ricardo Lagos https://arxiv.org/abs/1608.05457 https://arxiv.org/abs/1608.05457
rspeer 9y ago
Heavily trained on SQuAD questions. There are lots of models out there that are very good at recognizing SQuAD questions, and reverse-engineering the predictable ways that the Turkers who wrote the questions pulled the information out of the paragraph -- allowing them to answer the question without ever understanding it. https://arxiv.org/abs/1707.07328 https://arxiv.org/abs/1707.07328