2 ms·
If you look at example questions from the dev set[1], you'll realize that they all use the same words as the sentence containing the answer. Additionally, the t
by yorwba 9y ago
If you look at example questions from the dev set[1], you'll realize that they all use the same words as the sentence containing the answer. Additionally, the topics aren't everyday stuff, but something you'd write a Wikipedia article about. So I guess the model just learns to find the sentence most similar to the question and then selects an answer based on a coarse categorization, which fails when it is presented with unseen situations.
Your example works if you rephrase the question to be more similar to the text:
Paul gives a coin to Joe.
Whom does Paul give a coin to?
Answer: Joe
You can cut the question down to "gives to?" or "coin to?", because that's enough to single out the answer. But as soon as you use s̶y̶n̶o̶n̶y̶m̶s̶ (EDIT: related words) that are not recognized (like "receive"), you have no chance of getting a meaningful answer.
[1] https://rajpurkar.github.io/SQuAD-explorer/explore/1.1/dev/ https://rajpurkar.github.io/SQuAD-explorer/explore/1.1/dev/
- TekMol 9y agoYes, that might be a more accurate description. It picks the "who" thingy from the most similar context. With zero further understanding as it seems: Paul gives no coin to Marray. Paul gives a coin to Joe. Who got something from Paul? Anser: Marray Paul gives no coin to Marray. Paul gives a coin to Joe. Who received a coin? Answer: Paul
- adrianbg 9y agoThe "Who did What" dataset seems much better in this respect: Passage: Britain’s decision on Thursday to drop extradition proceedings against Gen. Augusto Pinochet and allow him to return to Chile is understandably frustrating ... Jack Straw, the home secretary, said the 84-year-old former dictator’s ability to understand the charges against him and to direct his defense had been seriously impaired by a series of strokes. ... Chile’s president-elect, Ricardo Lagos, has wisely pledged to let justice run its course. But the outgoing government of President Eduardo Frei is pushing a constitutional reform that would allow Pinochet to step down from the Senate and retain parliamentary immunity from prosecution. ... Question: Sources close to the presidential palace said that Fujimori declined at the last moment to leave the country and instead he will send a high level delegation to the ceremony, at which Chilean President Eduardo Frei will pass the mandate to XXX. Choices: (1) Augusto Pinochet (2) Jack Straw (3) Ricardo Lagos https://arxiv.org/abs/1608.05457 https://arxiv.org/abs/1608.05457