4 ms·

Just tried a set of classic non-monotonic reasoning statements and it didn't like it much: https://rule-reasoning.apps.allenai.org/?p=Penguins%20are%20birds%0AB

• smoyer 7y ago
  That one seemed to work for me too ... I'm wondering if there is something weird going on with the links (or is the transformer training as we go?).
  • scribu 7y ago
    Yeah, I'm now getting the corret answer: `Can tweety fly? True (confidence = 0.99)`
  • asacalowww 7y ago
    Mm don't think so. It's a non-monotonic statement, in that given Tweety is a bird, and therefore might also be a penguin then it can't make an assertion either way as to whether Tweety can fly or not. Non-monotonic reasoning is especially useful in many real world scenarios where a derived logic system doesn't have tidy yes/no answers throughout.
    • YeGoblynQueenne 7y ago
      The funny thing is that they report results on some hand-crafted examples of reasoning over Birds flying or not, (by Marek Sergot from Imperial) and they have a little table where all the language models perform almost perfectly. And yet, as you have found, they do not. Or perhaps- they do, but you have to run the same experiment a few thousand times (it's a tiny dataset of five clauses or so) until you get the good results, eh? I'm just being a big old meanie now.