4 ms·

I should point out that is an artificial dataset generated by computers according to some simple rules. Doing good on these doesn't necessarily mean anything.

by ma2rten 11y ago

I should point out that is an artificial dataset generated by computers according to some simple rules. Doing good on these doesn't necessarily mean anything. Also, they didn't report the accuracy on any of the other 18 tasks. That makes me believe that they did pretty badly on those.

fpgaminer 11y ago
The stated reason for only testing on two of the task types: > For simplicity, we focus on the reasoning tasks which can be formulated as classification with predetermined classes.
336f5 11y ago
The /r/machinelearning discussion (https://www.reddit.com/r/MachineLearning/comments/3iayyz/neural_reasoner_achieves_90_accuracies_on/ https://www.reddit.com/r/MachineLearning/comments/3iayyz/neu...) makes an interesting point about the 18 tasks thing: > That was my first instinct too. Going back to the dynamic memory networks paper though, it turns out (§4.1) that these are the only two tasks it didn't solve almost-perfectly.
nl 11y ago
It is true that this is an artificial test, but it seems to me that doing well at this is a necessary prerequisite to do well at more complex scenarios. It's worth noting that these tests were introduced by Jason Weston[1] as "A Set of Prerequisite Toy Tasks"[2] Also, they didn't report the accuracy on any of the other 18 tasks. That makes me believe that they did pretty badly on those. Weston's work on Memory Networks has "solved" (or at least shown a good way forward) on 16 of those other tasks. These were the two that were furthest away from being solved (See [2], again). The remaining 2 look like they will fall to Memory Networks with some work. [1] http://www.thespermwhale.com/jaseweston/ http://www.thespermwhale.com/jaseweston/ [2] http://arxiv.org/abs/1502.05698 http://arxiv.org/abs/1502.05698
pberlin 11y ago
Copied from Reddit FYI. As the second author of the paper, I have to say "Wrong Guess", :), we didn't try the other 18. The story is as follows. We had this idea in early March as a reasoning component of our bigger (and cool) neural system. In early May my student intern (the first author) decided to implement it, but he said "why don't we try this on a couple of hard bAbi tasks on which Memory Net failed？", and then we are here. That said, I don't mean Neural Reasoner can easily solve them without any model tweaking, considering some of them are formulated slightly differently from the two we tried . But more importantly, it was never our goal to nail an artificial data set.

by ma2rten 11y ago

fpgaminer 11y ago

The stated reason for only testing on two of the task types: > For simplicity, we focus on the reasoning tasks which can be formulated as classification with predetermined classes.

336f5 11y ago

The /r/machinelearning discussion (https://www.reddit.com/r/MachineLearning/comments/3iayyz/neural_reasoner_achieves_90_accuracies_on/ https://www.reddit.com/r/MachineLearning/comments/3iayyz/neu...) makes an interesting point about the 18 tasks thing: > That was my first instinct too. Going back to the dynamic memory networks paper though, it turns out (§4.1) that these are the only two tasks it didn't solve almost-perfectly.

nl 11y ago

It is true that this is an artificial test, but it seems to me that doing well at this is a necessary prerequisite to do well at more complex scenarios. It's worth noting that these tests were introduced by Jason Weston[1] as "A Set of Prerequisite Toy Tasks"[2] Also, they didn't report the accuracy on any of the other 18 tasks. That makes me believe that they did pretty badly on those. Weston's work on Memory Networks has "solved" (or at least shown a good way forward) on 16 of those other tasks. These were the two that were furthest away from being solved (See [2], again). The remaining 2 look like they will fall to Memory Networks with some work. [1] http://www.thespermwhale.com/jaseweston/ http://www.thespermwhale.com/jaseweston/ [2] http://arxiv.org/abs/1502.05698 http://arxiv.org/abs/1502.05698

pberlin 11y ago

Copied from Reddit FYI. As the second author of the paper, I have to say "Wrong Guess", :), we didn't try the other 18. The story is as follows. We had this idea in early March as a reasoning component of our bigger (and cool) neural system. In early May my student intern (the first author) decided to implement it, but he said "why don't we try this on a couple of hard bAbi tasks on which Memory Net failed？", and then we are here. That said, I don't mean Neural Reasoner can easily solve them without any model tweaking, considering some of them are formulated slightly differently from the two we tried . But more importantly, it was never our goal to nail an artificial data set.