If people are interested in getting started with question answering on the bAbi dataset, I wrote a baseline RNN example that's now in Keras[1]. Whilst I love t

If people are interested in getting started with question answering on the bAbi dataset, I wrote a baseline RNN example that's now in Keras[1]. Whilst I love the theory behind the bAbi dataset, there are issues with the data itself that still need to be fixed, particularly that there is duplication in the datasets[2]. This duplication leads to less unique data than expected and even overlaps between training and testing data. The problem gets worse as you get to the 10k subsets[3]. Most worryingly, authors who I've spoken to aren't aware of these issues. As an example, QA17 (Positional Reasoning) which is one of the focus areas of this paper has two major issues. First, instead of 10,000 / 1,000 unique train / test questions, duplication means it only has 5,812 / 632. Second, 15% of the exact training questions in the 10k subset are repeated in the test set. I've contacted the creators of the dataset at Facebook but they don't seem to think the issue is that extreme. The authors of papers using the bAbi dataset are also not noting which version of the data they've used, which is problematic if there are future updates... [1]: https://github.com/fchollet/keras/blob/master/examples/babi_rnn.py https://github.com/fchollet/keras/blob/master/examples/babi_... [2]: http://smerity.com/articles/2015/keras_qa.html#dataset-issues http://smerity.com/articles/2015/keras_qa.html#dataset-issue... [3]: https://gist.github.com/Smerity/8ceb539c125cbe648bfe https://gist.github.com/Smerity/8ceb539c125cbe648bfe