"No feature engineering" claim would be far more convincing if they demonstrated network's abilities on several datasets of different types. It's not like that

"No feature engineering" claim would be far more convincing if they demonstrated network's abilities on several datasets of different types. It's not like that would be hard to generate synthetically. An interesting test would be "logic puzzles" that use shapes, object counts and so on. For example, a test where you have to label a picture based on the number of objects, regardless of their shapes. As it is, it could be that this particular setup happens to extract features of this particular dataset, while failing miserably on others. --- Thinking about this at a higher level, I ask myself what constitutes true one-shot learning. The reason we care about it is because in real life most problems don't involve huge datasets of available solutions. On the other hand, this paper does involve training a model based on a large dataset of similarly typed, labeled data. The problem the algorithm solves involves items from the same set. The first obvious question is the one I asked above: does this approach work for other types of data? The second one is whether the mechanism would work for more diverse datasets. Finally, the most important question is: how well will it perform on tasks that fall far outside of the initial training data? Because that's the true challenge behind one-shot learning. It's kind of amazing that the paper doesn't try to answer any of those questions. Isn't that the real purpose of research in AI? (Most likely they tried and the results weren't good, but hey, there is no way to tell without re-implementing the whole thing.)