The problem with the second paper is that having a "test set" is meaningless when you can access that test set during training - "you" as in the researcher, who

The problem with the second paper is that having a "test set" is meaningless when you can access that test set during training - "you" as in the researcher, who develops a system. This is especially so for machine vision datasets like the ones in the paper that have been "done to death". Basically anyone who has access to the test set of a popular benchmark and wants to get their paper published will do everything that can be done to ensure their system does well on the test set. That is a big flaw in machine learning research in general, but that's for another conversation, I guess. My point above is that if neural nets could generalise well, they wouldn't need so much data. In a sense, even if trained neural net models can generalise to instances outside the dense region of instance space circumscribed by their training set that is not that important, if that region has to be gigantic for this generalisation to be possbile in the first place. For one thing, at that point it becomes difficult to separate what is "training" and what is "test", especially so when test sets are four times the size of training sets as in typical practice.