3 ms·
The passage I quote above speaks of "out-of-sample" generalisation, not "test-set" generalisation. These are not the same. Unfortunately such terminological co
by YeGoblynQueenne 6y ago
The passage I quote above speaks of "out-of-sample" generalisation, not "test-set" generalisation. These are not the same.
Unfortunately such terminological confusion is common but "out-of-sample" should really be reserved for data that was not available during development of a system, either as a training, evaluation or testing partition. That, because "out of sample" suggests that the data was drawn from a different distribution than the, well, training sample (where the training sample is then subdivided to training, evaluation and testing partitions), i.e. the true distribution, so the real world.
I guess the OP is instead using "out-of-sample" to mean "test set" (which is not uncommon), but in that case we don't need to look all the way to learning theory to figure it out: published results are well known to select for successful experiments, in machine learning as in many areas of research, unfortunately.