3 ms·

I don't understand why there is any doubt about this. From OP, section "Methods", subset "Data set creation and splits": "The training, validation and test dat

I don't understand why there is any doubt about this. From OP, section "Methods", subset "Data set creation and splits": "The training, validation and test data sets are generated without overlap from periods in sequence. Successive periods of 400, 12, 40, 40 and 12 h are used to sample, respectively, training, validation, and test data, with the two 12 h periods inserted as hiatus." I know nothing about machine learning, but I was in grad school with others studying machine learning and I took a Coursera course on the subject, and carving out a training set is utterly standard practice. A paper that didn't do this would get filtered by a grad student reviewer. So I'm not sure what possible sub-category of machine learning your statements could apply to. Can you share any examples? For example, do any of the famous papers in the field fail to hold out test data? Something like AlphaGo or AlphaZero is indeed testing on all new data -- new games it plays with others. Do any papers in well-known machine learning fora fail to hold out test data? Does anything that has bubbled to the front page of HN? It would be interesting to look at if so. Do you mean p-hacking? That's a much more subtle thing.

zamfi 4y ago
Very few ML papers use a held-out test set that is only used for publication. When you use your test set as part of model selection, you don't really have an empirical test set -- you do have a training set that you've partitioned into various parts for various purposes, but you don't have a proxy for how the system will perform in the face of new data, because you're reporting results that are based on data that you used as part of your model-generating process. If you try out models A, B, C, and D while developing, and then it turns out model D has the best performance on your test set, and you report your results using model D and that same test set, you are essentially "training" using your test set. Yes, in some problem contexts, for ML researchers who are also in the business of generating their own datasets, the data they test on can be entirely new. But for anyone using a pre-existing dataset, holding out a test set for your project is not yet a well-established practice -- and this makes it somewhat harder to trust the results.
- akkartik 4y ago
  I see, thanks for the explanation. Edit: Oh, and I see now that you explained it below, sorry.

zamfi 4y ago

Very few ML papers use a held-out test set that is only used for publication. When you use your test set as part of model selection, you don't really have an empirical test set -- you do have a training set that you've partitioned into various parts for various purposes, but you don't have a proxy for how the system will perform in the face of new data, because you're reporting results that are based on data that you used as part of your model-generating process. If you try out models A, B, C, and D while developing, and then it turns out model D has the best performance on your test set, and you report your results using model D and that same test set, you are essentially "training" using your test set. Yes, in some problem contexts, for ML researchers who are also in the business of generating their own datasets, the data they test on can be entirely new. But for anyone using a pre-existing dataset, holding out a test set for your project is not yet a well-established practice -- and this makes it somewhat harder to trust the results.

akkartik 4y ago
I see, thanks for the explanation. Edit: Oh, and I see now that you explained it below, sorry.