4 ms·
>To conclude: one must have different standards for developing systems than for testing, deploying, or using systems. Or: testing on your training data is a com
by sebzim4500 4y ago
>To conclude: one must have different standards for developing systems than for testing, deploying, or using systems. Or: testing on your training data is a common way to cheat, but so is training on your test data.
Isn't this already a solved problem? Every reasonable paper on ML separates their test data from their validation data already.
- version_five 4y agoThat in no way prevents overfitting though hyperparameter optimization / graduate student descent. All the common benchmarks, by definition of being a common benchmark, are susceptible to overfitting
- sebzim4500 4y agoThe idea is that you do all your hyperparameter optimization with the test data and then only run through the validation data once before you submit your paper.
- ummonk 4y agoThat's why you split it 3-way into train, validation, and test datasets, to ensure you didn't use too many hyperparameters and overfit to the validation data.