3 ms·
> This brings quite reasonable questions about their past performance. I don't think this second claim is necessarily derivable from your first. Yes, they were
by jdmichal 11y ago
> This brings quite reasonable questions about their past performance.
I don't think this second claim is necessarily derivable from your first. Yes, they were caught cheating... By making multiple submissions of their own systems in order to faster derive which is the most promising. IMO, the performance they have achieved is still theirs to claim, because in the end they did create a system that reaches that performance.
- Retric 11y agohttp://en.m.wikipedia.org/wiki/Overfitting http://en.m.wikipedia.org/wiki/Overfitting Ideally, your final test data set should be completely separate from training data. Because even limited exposure quickly renders that test data meaningless. PS: In the end it's much like taking the same test with word for word identical questions a second time. Yes, it's a test of something, just not the intended subject matter.
- ninjin 11y agoIt is hard to emphasize how bad overfitting on the test set is. Your cheating analogy is accurate, and cheating in science is a serious matter. If it is unintentional, you should have your work rejected or retracted. If it is intentional, you should have your work rejected or retracted and, frankly speaking, you should most likely leave the field. Science is based on a huge deal of trust and violating that trust for your own short term gains is inexcusable. Even worse, any honest scientist will have find it more difficult to improve upon results that can be explained by overfitting. A good short description on several issues that plague the field is "Clever Methods of Overfitting". http://hunch.net/?p=22 http://hunch.net/?p=22