3 ms·

Suppose you do this, try some different models, get the error down on the training sample. Then you apply to the test sample and the results are ... about the

Suppose you do this, try some different models, get the error down on the training sample. Then you apply to the test sample and the results are ... about the same as the well known results. The best practice says you should now quit and move on to another project, and either publish a negative result or don't bother publishing at all. Human nature, and publish-or-perish rewards, mean the researcher will say "well, what if I try this other model? Or tweak this parameter, or ..." In other words, it's unrealistic to say you have one shot to confirm that have, indeed, solved it.

zamfi 4y ago
Not unrealistic -- it just causes people to be very, very careful about not fooling themselves. Do you really believe that this practice is unrealistic because anyone with a model that doesn't do well on the held-out set will simply engage in fraud? That is a very, very cynical take. FWIW, I have published this way [0]. There were definitely some aspects of the model's performance on the held-out set that did not meet test set performance; it's not fatal to the paper, it's an invitation to discuss. [0] https://arxiv.org/pdf/2205.07333.pdf https://arxiv.org/pdf/2205.07333.pdf
- YeGoblynQueenne 4y ago
  (Not the person you're replying to) Fraud is not needed. My intuition is that most people publishing in machine learning have simply not thought carefully enough about their experimental methods. They're not defrauding anyone, they're misunderstanding their own results. And since the majority of published papers do the same, it's very difficult to convince anyone what the good practice is. Good on you for doing it the right way.
  - zamfi 4y ago
    Oh, we're on the same page -- I don't think they're engaging in fraud currently, just overlooking the implications of their process, as you note. But explicitly holding out a test set, throwing it away because your results on that set were no good, trying again, and then representing that held-out set as, indeed, held-out in later work -- as the parent suggested? I'm not sure what other word to use there.

zamfi 4y ago

Not unrealistic -- it just causes people to be very, very careful about not fooling themselves. Do you really believe that this practice is unrealistic because anyone with a model that doesn't do well on the held-out set will simply engage in fraud? That is a very, very cynical take. FWIW, I have published this way [0]. There were definitely some aspects of the model's performance on the held-out set that did not meet test set performance; it's not fatal to the paper, it's an invitation to discuss. [0] https://arxiv.org/pdf/2205.07333.pdf https://arxiv.org/pdf/2205.07333.pdf

YeGoblynQueenne 4y ago
(Not the person you're replying to) Fraud is not needed. My intuition is that most people publishing in machine learning have simply not thought carefully enough about their experimental methods. They're not defrauding anyone, they're misunderstanding their own results. And since the majority of published papers do the same, it's very difficult to convince anyone what the good practice is. Good on you for doing it the right way.
- zamfi 4y ago
  Oh, we're on the same page -- I don't think they're engaging in fraud currently, just overlooking the implications of their process, as you note. But explicitly holding out a test set, throwing it away because your results on that set were no good, trying again, and then representing that held-out set as, indeed, held-out in later work -- as the parent suggested? I'm not sure what other word to use there.