4 ms·

Best practice here is to keep a chunk of your dataset held out from your entire development process (a "held out test set"), and then only evaluate it when you

by zamfi 4y ago

Best practice here is to keep a chunk of your dataset held out from your entire development process (a "held out test set"), and then only evaluate it when you write your final report on your work. The idea is that you don't even look at this data until you're confident you've solved the problem, and then you have one shot to confirm that you have, indeed, solved it. Obviously you have to be careful in selecting this set -- if you're working with time-series, for example, you want it to be temporally disjoint -- but done well it's a reasonable approximation for "future results". About as good as you can get without collecting new data. I can't tell if they did that here, but if they did, they didn't use the common jargon for it.

dr_dshiv 4y ago
> About as good as you can get without collecting new data. So let’s call holdout datasets “second-best practice”?
- zamfi 4y ago
  LOL, love it! Sadly too common that collecting new data is practically speaking impossible, because the actual collection process is underspecified.
martincmartin 4y ago
Suppose you do this, try some different models, get the error down on the training sample. Then you apply to the test sample and the results are ... about the same as the well known results. The best practice says you should now quit and move on to another project, and either publish a negative result or don't bother publishing at all. Human nature, and publish-or-perish rewards, mean the researcher will say "well, what if I try this other model? Or tweak this parameter, or ..." In other words, it's unrealistic to say you have one shot to confirm that have, indeed, solved it.
- zamfi 4y ago
  Not unrealistic -- it just causes people to be very, very careful about not fooling themselves. Do you really believe that this practice is unrealistic because anyone with a model that doesn't do well on the held-out set will simply engage in fraud? That is a very, very cynical take. FWIW, I have published this way [0]. There were definitely some aspects of the model's performance on the held-out set that did not meet test set performance; it's not fatal to the paper, it's an invitation to discuss. [0] https://arxiv.org/pdf/2205.07333.pdf https://arxiv.org/pdf/2205.07333.pdf
  - YeGoblynQueenne 4y ago
    (Not the person you're replying to) Fraud is not needed. My intuition is that most people publishing in machine learning have simply not thought carefully enough about their experimental methods. They're not defrauding anyone, they're misunderstanding their own results. And since the majority of published papers do the same, it's very difficult to convince anyone what the good practice is. Good on you for doing it the right way.
    - zamfi 4y ago
      Oh, we're on the same page -- I don't think they're engaging in fraud currently, just overlooking the implications of their process, as you note. But explicitly holding out a test set, throwing it away because your results on that set were no good, trying again, and then representing that held-out set as, indeed, held-out in later work -- as the parent suggested? I'm not sure what other word to use there.
muraiki 4y ago
"The training, validation and test data sets are generated without overlap from periods in sequence. Successive periods of 400, 12, 40, 40 and 12 h are used to sample, respectively, training, validation, and test data, with the two 12 h periods inserted as hiatus."
- zamfi 4y ago
  What's missing is that they did not use the test set during the training process. A common split is train/validate/test, but all three are used during training -- train to actually train, validate for intermediate loss, test for model comparison. What you want is a fourth, held-out test set that isn't looked at until publish time. This paper has two test sets, but they have different data properties, and it's not clear they were held out until publication.
  - whatshisface 4y ago
    I can't believe you can publish with such an obvious flaw in study design. You don't have to have machine learning expertise to catch that, because the same idea applies to backtesting any kind of model at all.
    - YeGoblynQueenne 4y ago
      And yet they do. I don't remember ever reading any machine learning paper where the authors point out that they held out a test partition that they couldn't access until some "final" step of the experiment.
YeGoblynQueenne 4y ago
You're absolutely right, but the problem is that very few people do that, and that the result is we don't really know what result we can trust.
- akkartik 4y ago
  I don't understand why there is any doubt about this. From OP, section "Methods", subset "Data set creation and splits": "The training, validation and test data sets are generated without overlap from periods in sequence. Successive periods of 400, 12, 40, 40 and 12 h are used to sample, respectively, training, validation, and test data, with the two 12 h periods inserted as hiatus." I know nothing about machine learning, but I was in grad school with others studying machine learning and I took a Coursera course on the subject, and carving out a training set is utterly standard practice. A paper that didn't do this would get filtered by a grad student reviewer. So I'm not sure what possible sub-category of machine learning your statements could apply to. Can you share any examples? For example, do any of the famous papers in the field fail to hold out test data? Something like AlphaGo or AlphaZero is indeed testing on all new data -- new games it plays with others. Do any papers in well-known machine learning fora fail to hold out test data? Does anything that has bubbled to the front page of HN? It would be interesting to look at if so. Do you mean p-hacking? That's a much more subtle thing.
  - zamfi 4y ago
    Very few ML papers use a held-out test set that is only used for publication. When you use your test set as part of model selection, you don't really have an empirical test set -- you do have a training set that you've partitioned into various parts for various purposes, but you don't have a proxy for how the system will perform in the face of new data, because you're reporting results that are based on data that you used as part of your model-generating process. If you try out models A, B, C, and D while developing, and then it turns out model D has the best performance on your test set, and you report your results using model D and that same test set, you are essentially "training" using your test set. Yes, in some problem contexts, for ML researchers who are also in the business of generating their own datasets, the data they test on can be entirely new. But for anyone using a pre-existing dataset, holding out a test set for your project is not yet a well-established practice -- and this makes it somewhat harder to trust the results.

by zamfi 4y ago

dr_dshiv 4y ago

> About as good as you can get without collecting new data. So let’s call holdout datasets “second-best practice”?

zamfi 4y ago
LOL, love it! Sadly too common that collecting new data is practically speaking impossible, because the actual collection process is underspecified.

martincmartin 4y ago

Suppose you do this, try some different models, get the error down on the training sample. Then you apply to the test sample and the results are ... about the same as the well known results. The best practice says you should now quit and move on to another project, and either publish a negative result or don't bother publishing at all. Human nature, and publish-or-perish rewards, mean the researcher will say "well, what if I try this other model? Or tweak this parameter, or ..." In other words, it's unrealistic to say you have one shot to confirm that have, indeed, solved it.

zamfi 4y ago
Not unrealistic -- it just causes people to be very, very careful about not fooling themselves. Do you really believe that this practice is unrealistic because anyone with a model that doesn't do well on the held-out set will simply engage in fraud? That is a very, very cynical take. FWIW, I have published this way [0]. There were definitely some aspects of the model's performance on the held-out set that did not meet test set performance; it's not fatal to the paper, it's an invitation to discuss. [0] https://arxiv.org/pdf/2205.07333.pdf https://arxiv.org/pdf/2205.07333.pdf
- YeGoblynQueenne 4y ago
  (Not the person you're replying to) Fraud is not needed. My intuition is that most people publishing in machine learning have simply not thought carefully enough about their experimental methods. They're not defrauding anyone, they're misunderstanding their own results. And since the majority of published papers do the same, it's very difficult to convince anyone what the good practice is. Good on you for doing it the right way.
  - zamfi 4y ago
    Oh, we're on the same page -- I don't think they're engaging in fraud currently, just overlooking the implications of their process, as you note. But explicitly holding out a test set, throwing it away because your results on that set were no good, trying again, and then representing that held-out set as, indeed, held-out in later work -- as the parent suggested? I'm not sure what other word to use there.

muraiki 4y ago

"The training, validation and test data sets are generated without overlap from periods in sequence. Successive periods of 400, 12, 40, 40 and 12 h are used to sample, respectively, training, validation, and test data, with the two 12 h periods inserted as hiatus."

zamfi 4y ago
What's missing is that they did not use the test set during the training process. A common split is train/validate/test, but all three are used during training -- train to actually train, validate for intermediate loss, test for model comparison. What you want is a fourth, held-out test set that isn't looked at until publish time. This paper has two test sets, but they have different data properties, and it's not clear they were held out until publication.
- whatshisface 4y ago
  I can't believe you can publish with such an obvious flaw in study design. You don't have to have machine learning expertise to catch that, because the same idea applies to backtesting any kind of model at all.
  - YeGoblynQueenne 4y ago
    And yet they do. I don't remember ever reading any machine learning paper where the authors point out that they held out a test partition that they couldn't access until some "final" step of the experiment.

YeGoblynQueenne 4y ago

You're absolutely right, but the problem is that very few people do that, and that the result is we don't really know what result we can trust.

akkartik 4y ago
I don't understand why there is any doubt about this. From OP, section "Methods", subset "Data set creation and splits": "The training, validation and test data sets are generated without overlap from periods in sequence. Successive periods of 400, 12, 40, 40 and 12 h are used to sample, respectively, training, validation, and test data, with the two 12 h periods inserted as hiatus." I know nothing about machine learning, but I was in grad school with others studying machine learning and I took a Coursera course on the subject, and carving out a training set is utterly standard practice. A paper that didn't do this would get filtered by a grad student reviewer. So I'm not sure what possible sub-category of machine learning your statements could apply to. Can you share any examples? For example, do any of the famous papers in the field fail to hold out test data? Something like AlphaGo or AlphaZero is indeed testing on all new data -- new games it plays with others. Do any papers in well-known machine learning fora fail to hold out test data? Does anything that has bubbled to the front page of HN? It would be interesting to look at if so. Do you mean p-hacking? That's a much more subtle thing.
- zamfi 4y ago
  Very few ML papers use a held-out test set that is only used for publication. When you use your test set as part of model selection, you don't really have an empirical test set -- you do have a training set that you've partitioned into various parts for various purposes, but you don't have a proxy for how the system will perform in the face of new data, because you're reporting results that are based on data that you used as part of your model-generating process. If you try out models A, B, C, and D while developing, and then it turns out model D has the best performance on your test set, and you report your results using model D and that same test set, you are essentially "training" using your test set. Yes, in some problem contexts, for ML researchers who are also in the business of generating their own datasets, the data they test on can be entirely new. But for anyone using a pre-existing dataset, holding out a test set for your project is not yet a well-established practice -- and this makes it somewhat harder to trust the results.