3 ms·

Scientists needs to start using a "training set" and a "test set". If you have 2000 samples of data, you don't train your model on it and then say that's your

Scientists needs to start using a "training set" and a "test set". If you have 2000 samples of data, you don't train your model on it and then say that's your success rate. You'll end up with conclusions that don't generalize. Instead train on 1600 and measure your success on the remaining 400. Similarly, don't look for statistical significance among your 2000 samples and conclude that's the result. Do it across 1600 and then validate it on the 400. If there is a real result there, it'll reproduce. It now makes your process robust to overfitting / param hacking. You avoid the green jelly bean problem entirely.

iskander 9y ago
Science often works with data set sizes that make the standard ML approach unworkable (e.g. 12 measurements which took months and thousands of dollars to make)
- BoiledCabbage 9y ago
  The solution isn't proposed to use ML. It's simply an analogy. If you're performing statistical inference on sample sizes of 12 you can't then be surprised by a lack of predictability. And I'm pretty confident the majority of experiments are using sample sizes larger than 12.
  - ralmeida 9y ago
    I don't think iskander though you were advocating using ML, but generally saying that the approaches _usually_ used in ML would not work in some datasets which are way too small. A 12-point dataset is not desirable _per se_, but sometimes it's all one has - as is the case, for example in health-related research.
    - cwyers 9y ago
      In which case, you can't really afford train and test splits but you can look at approaches like leave-one-out to estimate what your sampling error looks like.
      - jpeloquin 9y ago
        And, more generally, bootstrapping, which is a class of methods for re-sampling one's data in order to estimate uncertainty. https://en.wikipedia.org/wiki/Bootstrapping_(statistics) https://en.wikipedia.org/wiki/Bootstrapping_(statistics)
        cwyers 9y ago
        I think leave one out is actually a special case of k-fold cross validation where k is equal to the number of datapoints. But yeah, you can use bootstrapping, cross validation, the jackknife. There's a large toolset there, and I trust an analyst who explains what motivated them to choose a certain tool rather than one who just uses a tool because it's what they always use or it's expected of them.
        deleted 9y ago
        [deleted]
  - iskander 9y ago
    >And I'm pretty confident the majority of experiments are using sample sizes larger than 12. It definitely varies by field, testing 3 doses and a negative control with 3x replication would be a pretty good experiment for testing a drug in mice. It's plenty of data for identifying a drug that actually works, as long as your statistical tools aren't designed with expectation of thousands of samples.
kbutler 9y ago
This is the standard recommendation, but does it really help? You do that test on the second subset, and then you discard every theory that doesn't pass both the 1600 set and the 400 set. So you end up with the predictions that pass all the data in your original 2000 samples. Is it really any better at generalizing to new data? If so, can you just evaluate your theories on the unpartitioned data by doing randomized subset testing after-the-fact?
- BoiledCabbage 9y ago
  Probably would be helpful to see the math on it to confirm, but I believe if it's a spurious relationship, the probability it will pass both the 1600 & the 400 are less than the prob of it passing the 2000.
  - ralmeida 9y ago
    It may (or may not) be less, but if scientists just discard the tests which don't pass both, then passing both becomes the new prior, and we go back to the same situation.
- cwyers 9y ago
  It depends on how often you go to the well. This is why things like Kaggle have limitations on how often you can submit entries for evaluation. This is why some studies go with a three-way split of train-calibration-test. And you can look at things like cross validation, boot-strapping etc. Nothing is perfect. As soon as you lock yourself into a one-size-fits-all way of validating your out-of-sample validity, you'll find lots of people who are optimizing for whatever test you come up with, not out-of-sample validity. It pays to pay attention to how well the researcher understands and deals with their particular set of problems, not how well they meet some arbitrary criteria. (Which is exactly how we ended up having all these problems with p-values! P-values aren't inherently bad, they're just applied indiscriminately because It's The Way We Do Things. If we start indiscriminately applying test/train splits we'll end up at a different place with its own set of problems. The chief problem is the lack of discrimination, not the specific tools under consideration.)
bunderbunder 9y ago
> You avoid the green jelly bean problem entirely. Not entirely. Quite often, the big fishing expedition research is done on things like epidemiological data sets, where you've got a million monkeys with a million typewriters trying to publish a million papers, all based on one data set. Under that kind of situation, assuming that everything works out perfectly, after the first pass you'll have some random non-negative number of hypotheses that were collected from fishing that you're going to test again. And those tests will also have a 5% type I error rate, so the fishing expedition will still have a problem with multiple comparisons. Your cumulative risk of a type I error will be mitigated, but not reduced to the traditional 5% (or whatever alpha you were shooting for). Meanwhile, your risk of a type II error will have been increased considerably: You're replacing one test of a relatively high statistical power with two tests of relatively lower power, and set up a situation where a type II error in either of them yields a type II error in the overall test. Not saying that there's no value to approaching things that way, but it's not the free lunch you're suggesting it is.

iskander 9y ago

Science often works with data set sizes that make the standard ML approach unworkable (e.g. 12 measurements which took months and thousands of dollars to make)

BoiledCabbage 9y ago
The solution isn't proposed to use ML. It's simply an analogy. If you're performing statistical inference on sample sizes of 12 you can't then be surprised by a lack of predictability. And I'm pretty confident the majority of experiments are using sample sizes larger than 12.
- ralmeida 9y ago
  I don't think iskander though you were advocating using ML, but generally saying that the approaches _usually_ used in ML would not work in some datasets which are way too small. A 12-point dataset is not desirable _per se_, but sometimes it's all one has - as is the case, for example in health-related research.
  - cwyers 9y ago
    In which case, you can't really afford train and test splits but you can look at approaches like leave-one-out to estimate what your sampling error looks like.
    - jpeloquin 9y ago
      And, more generally, bootstrapping, which is a class of methods for re-sampling one's data in order to estimate uncertainty. https://en.wikipedia.org/wiki/Bootstrapping_(statistics) https://en.wikipedia.org/wiki/Bootstrapping_(statistics)
      - cwyers 9y ago
        I think leave one out is actually a special case of k-fold cross validation where k is equal to the number of datapoints. But yeah, you can use bootstrapping, cross validation, the jackknife. There's a large toolset there, and I trust an analyst who explains what motivated them to choose a certain tool rather than one who just uses a tool because it's what they always use or it's expected of them.
      - deleted 9y ago
        [deleted]
- iskander 9y ago
  >And I'm pretty confident the majority of experiments are using sample sizes larger than 12. It definitely varies by field, testing 3 doses and a negative control with 3x replication would be a pretty good experiment for testing a drug in mice. It's plenty of data for identifying a drug that actually works, as long as your statistical tools aren't designed with expectation of thousands of samples.

kbutler 9y ago

This is the standard recommendation, but does it really help? You do that test on the second subset, and then you discard every theory that doesn't pass both the 1600 set and the 400 set. So you end up with the predictions that pass all the data in your original 2000 samples. Is it really any better at generalizing to new data? If so, can you just evaluate your theories on the unpartitioned data by doing randomized subset testing after-the-fact?

BoiledCabbage 9y ago
Probably would be helpful to see the math on it to confirm, but I believe if it's a spurious relationship, the probability it will pass both the 1600 & the 400 are less than the prob of it passing the 2000.
- ralmeida 9y ago
  It may (or may not) be less, but if scientists just discard the tests which don't pass both, then passing both becomes the new prior, and we go back to the same situation.
cwyers 9y ago
It depends on how often you go to the well. This is why things like Kaggle have limitations on how often you can submit entries for evaluation. This is why some studies go with a three-way split of train-calibration-test. And you can look at things like cross validation, boot-strapping etc. Nothing is perfect. As soon as you lock yourself into a one-size-fits-all way of validating your out-of-sample validity, you'll find lots of people who are optimizing for whatever test you come up with, not out-of-sample validity. It pays to pay attention to how well the researcher understands and deals with their particular set of problems, not how well they meet some arbitrary criteria. (Which is exactly how we ended up having all these problems with p-values! P-values aren't inherently bad, they're just applied indiscriminately because It's The Way We Do Things. If we start indiscriminately applying test/train splits we'll end up at a different place with its own set of problems. The chief problem is the lack of discrimination, not the specific tools under consideration.)

bunderbunder 9y ago

> You avoid the green jelly bean problem entirely. Not entirely. Quite often, the big fishing expedition research is done on things like epidemiological data sets, where you've got a million monkeys with a million typewriters trying to publish a million papers, all based on one data set. Under that kind of situation, assuming that everything works out perfectly, after the first pass you'll have some random non-negative number of hypotheses that were collected from fishing that you're going to test again. And those tests will also have a 5% type I error rate, so the fishing expedition will still have a problem with multiple comparisons. Your cumulative risk of a type I error will be mitigated, but not reduced to the traditional 5% (or whatever alpha you were shooting for). Meanwhile, your risk of a type II error will have been increased considerably: You're replacing one test of a relatively high statistical power with two tests of relatively lower power, and set up a situation where a type II error in either of them yields a type II error in the overall test. Not saying that there's no value to approaching things that way, but it's not the free lunch you're suggesting it is.