5 ms·

Science often works with data set sizes that make the standard ML approach unworkable (e.g. 12 measurements which took months and thousands of dollars to make)

BoiledCabbage 9y ago
The solution isn't proposed to use ML. It's simply an analogy. If you're performing statistical inference on sample sizes of 12 you can't then be surprised by a lack of predictability. And I'm pretty confident the majority of experiments are using sample sizes larger than 12.
- ralmeida 9y ago
  I don't think iskander though you were advocating using ML, but generally saying that the approaches _usually_ used in ML would not work in some datasets which are way too small. A 12-point dataset is not desirable _per se_, but sometimes it's all one has - as is the case, for example in health-related research.
  - cwyers 9y ago
    In which case, you can't really afford train and test splits but you can look at approaches like leave-one-out to estimate what your sampling error looks like.
    - jpeloquin 9y ago
      And, more generally, bootstrapping, which is a class of methods for re-sampling one's data in order to estimate uncertainty. https://en.wikipedia.org/wiki/Bootstrapping_(statistics) https://en.wikipedia.org/wiki/Bootstrapping_(statistics)
      - cwyers 9y ago
        I think leave one out is actually a special case of k-fold cross validation where k is equal to the number of datapoints. But yeah, you can use bootstrapping, cross validation, the jackknife. There's a large toolset there, and I trust an analyst who explains what motivated them to choose a certain tool rather than one who just uses a tool because it's what they always use or it's expected of them.
      - deleted 9y ago
        [deleted]
- iskander 9y ago
  >And I'm pretty confident the majority of experiments are using sample sizes larger than 12. It definitely varies by field, testing 3 doses and a negative control with 3x replication would be a pretty good experiment for testing a drug in mice. It's plenty of data for identifying a drug that actually works, as long as your statistical tools aren't designed with expectation of thousands of samples.

BoiledCabbage 9y ago

The solution isn't proposed to use ML. It's simply an analogy. If you're performing statistical inference on sample sizes of 12 you can't then be surprised by a lack of predictability. And I'm pretty confident the majority of experiments are using sample sizes larger than 12.

ralmeida 9y ago
I don't think iskander though you were advocating using ML, but generally saying that the approaches _usually_ used in ML would not work in some datasets which are way too small. A 12-point dataset is not desirable _per se_, but sometimes it's all one has - as is the case, for example in health-related research.
- cwyers 9y ago
  In which case, you can't really afford train and test splits but you can look at approaches like leave-one-out to estimate what your sampling error looks like.
  - jpeloquin 9y ago
    And, more generally, bootstrapping, which is a class of methods for re-sampling one's data in order to estimate uncertainty. https://en.wikipedia.org/wiki/Bootstrapping_(statistics) https://en.wikipedia.org/wiki/Bootstrapping_(statistics)
    - cwyers 9y ago
      I think leave one out is actually a special case of k-fold cross validation where k is equal to the number of datapoints. But yeah, you can use bootstrapping, cross validation, the jackknife. There's a large toolset there, and I trust an analyst who explains what motivated them to choose a certain tool rather than one who just uses a tool because it's what they always use or it's expected of them.
    - deleted 9y ago
      [deleted]
iskander 9y ago
>And I'm pretty confident the majority of experiments are using sample sizes larger than 12. It definitely varies by field, testing 3 doses and a negative control with 3x replication would be a pretty good experiment for testing a drug in mice. It's plenty of data for identifying a drug that actually works, as long as your statistical tools aren't designed with expectation of thousands of samples.