3 ms·
There is a fundamental difference between a scientific study and data-mining. Science is based on probability theory. Until we discover the "grand theory of ev
by tomp 15y ago
There is a fundamental difference between a scientific study and data-mining.
Science is based on probability theory. Until we discover the "grand theory of everything", out other theories will be only approximate, and out experimental results not 100% predictable. Therefore, scientists consider a prediction as correct if the chance of predicting something at random is less than some probability, usually 10%, 5% or 1%.
However, for this to work, each study must be based on new data. If you use the same data to check e.g. 10 predictions, each of which has 10% chance of happening even if incorrect, you will in average confirm 1 of your predictions, even if all are incorrect!
- _delirium 15y agoIt's also fairly easy to, even unwittingly, begin tuning some of your procedures to the data set (i.e. overfitting), even if you're employing the usual precautions, like cross-validation. As an extreme example, if we had one gigantic fixed data set and gave researchers 10 years to work on it, by the end of that 10 years there would be entire techniques at least partially specialized to that specific dataset, with poor generalization outside of it.
- lutorm 15y agoNo, if 10 predictions each have a 10% chance of a false positive, you will always have on average 1 false positive. Whether you test them on the same data or not doesn't matter (unless 10 predictions test the same thing, of course.) What you need is a control sample that you know should be negative, so you can actually measure the false positive rate. (But with a sufficiently large base sample, you can look for correlations in small subsamples and use the whole sample as a control.)
- tomp 15y ago> No, if 10 predictions each have a 10% chance of a false positive, you will always have on average 1 false positive. Whether you test them on the same data or not doesn't matter That's true. I should have said it differently. In fact, I'm not even sure that my understanding is correct. The problem with not using new data to test each new prediction is, that if a scientist wants to show A on data X, but data X doesn't confirm A, the scientist modifies A slightly and now tests A' on X, which is again rejected, and then modifies it again, testing A'' on X, and so on... until the data X actually confirms hypothesis A'''''''! That's the real problem - using data without a predefined plan for how you will use this data. In the above example, the data that you collected affected your decision-making process, so your results are not independent of the data (and thus not replicable!).
- lutorm 15y agoYes, this is correct. It can still be useful to look for things that warrant further study, but it won't be proof of something in and of itself.
- ericd 15y agoI was talking about discovering correlations which aid in potentially discovering relationships between behaviors and results. If you have a sample size which includes everyone in your study, that's as good as you can get on your confidence interval... you can use a different subset of the data for each experiment if you'd like, and that would be functionally equivalent to running individual experiments with a subset of the population each time.