4 ms·
When I was first taught statistics, I was told that the researcher had to justify a plausible hypothesis first - and then do a hypothesis test/p-value to prove
by learnstats2 9y ago
When I was first taught statistics, I was told that the researcher had to justify a plausible hypothesis first - and then do a hypothesis test/p-value to prove their theory.
If this combination of the scientist's intuitive understanding and the p-value test result align, then this is a credible result.
On the other hand, the trend now is to conduct every possible test whether or not there is any justification for doing so (corrected for multiple testing, no p-hacking, yes, sure)
For example, in tech, we might test every shade of blue. Some of those blues are gonna come up as p-value hits - but since we had no good reason to do this test, this was probably just random noise.
Similarly, in genetics, we're gonna test every single gene against everything - just to see what happens (yes, yes, do a Bonferroni correction on each set of tests). Hmm, recent results in genetics don't seem to be very robust or repeatable, for some reason.
The likelihood of a truthful link in these tests is incredibly low. When have no particular reason to believe there is a truthful link, and are just blind testing, the false positive rate is very high (as described in the article), and probably even higher than the article speculates with - almost all hits are gonna be false positives.
Maybe p-values just don't work well with modern day data. Or, maybe, Big Data just doesn't contain information about mysterious, unexplored, and innovative correlations that we hope it does.
- platz 9y agoYes, Modern day data is not obtained from a controlled clinical trials
- kolpa 9y agoIf you run an experiment twice and the same shade of blue wins both times, that should persuade you that the winning shade is better. And if you keep replicating the experiment and it keeps winning, that should increase confidence further.
- nonbel 8y agoIt isn't so much that there is no "truthful link", it is that everything is linked to everything else to some degree and the mathematical models they use for the null hypothesis are just "defaults". These assumptions are almost always violated. The statistical tests detect that, and are providing "true positives".
- learnstats2 8y agoYes, this is kind of exactly the attitude change I am talking about. The “true positives” found in your model can be expected to have no predictive value, so what is the point of identifying these?
- deleted 8y ago[deleted]
- teej 8y ago“On the other hand, the trend now is to conduct every possible test whether or not there is any justification for doing so (corrected for multiple testing, no p-hacking, yes, sure)” You are literally describing p-hacking.
- Fomite 8y agoHe's describing a multiple comparisons problem, not p-hacking, enabled by, essentially, ease of statistical computation. An honest researcher can trigger his problem without ever p-hacking. See: Genome Wide Association Studies.