5 ms·

I think the consensus on why this happens boils down to regression to the mean, selective experimentation (exciting things over boring things like attempting to

by ims 13y ago

I think the consensus on why this happens boils down to regression to the mean, selective experimentation (exciting things over boring things like attempting to duplicate results), and selective publishing (e.g. positive results are much more interesting to journals than hypotheses being rejected). Did I miss any reasons?

stefantalpalaru 13y ago
Unknowable chance events and hidden variables (John Crabbe's experiments towards the end of the article).
ploomans 13y ago
Plus if you have a million researches in your area of research a 1 per million chance your results are wrong is still no assurance. Should statistically happen to one of the researchers in your area. Very good article by Warren Buffet that touches on the same issue: http://www.tilsonfunds.com/superinvestors.html http://www.tilsonfunds.com/superinvestors.html
rubidium 13y ago
Just one. Financial incentive to obtain positive results. Especially in the drug market.
Jach 13y ago
I'd add bad math. Fisher significance testing is on shaky grounds[0] both theoretically (significance testing violates the likelihood principle) and practically (just by nature of having a small sample size, a lot of null hypotheses can be rejected, while with a larger sample size hardly none are rejected). Hypothesis testing is little better. The "Bayesian revolution" has barely begun and has yet to influence a majority of researchers. [0] http://uncertainty.stat.cmu.edu/ http://uncertainty.stat.cmu.edu/ Chapter 12.
- simonster 13y ago
  I don't think this is quite right. If the null hypothesis is true and you conduct the same study an infinite number of times, you will reject the null hypothesis 5% of the time at alpha = 0.05 regardless of sample size, unless there is something wrong with your sampling procedure or hypothesis test. This is the point of null hypothesis significance testing. In practice, if you fix p=0.05 and increase your n, the probability that you will find a statistically significant result often increases because your power increases, and in many situations, the probability that the null hypothesis is true is close to zero. (Andrew Gelman uses the example of asking whether there are significant differences between voting patterns of men and women.) On the other hand, effect size estimates become more accurate as the sample size increases. This mitigates the above issue, provided you actually report your effect size. It also means that small sample studies that report statistically significant results are more likely to overestimate their effect size, which is especially problematic if you are applying null hypothesis significance testing when you know the null hypothesis is false.

by ims 13y ago

stefantalpalaru 13y ago

Unknowable chance events and hidden variables (John Crabbe's experiments towards the end of the article).

ploomans 13y ago

Plus if you have a million researches in your area of research a 1 per million chance your results are wrong is still no assurance. Should statistically happen to one of the researchers in your area. Very good article by Warren Buffet that touches on the same issue: http://www.tilsonfunds.com/superinvestors.html http://www.tilsonfunds.com/superinvestors.html

rubidium 13y ago

Just one. Financial incentive to obtain positive results. Especially in the drug market.

Jach 13y ago

I'd add bad math. Fisher significance testing is on shaky grounds[0] both theoretically (significance testing violates the likelihood principle) and practically (just by nature of having a small sample size, a lot of null hypotheses can be rejected, while with a larger sample size hardly none are rejected). Hypothesis testing is little better. The "Bayesian revolution" has barely begun and has yet to influence a majority of researchers. [0] http://uncertainty.stat.cmu.edu/ http://uncertainty.stat.cmu.edu/ Chapter 12.

simonster 13y ago
I don't think this is quite right. If the null hypothesis is true and you conduct the same study an infinite number of times, you will reject the null hypothesis 5% of the time at alpha = 0.05 regardless of sample size, unless there is something wrong with your sampling procedure or hypothesis test. This is the point of null hypothesis significance testing. In practice, if you fix p=0.05 and increase your n, the probability that you will find a statistically significant result often increases because your power increases, and in many situations, the probability that the null hypothesis is true is close to zero. (Andrew Gelman uses the example of asking whether there are significant differences between voting patterns of men and women.) On the other hand, effect size estimates become more accurate as the sample size increases. This mitigates the above issue, provided you actually report your effect size. It also means that small sample studies that report statistically significant results are more likely to overestimate their effect size, which is especially problematic if you are applying null hypothesis significance testing when you know the null hypothesis is false.