3 ms·

Totally agree with Timothy Bates here: > [Bates] called the proposal “a risky distraction”... [It] wouldn’t address many other practices linked to irreproducib

Totally agree with Timothy Bates here: > [Bates] called the proposal “a risky distraction”... [It] wouldn’t address many other practices linked to irreproducible results: poor study design, a bias toward publishing positive results, and the practice of “p-value hacking”—fishing for significant-looking results from a huge number of hypotheses. Since the reproducibilty crisis started getting attention, I've noticed that these problems are really ubiquitous. Tweaking study designs and modifying hypotheses after collecting data are very commonplace and much more subtly dangerous than a potential lack of statistical power. Gelman says "Valid p-values cannot be drawn without knowing, not just what was done with the existing data, but what the choices in data coding, exclusion,and analysis would have been, had the data been different." [1] This proposal also says nothing about effect sizes, which also should play a role in how we interpret scientific results. We should probably draw different conclusions given a tiny effect at p=0.05 versus a large, obvious effect at p=0.05. [1]http://www.stat.columbia.edu/~gelman/research/published/asa_pvalues.pdf http://www.stat.columbia.edu/~gelman/research/published/asa_...

LoSboccacc 9y ago
P-values, reproducibility and the focus on statistics before understanding is, of itself, a problem: https://xkcd.com/882/ https://xkcd.com/882/
petters 9y ago
> We should probably draw different conclusions given a tiny effect at p=0.05 versus a large, obvious effect at p=0.05 I'm not sure that "large, obvious effect" and "p=0.05" are compatible, though.
- shmageggy 9y ago
  Sure it is, of course depending on your domain-dependent definition of "large, obvious". You can have different effect sizes at any given p-value.
mabbo 9y ago
I disagree. > a bias toward publishing positive results ... fishing for significant-looking results from a huge number of hypotheses If we accept a lower p-value as significant, there will be fewer hypotheses that have positive results, and therefore less to publish. Should we really do nothing at all until we can come up with a perfect solution to all aspects of the problem? This won't fix the real problem, but it will reduce the number of pop-sci articles spouting bullshit results that don't reproduce next week.
- shmageggy 9y ago
  > it will reduce the number of pop-sci articles spouting bullshit results that don't reproduce next week. I agree that doing something is better than doing nothing, but I disagree that this will reduce bad results in a meaningful way. Setting a more stringent threshold for significance will elimnate studies that reach the wrong conclusion due to a statistical fluke, i.e. there's no effect but due to random chance, the noise in the data makes it look like there is. But IMO these are relatively rare. It seems to me that the vast majority of bad studies reach significance by virtue of methodological issues that enable one to reach any arbitrary level of significance just by trying hard enough. If Daryl Bem continued doing what he was doing, he would have eventually reached p<0.005. edit: I just glanced over Bem's paper and he reports several p-values under 0.005 and several other very close to it.

LoSboccacc 9y ago

P-values, reproducibility and the focus on statistics before understanding is, of itself, a problem: https://xkcd.com/882/ https://xkcd.com/882/

petters 9y ago

> We should probably draw different conclusions given a tiny effect at p=0.05 versus a large, obvious effect at p=0.05 I'm not sure that "large, obvious effect" and "p=0.05" are compatible, though.

shmageggy 9y ago
Sure it is, of course depending on your domain-dependent definition of "large, obvious". You can have different effect sizes at any given p-value.

mabbo 9y ago

I disagree. > a bias toward publishing positive results ... fishing for significant-looking results from a huge number of hypotheses If we accept a lower p-value as significant, there will be fewer hypotheses that have positive results, and therefore less to publish. Should we really do nothing at all until we can come up with a perfect solution to all aspects of the problem? This won't fix the real problem, but it will reduce the number of pop-sci articles spouting bullshit results that don't reproduce next week.

shmageggy 9y ago
> it will reduce the number of pop-sci articles spouting bullshit results that don't reproduce next week. I agree that doing something is better than doing nothing, but I disagree that this will reduce bad results in a meaningful way. Setting a more stringent threshold for significance will elimnate studies that reach the wrong conclusion due to a statistical fluke, i.e. there's no effect but due to random chance, the noise in the data makes it look like there is. But IMO these are relatively rare. It seems to me that the vast majority of bad studies reach significance by virtue of methodological issues that enable one to reach any arbitrary level of significance just by trying hard enough. If Daryl Bem continued doing what he was doing, he would have eventually reached p<0.005. edit: I just glanced over Bem's paper and he reports several p-values under 0.005 and several other very close to it.