3 ms·

If the standard in Psychology research is the so-called p-value being less that 0.05, it means that they publish results whenever the chance of the findings bei

If the standard in Psychology research is the so-called p-value being less that 0.05, it means that they publish results whenever the chance of the findings being a coincidence is less than 5%. It stands to reason that 5% of the published results in Psychology will be coincidences and not real.

masterjack 12y ago
It is even worse than that when negative results are not published. This is because a study can keep getting repeated by different parties until they get that p-value. In other words, if you never publish a negative or inconclusive result (or stop studying once a conclusive result is published) then given enough time you /will/ get a positive result for every single plausible hypothesis.
beagle3 12y ago
It is actually much much worse. Let's do the math. 1,000 non-tenured professors do 20 experiments each. On average, each of them has one p<0.05 result that is actually a coincidence. They all rush through publication in an attempt to get tenure. Unless you have reason to believe some results are actually plausible, it's possible ALL published results are wrong. The 5% is the ratio between "experiments done" and "results that are not meaningful yet randomly deemed worthy of publication". It has no relation to the number of "results that are actually meaningful", which may outnumber the former 100:1 ... or, more likely, be outnumbered by them 100:1. I wish Neyman/Pearson theory was taught more widely, or at least its implication to the false-positive/false-negative tradeoff. Alas, that would mean a lot less published results and a lot less bragging/tenure rights.
analog31 12y ago
My limited understanding is that the simplistic formula for computing p-value assumes certain things about the lack of hidden correlations and effects. If those things are present and correctly taken into account, then the correct p-value would be much higher. Arbitrarily multiplying reported p-values by 10 would not seem to be too far from the mark.
capnrefsmmat 12y ago
That's exactly the opposite of what p values mean. Other replies addressed the publication bias problem, but even absent that, p < 0.05 usually implies a probability much higher than 5% that any single "significant" result is false: http://www.statisticsdonewrong.com/p-value.html http://www.statisticsdonewrong.com/p-value.html The p value describes the probability that a nonexistent effect will falsely be be called significant. You can't turn that around (without Bayes' theorem and a prior) to calculate the probability that the effect is nonexistent. To compound this, most studies are underpowered -- they don't collect enough data to detect the effect they're looking for. So a statistically insignificant result usually does not mean the effect does not exist.
- neumann 12y ago
  There is a great (cheesy) explanation of the statistical 'significance' of p-values: http://theconversation.com/the-problem-with-p-values-how-significant-are-they-really-20029 http://theconversation.com/the-problem-with-p-values-how-sig...

masterjack 12y ago

It is even worse than that when negative results are not published. This is because a study can keep getting repeated by different parties until they get that p-value. In other words, if you never publish a negative or inconclusive result (or stop studying once a conclusive result is published) then given enough time you /will/ get a positive result for every single plausible hypothesis.

beagle3 12y ago

It is actually much much worse. Let's do the math. 1,000 non-tenured professors do 20 experiments each. On average, each of them has one p<0.05 result that is actually a coincidence. They all rush through publication in an attempt to get tenure. Unless you have reason to believe some results are actually plausible, it's possible ALL published results are wrong. The 5% is the ratio between "experiments done" and "results that are not meaningful yet randomly deemed worthy of publication". It has no relation to the number of "results that are actually meaningful", which may outnumber the former 100:1 ... or, more likely, be outnumbered by them 100:1. I wish Neyman/Pearson theory was taught more widely, or at least its implication to the false-positive/false-negative tradeoff. Alas, that would mean a lot less published results and a lot less bragging/tenure rights.

analog31 12y ago

My limited understanding is that the simplistic formula for computing p-value assumes certain things about the lack of hidden correlations and effects. If those things are present and correctly taken into account, then the correct p-value would be much higher. Arbitrarily multiplying reported p-values by 10 would not seem to be too far from the mark.

capnrefsmmat 12y ago

That's exactly the opposite of what p values mean. Other replies addressed the publication bias problem, but even absent that, p < 0.05 usually implies a probability much higher than 5% that any single "significant" result is false: http://www.statisticsdonewrong.com/p-value.html http://www.statisticsdonewrong.com/p-value.html The p value describes the probability that a nonexistent effect will falsely be be called significant. You can't turn that around (without Bayes' theorem and a prior) to calculate the probability that the effect is nonexistent. To compound this, most studies are underpowered -- they don't collect enough data to detect the effect they're looking for. So a statistically insignificant result usually does not mean the effect does not exist.

neumann 12y ago
There is a great (cheesy) explanation of the statistical 'significance' of p-values: http://theconversation.com/the-problem-with-p-values-how-significant-are-they-really-20029 http://theconversation.com/the-problem-with-p-values-how-sig...