4 ms·

...20 researchers try to study something. But only one of them gets statistically significant results (p < 0.05), so they publish and the rest don't. http://x

...20 researchers try to study something. But only one of them gets statistically significant results (p < 0.05), so they publish and the rest don't. http://xkcd.com/882/ http://xkcd.com/882/ If we don't address this, then the whole research publication process is on fire.

throw_away_777 10y ago
The easiest way to address this is to simply move the p value necessary for significant results - instead of p < 0.05 use p < 0.001 or even p < 0.0000003 (3 x 10^-7), as is needed in particle physics. Though the 5 sigma threshold is a bit ridiculous. Lowering the p-value threshold significantly increases the cost of hunting for positives due to random chance, at the expense of needing more sensitive experiments.
- jjoonathan 10y ago
  Nope. It just means that you have to come up with a shittier null hypothesis. Of course, it still has to get past peer review, but this can often be accomplished by complication and obfuscation, which have the side benefit of giving you plausible deniability in case you are discovered. Sufficiently advanced cluelessness is indistinguishable from malice, and science, in its present form, rewards them both.
Balgair 10y ago
In a lot of fields, you just can't do this. We have to remember that a single article should not be taken as gospel, but more as a step in a direction, like a single log entry in a captain's journal. "We saw X happen Y many times in a 1000+Z step process with ABC cells in FGH media at P temperature at N phase of the moon. Our p value is being required by the journal, but is meaningless because we are just reporting that you should look out for this thing, if it exists at all"
- btilly 10y ago
  In a lot of fields you can. In fact an entire field (ego depletion) seems to have been ENTIRELY produced by this type of publication selection bias.
  - BeetleB 10y ago
    Interesting - I did not know that. Looking at the criticism section in the Wikipedia article, your claim is as suspect as ego depletion :-) It seems the jury is still out.
    - btilly 10y ago
      My selecting that as an example is based on http://www.slate.com/articles/health_and_science/cover_story/2016/03/ego_depletion_an_influential_theory_in_psychology_may_have_just_been_debunked.html http://www.slate.com/articles/health_and_science/cover_story.... My phrasing "seems to have been" reflects both my opinion, and the fact that it is far from certain. That said, regardless of what happens with this specific theory, it is well worth reading Richard Feynman's remarks from 42 years ago in http://calteches.library.caltech.edu/51/2/CargoCult.htm http://calteches.library.caltech.edu/51/2/CargoCult.htm. Until a field of research can demonstrate the kind of integrity that Feynman talks about there, nobody should believe what they claim. They can talk all that they want about how much work they did, the publication history, the supporting evidence - it is all BS until you have that. Increasingly people call themselves "scientists" and call what they are doing "science" while lacking that fundamental foundation. The result is that it is becoming harder and harder to tell which claims about reality deserve provisional belief versus healthy skepticism.

throw_away_777 10y ago

The easiest way to address this is to simply move the p value necessary for significant results - instead of p < 0.05 use p < 0.001 or even p < 0.0000003 (3 x 10^-7), as is needed in particle physics. Though the 5 sigma threshold is a bit ridiculous. Lowering the p-value threshold significantly increases the cost of hunting for positives due to random chance, at the expense of needing more sensitive experiments.

jjoonathan 10y ago
Nope. It just means that you have to come up with a shittier null hypothesis. Of course, it still has to get past peer review, but this can often be accomplished by complication and obfuscation, which have the side benefit of giving you plausible deniability in case you are discovered. Sufficiently advanced cluelessness is indistinguishable from malice, and science, in its present form, rewards them both.

Balgair 10y ago

In a lot of fields, you just can't do this. We have to remember that a single article should not be taken as gospel, but more as a step in a direction, like a single log entry in a captain's journal. "We saw X happen Y many times in a 1000+Z step process with ABC cells in FGH media at P temperature at N phase of the moon. Our p value is being required by the journal, but is meaningless because we are just reporting that you should look out for this thing, if it exists at all"

btilly 10y ago
In a lot of fields you can. In fact an entire field (ego depletion) seems to have been ENTIRELY produced by this type of publication selection bias.
- BeetleB 10y ago
  Interesting - I did not know that. Looking at the criticism section in the Wikipedia article, your claim is as suspect as ego depletion :-) It seems the jury is still out.
  - btilly 10y ago
    My selecting that as an example is based on http://www.slate.com/articles/health_and_science/cover_story/2016/03/ego_depletion_an_influential_theory_in_psychology_may_have_just_been_debunked.html http://www.slate.com/articles/health_and_science/cover_story.... My phrasing "seems to have been" reflects both my opinion, and the fact that it is far from certain. That said, regardless of what happens with this specific theory, it is well worth reading Richard Feynman's remarks from 42 years ago in http://calteches.library.caltech.edu/51/2/CargoCult.htm http://calteches.library.caltech.edu/51/2/CargoCult.htm. Until a field of research can demonstrate the kind of integrity that Feynman talks about there, nobody should believe what they claim. They can talk all that they want about how much work they did, the publication history, the supporting evidence - it is all BS until you have that. Increasingly people call themselves "scientists" and call what they are doing "science" while lacking that fundamental foundation. The result is that it is becoming harder and harder to tell which claims about reality deserve provisional belief versus healthy skepticism.