5 ms·

Imagine if p <= 1.0 was considered "significant", wouldn't this cut down on publication bias? Shouldn't every study worth funding be worth publishing? They shou

by nonbel 9y ago

Imagine if p <= 1.0 was considered "significant", wouldn't this cut down on publication bias? Shouldn't every study worth funding be worth publishing? They should all be considered significant results. So I think this proposal is: 1) Not really addressing the issues behind the replication crisis 2) Doing the exact opposite of what should be done (raise the significance level)

jimrandomh 9y ago
"Significance" is a social concept currently serving two roles: for determining what is considered publishable, and for determining what is considered true. This is meant to modify the second one, not the first. (Publishability should depend on quality of methodology, and be determined at the pre-registration phase before results are in.)
- nonbel 9y ago
  First, I agree that publishability should be independent of the results. Do you know anyone who disagrees with this though? That seems to be the standard opinion, yet the situation of filtering by p-value persists. Also, in the usual case where the null hypothesis is "no difference", no one really cares about what the p-value is telling them anyway. They care "is my theory accurate/true", when their theory predicts something other than the null hypothesis. I figure we may as well set the significance level in these cases so that everyone "wins", and go on to use analytic techniques that actually address the question of interest.
tpeo 9y ago
What's the point of fitting arbitrary functions to noise? Because that's what high p-values would lead to. Undue emphasis on p-values is bad, but what you're suggesting is bonkers.
- nonbel 9y ago
  Stop funding people to fit arbitrary functions to noise?
  - tpeo 9y ago
    Don't be disingenuous. That would require someone to actually read their papers, which would already be a net loss and a massive waste of time for everyone involved. Not to mention the amount of department politics which could be involved in this issue. Look, not every paper is worth publishing. There are many ways to write noise that can pass for signal given the reader's level of understanding (e.g. crack pottery and "fashionable nonsense"). The usage of p-values implements self-filtering to some degree, and makes everyone better off by increasing the likelihood that at least some papers aren't complete nonsense. And that's good. The only issue is that it is a much weaker constraint on publishing than it was previously thought.
    - JoshTriplett 9y ago
      > Look, not every paper is worth publishing. Many more results should be documented than currently are, though, including negative results. But I agree that changing p-value thresholds is unrelated to that.
      - tpeo 9y ago
        I agree, publication bias is a thing. There's no questioning that.
    - Houshalter 9y ago
      And not publishing wastes everyone's time. When researches perform the same experiment again and again until someone gets a significant result to publish. There are entire fields of research supported by nothing but publication bias (e.g. Parapsychology and possibly most soft sciences.)

by nonbel 9y ago

jimrandomh 9y ago

"Significance" is a social concept currently serving two roles: for determining what is considered publishable, and for determining what is considered true. This is meant to modify the second one, not the first. (Publishability should depend on quality of methodology, and be determined at the pre-registration phase before results are in.)

nonbel 9y ago
First, I agree that publishability should be independent of the results. Do you know anyone who disagrees with this though? That seems to be the standard opinion, yet the situation of filtering by p-value persists. Also, in the usual case where the null hypothesis is "no difference", no one really cares about what the p-value is telling them anyway. They care "is my theory accurate/true", when their theory predicts something other than the null hypothesis. I figure we may as well set the significance level in these cases so that everyone "wins", and go on to use analytic techniques that actually address the question of interest.

tpeo 9y ago

What's the point of fitting arbitrary functions to noise? Because that's what high p-values would lead to. Undue emphasis on p-values is bad, but what you're suggesting is bonkers.

nonbel 9y ago
Stop funding people to fit arbitrary functions to noise?
- tpeo 9y ago
  Don't be disingenuous. That would require someone to actually read their papers, which would already be a net loss and a massive waste of time for everyone involved. Not to mention the amount of department politics which could be involved in this issue. Look, not every paper is worth publishing. There are many ways to write noise that can pass for signal given the reader's level of understanding (e.g. crack pottery and "fashionable nonsense"). The usage of p-values implements self-filtering to some degree, and makes everyone better off by increasing the likelihood that at least some papers aren't complete nonsense. And that's good. The only issue is that it is a much weaker constraint on publishing than it was previously thought.
  - JoshTriplett 9y ago
    > Look, not every paper is worth publishing. Many more results should be documented than currently are, though, including negative results. But I agree that changing p-value thresholds is unrelated to that.
    - tpeo 9y ago
      I agree, publication bias is a thing. There's no questioning that.
  - Houshalter 9y ago
    And not publishing wastes everyone's time. When researches perform the same experiment again and again until someone gets a significant result to publish. There are entire fields of research supported by nothing but publication bias (e.g. Parapsychology and possibly most soft sciences.)