4 ms·

The graph shows that those low p-values are more likely to be in papers, not that they’re more likely to occur. Is that suspicious? I don’t know enough about it

The graph shows that those low p-values are more likely to be in papers, not that they’re more likely to occur. Is that suspicious? I don’t know enough about it to judge.

thaumasiotes 5y ago
> The graph shows that those low p-values are more likely to be in papers, not that they’re more likely to occur. It looks to me like the y-axis is measured in number of papers. The lower a p-value is, the more papers there are that happened to find a result beating the p-value. So low p-values are more likely to occur a priori than high p-values are. This is most certainly not true in general. We might guess that psychologists are fudging their p-values somehow, or that journals are much, much, much, much, much, much, much more likely to publish "chewing a stalk of grass makes you walk slower, p < 0.013" than they are to publish "chewing a stalk of grass makes you walk slower, p < 0.04". I've emphasized the level of bias the journals would need to be showing -- over fine distinctions in a value that is most often treated as a binary yes or no -- because it is much easier to get p < 0.04 than it is to get p < 0.013.
- disgruntledphd2 5y ago
  Conditional on being published, this is true. Hence studies of the file-drawer effect and what not. More generally, scientists are incentivised to find novel findings (i.e. unexpectedly low p-values) or lose their job. Given that, the plot doesn't surprise me at all (Also, people will normally not report a bunch of non-significant results, which is a similar but unrelated problem).
  - eru 5y ago
    That's part of why pre-registration of studies is so important.
- mannykannot 5y ago
  > This is most certainly not true in general. Are you saying that in other disciplines, the distribution of p-values in published papers does not follow this pattern?
  - disgruntledphd2 5y ago
    I think what they meant was that we would expect the distribution of p-values to be uniform, if we had access to every p-value ever calculated (or a random sample thereof). Publishing introduces a systematic bias, because it's difficult to get published where p>0.05 (or whatever the disciplinary standard is).
    - thaumasiotes 5y ago
      > Publishing introduces a systematic bias, because it's difficult to get published where p>0.05 (or whatever the disciplinary standard is). That explains why the p-values above 0.05 are rare compared to values below 0.05. But it fails to explain why p-values above 0.02 are rare compared to values below 0.02.
      - mannykannot 5y ago
        I agree with your point from your previous post, that lower p-values are harder to get than higher ones, at least if one is looking at all possible causal relationships, but there are at least two possible causes for the inversion seen in publishing. The first is a general preference for lower p-values on the part of publishers and their reviewers (by 'general' I mean not just at the 0.05 value); the second is that researchers do not randomly pick what to study - they use their expertise and existing knowledge to guide their investigations. Is that enough to tip the curve the other way across the range of p-values? Well, something is, and I am open to alternative suggestions. One other point: while the datum immediately below 0.05 would normally be considered an outlier, the fact that it is next to a discontinuity (actual or perceived) renders that call less clear. Personally, I suspect it is not an accidental outlier, but given that it does not produce much distortion in the overall trend, I am less inclined to see the 0.05 threshold (actual or perceived) as a problem than I did before I saw this chart.
setgree 5y ago
> The graph shows that those low p-values are more likely to be in papers This is an important distinction, in my experience [0]. Many papers will report a p-value only if it is below a significance threshold, otherwise they will report "n.s." (no statistic) or will give a range (e.g. p > .1). This just means that in addition to pressure to shelve insignificant results, publication bias also manifests as a tendency to emphasize and carefully report significant findings, while mentioning in passing those that don't meet whatever threshold. [0] I happen to be working on a meta-analysis of psychology and public health papers at the moment. One paper that we're reviewing constructs 32 separate statistical models, reports that many of the results are not significant, and then discusses the significant results at length.
- deleted 5y ago
  [deleted]
- thaumasiotes 5y ago
  > Many papers will report a p-value only if it is below a significance threshold, otherwise they will report "n.s." (no statistic) or will give a range (e.g. p > .1). But the oddity here is a pronounced trend in the reported p-values that meet the significance threshold. The behavior you mention cannot create that trend.

The graph shows that those low p-values are more likely to be in papers, not that they’re more likely to occur. Is that suspicious? I don’t know enough about it to judge.

thaumasiotes 5y ago

> The graph shows that those low p-values are more likely to be in papers, not that they’re more likely to occur. It looks to me like the y-axis is measured in number of papers. The lower a p-value is, the more papers there are that happened to find a result beating the p-value. So low p-values are more likely to occur a priori than high p-values are. This is most certainly not true in general. We might guess that psychologists are fudging their p-values somehow, or that journals are much, much, much, much, much, much, much more likely to publish "chewing a stalk of grass makes you walk slower, p < 0.013" than they are to publish "chewing a stalk of grass makes you walk slower, p < 0.04". I've emphasized the level of bias the journals would need to be showing -- over fine distinctions in a value that is most often treated as a binary yes or no -- because it is much easier to get p < 0.04 than it is to get p < 0.013.

disgruntledphd2 5y ago
Conditional on being published, this is true. Hence studies of the file-drawer effect and what not. More generally, scientists are incentivised to find novel findings (i.e. unexpectedly low p-values) or lose their job. Given that, the plot doesn't surprise me at all (Also, people will normally not report a bunch of non-significant results, which is a similar but unrelated problem).
- eru 5y ago
  That's part of why pre-registration of studies is so important.
mannykannot 5y ago
> This is most certainly not true in general. Are you saying that in other disciplines, the distribution of p-values in published papers does not follow this pattern?
- disgruntledphd2 5y ago
  I think what they meant was that we would expect the distribution of p-values to be uniform, if we had access to every p-value ever calculated (or a random sample thereof). Publishing introduces a systematic bias, because it's difficult to get published where p>0.05 (or whatever the disciplinary standard is).
  - thaumasiotes 5y ago
    > Publishing introduces a systematic bias, because it's difficult to get published where p>0.05 (or whatever the disciplinary standard is). That explains why the p-values above 0.05 are rare compared to values below 0.05. But it fails to explain why p-values above 0.02 are rare compared to values below 0.02.
    - mannykannot 5y ago
      I agree with your point from your previous post, that lower p-values are harder to get than higher ones, at least if one is looking at all possible causal relationships, but there are at least two possible causes for the inversion seen in publishing. The first is a general preference for lower p-values on the part of publishers and their reviewers (by 'general' I mean not just at the 0.05 value); the second is that researchers do not randomly pick what to study - they use their expertise and existing knowledge to guide their investigations. Is that enough to tip the curve the other way across the range of p-values? Well, something is, and I am open to alternative suggestions. One other point: while the datum immediately below 0.05 would normally be considered an outlier, the fact that it is next to a discontinuity (actual or perceived) renders that call less clear. Personally, I suspect it is not an accidental outlier, but given that it does not produce much distortion in the overall trend, I am less inclined to see the 0.05 threshold (actual or perceived) as a problem than I did before I saw this chart.

setgree 5y ago

> The graph shows that those low p-values are more likely to be in papers This is an important distinction, in my experience [0]. Many papers will report a p-value only if it is below a significance threshold, otherwise they will report "n.s." (no statistic) or will give a range (e.g. p > .1). This just means that in addition to pressure to shelve insignificant results, publication bias also manifests as a tendency to emphasize and carefully report significant findings, while mentioning in passing those that don't meet whatever threshold. [0] I happen to be working on a meta-analysis of psychology and public health papers at the moment. One paper that we're reviewing constructs 32 separate statistical models, reports that many of the results are not significant, and then discusses the significant results at length.

deleted 5y ago
[deleted]
thaumasiotes 5y ago
> Many papers will report a p-value only if it is below a significance threshold, otherwise they will report "n.s." (no statistic) or will give a range (e.g. p > .1). But the oddity here is a pronounced trend in the reported p-values that meet the significance threshold. The behavior you mention cannot create that trend.