5 ms·

There are two legitimate ways to get lower p-values: You can have a larger effect size, or you can have a larger number of samples[0]. Of course, you can't chan

There are two legitimate ways to get lower p-values: You can have a larger effect size, or you can have a larger number of samples[0]. Of course, you can't change the effect size, so this would lead to larger necessary samples sizes to study smaller effects. I think in general though, at least in biology, people are waking up to the fact that p-values aren't magical, and that having a really small p-value isn't a goal in and of itself. It is, however, necessary to do some statistics on your data to get it published, but the p-value is just checking a box more than being used as a tool for discovery. Yesterday a cool dataset was released from Jeff Leek which has over 3.6 million p-values from scientific literature[1]. The distribution is fun to look at by discipline[2]. [0] http://rpsychologist.com/d3/NHST/ http://rpsychologist.com/d3/NHST/ [1] https://github.com/jtleek/tidypvals https://github.com/jtleek/tidypvals [2] https://twitter.com/drob/status/890260541876338690 https://twitter.com/drob/status/890260541876338690

nonbel 9y ago
Thanks, direct link to the pic: https://pbs.twimg.com/media/DFqN_vqXkAAMGB_.jpg https://pbs.twimg.com/media/DFqN_vqXkAAMGB_.jpg And here it is zoomed in: https://pbs.twimg.com/media/DFrXCYTW0AEONA9.jpg https://pbs.twimg.com/media/DFrXCYTW0AEONA9.jpg These figures look very damning to p-values. This basically looks like they are made up anyway. I haven't checked the methods though so who knows. Edit: Well there are 33 "pvalues" > 1 and 20820 that equal zero, so this dataset probably could use more cleaning.
- Mathnerd314 9y ago
  Most of them are of the form "p > 0.05" or p < 0.01", hence the peaks. You really need to filter down to the red "equals" samples, but apparently they're pretty uncommon in papers to begin with. The only real info from that graph seems to be that geography/business/economics/psychology/sociology allow "p > 0.10" relatively more often, and that there's no data for the economics category alone (from Twitter, apparently the data source is not as good). This is a better pic: https://pbs.twimg.com/media/DFqssedWsAA3h4Q.jpg:large https://pbs.twimg.com/media/DFqssedWsAA3h4Q.jpg:large Can't really see any difference in fields, besides some artifact in plant biology. Which makes sense since it's just a statistical test.
  - nonbel 9y ago
    Looking just at the "equals" data still looks weird (although in a different way): require(tidypvals) allp = allp[!allp$pvalue > 1,] allp_eq = allp[allp$operator == "equals", ] hist(allp_eq$pvalue, xlim = c(0, .15), breaks = 1000) https://image.ibb.co/i2aBqQ/pvalue_equals.png https://image.ibb.co/i2aBqQ/pvalue_equals.png
    - Mathnerd314 9y ago
      Those look like rounding artifacts; the pic I posted just used some Gaussian smoothing, but there are (way more complex) methods for handling them correctly: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2847785/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2847785/ The overall function is a p-curve, here's a paper on it: http://journals.sagepub.com/doi/pdf/10.1177/1745691614553988 http://journals.sagepub.com/doi/pdf/10.1177/1745691614553988
      - nonbel 9y ago
        Yep, rounding would make sense here. I have to agree that this data is showing much less than at first appears.
  - wodenokoto 9y ago
    I haven't looked at the raw data, but the graph makes it looks like there are p-values less than zero. Would you happen to know why?
    - nonbel 9y ago
      That is just whatever smoothing ggplot is doing not getting truncated. There were p-values greater than 1 in that dataset though, so it does need cleaning.
JumpCrisscross 9y ago
What's going on with economics?
- LoSboccacc 9y ago
  Probably researcher don't publish unless their finding hit that magic p-number, it can be self censorship more than active standardization
- elsjaako 9y ago
  They go into this in the twitter discussion: It's worth noting the datasets (except Brodeur for economics) include "less/greater than", e.g. p < .05 This is the reason for the bimodality, and also for economics missing the second peak due to some difference in how economics does statistics, they don't use that specific statement, so it doesn't show up as a peak. https://twitter.com/drob/status/890260541876338690 https://twitter.com/drob/status/890260541876338690
revelation 9y ago
Meta studies such as this seem to be one of the biggest casualties of the ongoing Elsevier/Springer/.. publisher scam. I can't even begin to imagine how one can collect that number of papers in a machine-readable format without ending up like Aaron Swartz.
BeetleB 9y ago
>There are two legitimate ways to get lower p-values: You can have a larger effect size, or you can have a larger number of samples[0]. Of course, you can't change the effect size, so this would lead to larger necessary samples sizes to study smaller effects. Be careful with this. Larger sample sizes are more likely to give you a significant result even if one doesn't exist. Say my null hypothesis is that X=100. The alternative hypothesis is X>100. What if in reality X is really 100.5? Depending on the problem domain, this may well be the same as the null hypothesis. But a larger sample size is much more likely to give a significant result. There are ways to fix this, but one should just be aware, though.
- dragonwriter 9y ago
  > Say my null hypothesis is that X=100. The alternative hypothesis is X>100. > What if in reality X is really 100.5? Then the alternative hypothesis is true and the null is false. (Though normally if the tested hypothesis was X > 100, the null would be X ≤ 100.) > Depending on the problem domain, this may well be the same as the null hypothesis. No, X = 100.5 is not the same as X = 100, irrespective of problem domain, so long as the mathematical symbols have their usual definitions. > But a larger sample size is much more likely to give a significant result. Yes, if something is only just barely true, it is more likely to take a large sample size to distinguish it from the case where it is false. But since in this case, the alternative hypothesis actually is true, it's not a problem that a bigger sample is more likely to reject the null hypothesis.
  - BeetleB 9y ago
    >Then the alternative hypothesis is true and the null is false. (Though normally if the tested hypothesis was X > 100, the null would be X ≤ 100.) Mathematically, yes. Practically, not always. In many real world effects, we're not 100% accurate on what the null hypothesis is. We may think the existing process suggests a the temperature increase of 1C, and so we set that as the null hypothesis. But in that problem domain, we may have difficulty being more precise. Suppose that 1C in the existing process came from the mean of measuring a sample of 100, so the current belief is that the expected value of the increase should be 1C. Suppose in reality it is 1.02C. Now we're proposing a new process and claim it is better than the existing one. This time we take a much larger sample, and see the number hovering at 1.02C - we can then claim the new process is better than the existing process and have a p value to demonstrate it. The math will be correct, and the conclusion will be wrong.
wodenokoto 9y ago
About [2], I couldn't really tell the color of the graph and was confused about its story. But if you scroll down, it is revealed that there is no operator available for the data on the field of economics, because it hasn't been collected, and that is why there is no bimodal for economics. The quote is: "It's confounding with the meta-analysis the data comes from: Brodeur 2016 is the only one that did econ; didn't collect <= or >= from papers"
Amygaz 9y ago
I just want to buffer a little bit what you said about "the p-value is just checking a box more than being used as a tool for discovery.", but we are likely on the same page. Sometimes, it is necessary to use a significance test in order to convince the reader that your conclusion based on your interpretation of your observations is sensical, which would be "checking a box". Other times, and as a frequent reviewer I do see that, infrequently, but still, where the authors see something where the only thing to see seems completely anecdotal (small N, large or fake error bars, no significance or ranking test), or to support their innacurate interpretation of a phenomenon (I see that with young and inexperienced graduate student in a young and inexperienced PI's lab). In these cases, having a proper experimental setup, sample size, and analysis, is part of the discovery. At the end of the day, it is important to make sure that science stays evidence-based and statistic is one of the tools that can assist, but only assist, in doing that.

nonbel 9y ago

Thanks, direct link to the pic: https://pbs.twimg.com/media/DFqN_vqXkAAMGB_.jpg https://pbs.twimg.com/media/DFqN_vqXkAAMGB_.jpg And here it is zoomed in: https://pbs.twimg.com/media/DFrXCYTW0AEONA9.jpg https://pbs.twimg.com/media/DFrXCYTW0AEONA9.jpg These figures look very damning to p-values. This basically looks like they are made up anyway. I haven't checked the methods though so who knows. Edit: Well there are 33 "pvalues" > 1 and 20820 that equal zero, so this dataset probably could use more cleaning.

Mathnerd314 9y ago
Most of them are of the form "p > 0.05" or p < 0.01", hence the peaks. You really need to filter down to the red "equals" samples, but apparently they're pretty uncommon in papers to begin with. The only real info from that graph seems to be that geography/business/economics/psychology/sociology allow "p > 0.10" relatively more often, and that there's no data for the economics category alone (from Twitter, apparently the data source is not as good). This is a better pic: https://pbs.twimg.com/media/DFqssedWsAA3h4Q.jpg:large https://pbs.twimg.com/media/DFqssedWsAA3h4Q.jpg:large Can't really see any difference in fields, besides some artifact in plant biology. Which makes sense since it's just a statistical test.
- nonbel 9y ago
  Looking just at the "equals" data still looks weird (although in a different way): require(tidypvals) allp = allp[!allp$pvalue > 1,] allp_eq = allp[allp$operator == "equals", ] hist(allp_eq$pvalue, xlim = c(0, .15), breaks = 1000) https://image.ibb.co/i2aBqQ/pvalue_equals.png https://image.ibb.co/i2aBqQ/pvalue_equals.png
  - Mathnerd314 9y ago
    Those look like rounding artifacts; the pic I posted just used some Gaussian smoothing, but there are (way more complex) methods for handling them correctly: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2847785/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2847785/ The overall function is a p-curve, here's a paper on it: http://journals.sagepub.com/doi/pdf/10.1177/1745691614553988 http://journals.sagepub.com/doi/pdf/10.1177/1745691614553988
    - nonbel 9y ago
      Yep, rounding would make sense here. I have to agree that this data is showing much less than at first appears.
- wodenokoto 9y ago
  I haven't looked at the raw data, but the graph makes it looks like there are p-values less than zero. Would you happen to know why?
  - nonbel 9y ago
    That is just whatever smoothing ggplot is doing not getting truncated. There were p-values greater than 1 in that dataset though, so it does need cleaning.

JumpCrisscross 9y ago

What's going on with economics?

LoSboccacc 9y ago
Probably researcher don't publish unless their finding hit that magic p-number, it can be self censorship more than active standardization
elsjaako 9y ago
They go into this in the twitter discussion: It's worth noting the datasets (except Brodeur for economics) include "less/greater than", e.g. p < .05 This is the reason for the bimodality, and also for economics missing the second peak due to some difference in how economics does statistics, they don't use that specific statement, so it doesn't show up as a peak. https://twitter.com/drob/status/890260541876338690 https://twitter.com/drob/status/890260541876338690

revelation 9y ago

Meta studies such as this seem to be one of the biggest casualties of the ongoing Elsevier/Springer/.. publisher scam. I can't even begin to imagine how one can collect that number of papers in a machine-readable format without ending up like Aaron Swartz.

BeetleB 9y ago

>There are two legitimate ways to get lower p-values: You can have a larger effect size, or you can have a larger number of samples[0]. Of course, you can't change the effect size, so this would lead to larger necessary samples sizes to study smaller effects. Be careful with this. Larger sample sizes are more likely to give you a significant result even if one doesn't exist. Say my null hypothesis is that X=100. The alternative hypothesis is X>100. What if in reality X is really 100.5? Depending on the problem domain, this may well be the same as the null hypothesis. But a larger sample size is much more likely to give a significant result. There are ways to fix this, but one should just be aware, though.

dragonwriter 9y ago
> Say my null hypothesis is that X=100. The alternative hypothesis is X>100. > What if in reality X is really 100.5? Then the alternative hypothesis is true and the null is false. (Though normally if the tested hypothesis was X > 100, the null would be X ≤ 100.) > Depending on the problem domain, this may well be the same as the null hypothesis. No, X = 100.5 is not the same as X = 100, irrespective of problem domain, so long as the mathematical symbols have their usual definitions. > But a larger sample size is much more likely to give a significant result. Yes, if something is only just barely true, it is more likely to take a large sample size to distinguish it from the case where it is false. But since in this case, the alternative hypothesis actually is true, it's not a problem that a bigger sample is more likely to reject the null hypothesis.
- BeetleB 9y ago
  >Then the alternative hypothesis is true and the null is false. (Though normally if the tested hypothesis was X > 100, the null would be X ≤ 100.) Mathematically, yes. Practically, not always. In many real world effects, we're not 100% accurate on what the null hypothesis is. We may think the existing process suggests a the temperature increase of 1C, and so we set that as the null hypothesis. But in that problem domain, we may have difficulty being more precise. Suppose that 1C in the existing process came from the mean of measuring a sample of 100, so the current belief is that the expected value of the increase should be 1C. Suppose in reality it is 1.02C. Now we're proposing a new process and claim it is better than the existing one. This time we take a much larger sample, and see the number hovering at 1.02C - we can then claim the new process is better than the existing process and have a p value to demonstrate it. The math will be correct, and the conclusion will be wrong.

wodenokoto 9y ago

About [2], I couldn't really tell the color of the graph and was confused about its story. But if you scroll down, it is revealed that there is no operator available for the data on the field of economics, because it hasn't been collected, and that is why there is no bimodal for economics. The quote is: "It's confounding with the meta-analysis the data comes from: Brodeur 2016 is the only one that did econ; didn't collect <= or >= from papers"

Amygaz 9y ago

I just want to buffer a little bit what you said about "the p-value is just checking a box more than being used as a tool for discovery.", but we are likely on the same page. Sometimes, it is necessary to use a significance test in order to convince the reader that your conclusion based on your interpretation of your observations is sensical, which would be "checking a box". Other times, and as a frequent reviewer I do see that, infrequently, but still, where the authors see something where the only thing to see seems completely anecdotal (small N, large or fake error bars, no significance or ranking test), or to support their innacurate interpretation of a phenomenon (I see that with young and inexperienced graduate student in a young and inexperienced PI's lab). In these cases, having a proper experimental setup, sample size, and analysis, is part of the discovery. At the end of the day, it is important to make sure that science stays evidence-based and statistic is one of the tools that can assist, but only assist, in doing that.