4 ms·

Thanks, direct link to the pic: https://pbs.twimg.com/media/DFqN_vqXkAAMGB_.jpg https://pbs.twimg.com/media/DFqN_vqXkAAMGB_.jpg And here it is zoomed in: http

by nonbel 9y ago

Thanks, direct link to the pic: https://pbs.twimg.com/media/DFqN_vqXkAAMGB_.jpg https://pbs.twimg.com/media/DFqN_vqXkAAMGB_.jpg And here it is zoomed in: https://pbs.twimg.com/media/DFrXCYTW0AEONA9.jpg https://pbs.twimg.com/media/DFrXCYTW0AEONA9.jpg These figures look very damning to p-values. This basically looks like they are made up anyway. I haven't checked the methods though so who knows. Edit: Well there are 33 "pvalues" > 1 and 20820 that equal zero, so this dataset probably could use more cleaning.

Mathnerd314 9y ago
Most of them are of the form "p > 0.05" or p < 0.01", hence the peaks. You really need to filter down to the red "equals" samples, but apparently they're pretty uncommon in papers to begin with. The only real info from that graph seems to be that geography/business/economics/psychology/sociology allow "p > 0.10" relatively more often, and that there's no data for the economics category alone (from Twitter, apparently the data source is not as good). This is a better pic: https://pbs.twimg.com/media/DFqssedWsAA3h4Q.jpg:large https://pbs.twimg.com/media/DFqssedWsAA3h4Q.jpg:large Can't really see any difference in fields, besides some artifact in plant biology. Which makes sense since it's just a statistical test.
- nonbel 9y ago
  Looking just at the "equals" data still looks weird (although in a different way): require(tidypvals) allp = allp[!allp$pvalue > 1,] allp_eq = allp[allp$operator == "equals", ] hist(allp_eq$pvalue, xlim = c(0, .15), breaks = 1000) https://image.ibb.co/i2aBqQ/pvalue_equals.png https://image.ibb.co/i2aBqQ/pvalue_equals.png
  - Mathnerd314 9y ago
    Those look like rounding artifacts; the pic I posted just used some Gaussian smoothing, but there are (way more complex) methods for handling them correctly: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2847785/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2847785/ The overall function is a p-curve, here's a paper on it: http://journals.sagepub.com/doi/pdf/10.1177/1745691614553988 http://journals.sagepub.com/doi/pdf/10.1177/1745691614553988
    - nonbel 9y ago
      Yep, rounding would make sense here. I have to agree that this data is showing much less than at first appears.
- wodenokoto 9y ago
  I haven't looked at the raw data, but the graph makes it looks like there are p-values less than zero. Would you happen to know why?
  - nonbel 9y ago
    That is just whatever smoothing ggplot is doing not getting truncated. There were p-values greater than 1 in that dataset though, so it does need cleaning.

by nonbel 9y ago

Mathnerd314 9y ago

Most of them are of the form "p > 0.05" or p < 0.01", hence the peaks. You really need to filter down to the red "equals" samples, but apparently they're pretty uncommon in papers to begin with. The only real info from that graph seems to be that geography/business/economics/psychology/sociology allow "p > 0.10" relatively more often, and that there's no data for the economics category alone (from Twitter, apparently the data source is not as good). This is a better pic: https://pbs.twimg.com/media/DFqssedWsAA3h4Q.jpg:large https://pbs.twimg.com/media/DFqssedWsAA3h4Q.jpg:large Can't really see any difference in fields, besides some artifact in plant biology. Which makes sense since it's just a statistical test.

nonbel 9y ago
Looking just at the "equals" data still looks weird (although in a different way): require(tidypvals) allp = allp[!allp$pvalue > 1,] allp_eq = allp[allp$operator == "equals", ] hist(allp_eq$pvalue, xlim = c(0, .15), breaks = 1000) https://image.ibb.co/i2aBqQ/pvalue_equals.png https://image.ibb.co/i2aBqQ/pvalue_equals.png
- Mathnerd314 9y ago
  Those look like rounding artifacts; the pic I posted just used some Gaussian smoothing, but there are (way more complex) methods for handling them correctly: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2847785/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2847785/ The overall function is a p-curve, here's a paper on it: http://journals.sagepub.com/doi/pdf/10.1177/1745691614553988 http://journals.sagepub.com/doi/pdf/10.1177/1745691614553988
  - nonbel 9y ago
    Yep, rounding would make sense here. I have to agree that this data is showing much less than at first appears.
wodenokoto 9y ago
I haven't looked at the raw data, but the graph makes it looks like there are p-values less than zero. Would you happen to know why?
- nonbel 9y ago
  That is just whatever smoothing ggplot is doing not getting truncated. There were p-values greater than 1 in that dataset though, so it does need cleaning.