3 ms·

When I was working in research, we had a journal group which each week critiqued the statistics done in scientific journals - most often contemporary articles f

When I was working in research, we had a journal group which each week critiqued the statistics done in scientific journals - most often contemporary articles from Science or Nature. As I recall, every single one of the articles we looked at would have failed our "Red Team" standards. I don't (only) mean that they failed by making trivial errors of statistics - I mean a small team of researchers could find fundamental errors that potentially or likely undermined or contradicted the whole article. It's not surprising that something like 50% of the conclusions are not reproducible.

anitil 6y ago
I can't remember where I read the quote (maybe Ben Goldacre?), but it was along the lines of "If it's in Nature, it's probably false". I suppose because the likelihood of publication in Nature goes up if it's a surprising result.
phobosanomaly 6y ago
We're sure not great at teaching biomedical people statistics. It's not a rigorous study or anything, but this about sums it up: "Most respondents did not have any formal training in data literacy. Respondents considered most tasks highly relevant to their work but rated their expertise in tasks lower." Federer LM, Lu YL, Joubert DJ. Data literacy training needs of biomedical researchers. J Med Libr Assoc. 2016;104(1):52-57. doi:10.3163/1536-5050.104.1.008 I work in the field, and am often embarrassed by how rudimentary my stats skills are. I'm not an outlier.
abdullahkhalids 6y ago
I would say that among all math-oriented courses, statistics courses are the worst-taught. Part of the reason is that the field has never been able to agree upon a single philosophical underpinning, and so modern courses are often a hodge podge of techniques without enough discussion of basic conceptual issues.
- learnstats2 6y ago
  I partially agree, but I don't think this is a problem of statistics. If you were following statistics thoroughly, you would ideally set out your justified hypothesis before starting any research, and conclude whether your hypothesis was true or false. The problem is that researchers generally refuse to do this - if they know to do this - because the hope is to be rewarded for finding some novel and unanticipated result that changes the field. This leads to p-hacking and other techniques which immediately invalidate any statistics that are used, anyway.

anitil 6y ago

I can't remember where I read the quote (maybe Ben Goldacre?), but it was along the lines of "If it's in Nature, it's probably false". I suppose because the likelihood of publication in Nature goes up if it's a surprising result.

phobosanomaly 6y ago

We're sure not great at teaching biomedical people statistics. It's not a rigorous study or anything, but this about sums it up: "Most respondents did not have any formal training in data literacy. Respondents considered most tasks highly relevant to their work but rated their expertise in tasks lower." Federer LM, Lu YL, Joubert DJ. Data literacy training needs of biomedical researchers. J Med Libr Assoc. 2016;104(1):52-57. doi:10.3163/1536-5050.104.1.008 I work in the field, and am often embarrassed by how rudimentary my stats skills are. I'm not an outlier.

abdullahkhalids 6y ago

I would say that among all math-oriented courses, statistics courses are the worst-taught. Part of the reason is that the field has never been able to agree upon a single philosophical underpinning, and so modern courses are often a hodge podge of techniques without enough discussion of basic conceptual issues.

learnstats2 6y ago
I partially agree, but I don't think this is a problem of statistics. If you were following statistics thoroughly, you would ideally set out your justified hypothesis before starting any research, and conclude whether your hypothesis was true or false. The problem is that researchers generally refuse to do this - if they know to do this - because the hope is to be rewarded for finding some novel and unanticipated result that changes the field. This leads to p-hacking and other techniques which immediately invalidate any statistics that are used, anyway.