3 ms·
Outside of fields with direct penalties for invalid science (think aviation, nuclear reactors, sinking boats, collapsing bridges), we're just relying on the int
by hedgew 10y ago
Outside of fields with direct penalties for invalid science (think aviation, nuclear reactors, sinking boats, collapsing bridges), we're just relying on the integrity and honor of researchers. On an individual level it's much more effective to cut corners and publish trendy research than it is to make valid science.
The situation is only made worse by the fact that universities reward this; because they too are rewarded more for high-profile research than they are for scientific validity.
My guess is that most studies in softer sciences are simply erroneous. Many social sciences have already been shown to be unreliable by the replication crisis.
My personal experience comes from reviewing hundreds of published studies that evaluated the applied effectiveness of machine learning models. Half had significant statistical errors.
- chestervonwinch 10y ago> My personal experience comes from reviewing hundreds of published studies that evaluated the applied effectiveness of machine learning models. In the social sciences? Just curious. > Half had significant statistical errors. Such as? Have you enumerated the typical errors somewhere, so that other can learn to avoid these common pitfalls?
- Balgair 10y agoThe stats errors are so common and easy to correct these days that there is a statscheck (ala spellcheck) program (can't find the link, sorry) out of the Belgium that trawls through journals and checks the stats. Mind you, these are peer reviewed and published articles. Still, about half have erroneous stats, sometimes ones that invalidate the paper. Yeah, we all screw up, but it seems like a lot of the screw ups happen much too often. As for a place that has common stats errors and how to fix them? Well, just about every single intro to stats book, the entire R language, just about any library for just about any language, google, etc. The issue is not that the folks are making honest mistakes, that happens, it's that the system is perverse and incentivizes them.
- deleted 10y ago[deleted]
- maverick_iceman 10y agoAndrew Gelman's blog[1] is a great resource for identifying erroneous studies in social science. [1] http://andrewgelman.com/ http://andrewgelman.com/