3 ms·
Well, that looks at first like a pretty comprehensive list of possible problems in a scientific studies, but the correct reaction on reading it should be "wait,
by mike_hearn 6y ago
Well, that looks at first like a pretty comprehensive list of possible problems in a scientific studies, but the correct reaction on reading it should be "wait, what about the really serious problems?". I mean, (ab)using Google Surveys or experimenter demand effects is one kind of problem that indeed probably deserves a lot of nuance about criticizing it. But it's also not really where the biggest problems are or where the attention should be focused.
We're in an environment where people routinely discover flat out academic fraud, as in, scientists just made whole tables of data up out of thin air. Someone notices, informs the journal, and with 95% likelihood nothing happens. Or maybe the researcher is allowed to 'correct' their made up data. We're in an environment where you literally cannot take any number in a COVID-related paper at face value, even when there are multiple citations for it, because on checking citations routinely turn out to be fraudulent e.g. the cited paper doesn't actually contain the claimed data anywhere in it, or explicitly states the opposite of what's being claimed, or the number turns out to have been a hypothetical scenario but is being cited as a "widely believed" empirical fact, etc. We're in an environment where literature reviews argue that whilst very few public health models can ever be validated against reality, that's not a big deal and they should be used anyway.
Gwern criticises people who learn about logical fallacies and then go around over-criticising people for engaging in them. Yeah, sure, if you claim someone is taking bribes and they actually are then it's technically an ad hominem but still correct to say. Granted. But we are not suffering from an over-abundance of nitpicky fallacy-criticizers. Where are these people when you need them? COVID related research frequently contains or is completely built on circular logic! That's a pretty basic fallacy yet papers that engage in it manage to be written by teams of 20, sail through multiple peer reviews and appear in Nature or the BMJ. As far as I can tell the scientific institutions cannot reliably detect logical fallacies even when papers are dealing with things that should be entirely logical like data, maths, code, study design.
The notion that scientific criticism should be refined to focus on what really matters sounds completely reasonable in the abstract, and is probably an important discussion to have in some very specific fields (maybe genetics is one). But I'd worry that if people are arguing about p-hacking or lack of negative results, that means they're not arguing about the apparently legion researchers making "mistakes" that cannot plausibly be actual mistakes. Stuff that should be actually criminal needs to be fixed first, before worrying about mere low standards.
- exmadscientist 6y agoIn practice, there's a massive bikeshedding problem in scientific review. It is very frustrating to see proposals or preprints criticized for missing a dot in paragraph 37 when the bigger problem is that the experiment is overall worthless because it can never change a decision.... This is bad enough when it's an honest mistake, but then there are all the dishonest or plainly incompetent people to worry about. So, no, in fact I think we are suffering from an abundance of nitpickers. Science desperately needs more reviewers who can see the bigger pictures.
- mike_hearn 6y agoThat's fair enough. I was meaning nitpickers about logical fallacies specifically, not stuff like grammar or minor details of an experiment. That said, is peer review really the place to dunk on a study because the whole goal is pointless? The work is done by that point, it's too late. It's the granting bodies that should be getting peer reviewed in that regard, but one of the root causes of the malaise in research is that the granting bodies appear to be entirely blind buyers. They care about dispersing money, not what they get out of the spending. If they didn't spend the money they'd be fired, so that's understandable. The core problem IMHO is the huge level of state funding of research. The buck stops at politicians but they are in no position to evaluate the quality of academic studies.
- exmadscientist 6y agoI was actually thinking of grant applications when I wrote that. And it's not even politicians, it's grant committees too. For the usual reasons, they'd rather fund study #13 from a researcher than anything that is the slightest bit uncomfortable to them.