3 ms·
Be aware that despite how much focus replicability gets, it's only one of many things that goes wrong with research papers. Even if you somehow waved a magic wa
by mike_hearn 5y ago
Be aware that despite how much focus replicability gets, it's only one of many things that goes wrong with research papers. Even if you somehow waved a magic wand and fixed replicability perfectly tomorrow, entire academic fields would still be worthless and misleading.
How can replicable research go wrong? Here's just a fraction of the things I've seen reading papers:
1. Logic errors. So many logic errors. Replicating something that doesn't make sense leaves you with two things that don't make sense: a waste of time and money.
2. Tiny effect sizes. Often an effect will "replicate" but with a smaller effect than the one claimed; is this a successful replication or not?
3. Intellectual fraud. Often this works by taking a normal English term and then at the start of your paper giving it an incorrect definition. Again this will replicate just fine but the result is still misinformation.
4. Incoherent concepts. What exactly does R0 mean in epidemiology and precisely how is it determined? You can replicate the calculations that are used but you won't be calculating what you think you are.
5. A lot of research isn't experimental, it's purely observational. You can't go back and re-observe the things being studied, only re-analyze the data they originally collected. Does this count?
6. Incredibly obvious findings. Wealthy parents have more successful children, etc. It'll replicate all right but so what? Why are taxpayers being made to fund this stuff?
7. Fraudulent practices that are nonetheless normalized within a field. The article complains about scientists Photoshopping western blots (a type of artifact produced in biology experiments). That's because editing your data in ways that make it fit your theory is universally understood to be fraud ... except in climatology, where scientists have developed a habit of constantly rewriting the databases that contain historical temperature records. And by "historical" we mean "last year" here, not 1000 years ago. These edits always make global warming more pronounced, and sometimes actually create warming trends where previously there were none (e.g. [1]). Needless to say climatologists don't consider this fraud. It means if you're trying to replicate a claim from climatology, even an apparently factual claim about a certain fixed year, you may run into the problem that it was "true" at the time it was made and may even have been replicated, but is now "false" because the data has been edited since.
Epidemiology has a somewhat similar problem - they don't consider deterministic models to be important, i.e. it may be impossible to get the same numbers out of a model as a paper presents, even if you give it identical inputs, due to race conditions/memory corruption bugs in the code. They do not consider this a problem and will claim it doesn't matter because the model uses a PRNG somewhere, or that they "replicated" the model outputs because they got numbers only 25% different.
What does it even mean to say a claim does or does not replicate, in fields like these?
All this takes place in an environment of near total institutional indifference. Paper replicates? Great. Nobody cares, because they all assumed it would. Paper doesn't replicate, or has methodological errors making replication pointless? Nobody cares about that either.
Your proposal suggests blocking publication until replication is done by independent labs. That won't work, because even if you found some way to actually enforce that (not all grants come from the government!), you'll just end up with lots of papers that can be replicated but are still nonsensical for other reasons.
[1] https://nature.com/articles/nature.2015.17700 https://nature.com/articles/nature.2015.17700
- qchris 5y ago> Even if you somehow waved a magic wand and fixed replicability perfectly tomorrow, entire academic fields would still be worthless and misleading. Even ignoring how, frankly, paternalistic and condescending this sentence comes off as, let's take the rest of your comment on face value. First, I can agree with you that the parent comment's idea about not allowing publication without replication is a non-starter; it would basically be impossible to implement (in the U.S., you'd immediately run into 1st Amendment issues), and on a practical level, that doesn't match with almost any realistic scenario of how research both is or could be conducted. Anyone that can get published should be welcome to do so. However, I think that the idea of funding a subset of replication studies would have great value, and your list of "issues" are largely knocking down strawmen. Let's look at this from the point of view of a funding agency, which annually pour billions of dollars into research. 1. Logic errors would be great! A replication report stating "the conclusions of this research are deeply invalid due to x,y, and z logic errors here, here, and here" would make the agency less likely to fund authors that make those errors. If the errors don't invalidate the conclusions, they're still worth noting and can be taken into account. 2. Effect sizes varying in replicated studies is still useful information. Not all "does/doesn't replicate" questions need a binary yes/no answer. 3. Similarly to #1, catching and publishing instances where authors are making invalid claims (even if not doing invalid research) seems like a good thing. "Hey, these researchers have a tendency to claim X when really they're talking about Y. What the heck?" is great information to know for a funding agency to know. 4. See #1 and #3. Generally, the authors of the replication studies are also going to be intelligent; if they're being paid to check for logical fallacies and incoherent concepts (one can imagine a first-year required grad course that lays out these ideas and case studies of how to catch them). 5. I've been thinking about a situations where this still wouldn't produce useful meta information, and I'm having a hard time coming up with them. Imagine a fish survey, where divers are doing fish counts along transects. You can't go re-observe those fish, but a) research groups might want to change their methods to include using cameras instead of marking dive slates, or b) if you are able to re-run their transects due to good record-keeping (yay, replication!) but don't see the same fish, the funding agency now knows that a) the original research group has solid experimental methods and b) maybe they should keep funding groups to run that transect because of the variation, or not fund groups using a single transect data as the basis for forming conclusions. Happy to consider non-outlier (i.e. building a second LHC) examples where this breaks down. 6. Yes, maybe don't fund groups to try replicating these findings. Or maybe do occasionally anyway--if the research groups are doing good work, it should be cheaper to conduct replication-based research than doing a novel one anyway. 7. This is a good argument for replicable research. A huge number of research proposals go unfunded, which is to say that many researchers are somewhat competing for funding. In this paradigm, saying "hey, my group found and thoroughly documented a huge, glaring issue in the methods used in these studies due to a bad-faith corruption of historical data," in a situation where agencies would be looking to pay groups for valid claims. Increasing the number of replication studies adds a greater adversarial capacity to a research system which helps to catch/prevent this kind of fraud in the first place. This is already a long comment, but I can make similar points about almost all of your claims. In summary: you seem to be making the assumption that replication studies are basically done by robots blindly following an instruction manual of someone else's research. Especially given the funding incentives mentioned above, I don't think that would be true; rather, many errors would be caught (or prevented ahead of time) and lead to better, more solid research being done in the first place.