4 ms·

> Even if you somehow waved a magic wand and fixed replicability perfectly tomorrow, entire academic fields would still be worthless and misleading. Even ignor

by qchris 5y ago

> Even if you somehow waved a magic wand and fixed replicability perfectly tomorrow, entire academic fields would still be worthless and misleading. Even ignoring how, frankly, paternalistic and condescending this sentence comes off as, let's take the rest of your comment on face value. First, I can agree with you that the parent comment's idea about not allowing publication without replication is a non-starter; it would basically be impossible to implement (in the U.S., you'd immediately run into 1st Amendment issues), and on a practical level, that doesn't match with almost any realistic scenario of how research both is or could be conducted. Anyone that can get published should be welcome to do so. However, I think that the idea of funding a subset of replication studies would have great value, and your list of "issues" are largely knocking down strawmen. Let's look at this from the point of view of a funding agency, which annually pour billions of dollars into research. 1. Logic errors would be great! A replication report stating "the conclusions of this research are deeply invalid due to x,y, and z logic errors here, here, and here" would make the agency less likely to fund authors that make those errors. If the errors don't invalidate the conclusions, they're still worth noting and can be taken into account. 2. Effect sizes varying in replicated studies is still useful information. Not all "does/doesn't replicate" questions need a binary yes/no answer. 3. Similarly to #1, catching and publishing instances where authors are making invalid claims (even if not doing invalid research) seems like a good thing. "Hey, these researchers have a tendency to claim X when really they're talking about Y. What the heck?" is great information to know for a funding agency to know. 4. See #1 and #3. Generally, the authors of the replication studies are also going to be intelligent; if they're being paid to check for logical fallacies and incoherent concepts (one can imagine a first-year required grad course that lays out these ideas and case studies of how to catch them). 5. I've been thinking about a situations where this still wouldn't produce useful meta information, and I'm having a hard time coming up with them. Imagine a fish survey, where divers are doing fish counts along transects. You can't go re-observe those fish, but a) research groups might want to change their methods to include using cameras instead of marking dive slates, or b) if you are able to re-run their transects due to good record-keeping (yay, replication!) but don't see the same fish, the funding agency now knows that a) the original research group has solid experimental methods and b) maybe they should keep funding groups to run that transect because of the variation, or not fund groups using a single transect data as the basis for forming conclusions. Happy to consider non-outlier (i.e. building a second LHC) examples where this breaks down. 6. Yes, maybe don't fund groups to try replicating these findings. Or maybe do occasionally anyway--if the research groups are doing good work, it should be cheaper to conduct replication-based research than doing a novel one anyway. 7. This is a good argument for replicable research. A huge number of research proposals go unfunded, which is to say that many researchers are somewhat competing for funding. In this paradigm, saying "hey, my group found and thoroughly documented a huge, glaring issue in the methods used in these studies due to a bad-faith corruption of historical data," in a situation where agencies would be looking to pay groups for valid claims. Increasing the number of replication studies adds a greater adversarial capacity to a research system which helps to catch/prevent this kind of fraud in the first place. This is already a long comment, but I can make similar points about almost all of your claims. In summary: you seem to be making the assumption that replication studies are basically done by robots blindly following an instruction manual of someone else's research. Especially given the funding incentives mentioned above, I don't think that would be true; rather, many errors would be caught (or prevented ahead of time) and lead to better, more solid research being done in the first place.

mike_hearn 4y ago
To be clear, I'm not against funding replication studies. That's a great thing that should be done. The risk is that if it were done people would think - great! There was a problem but science is fixed now. That wouldn't be true. The clear majority of papers I've read that were bad/unusable in some way in the past 10 years wouldn't have been helped by funding replication studies. > (1) (3) (4) you seem to be making the assumption that replication studies are basically done by robots blindly following an instruction manual of someone else's research. I guess we're using the word replication differently. You seem to be using it to mean a general re-review and re-analysis of everything - basically a funded more aggressive peer review followed by an actual re-running of the study sometimes, if it makes sense, whereas I'm using it to just mean re-running the study exactly as originally described to check the results are the same. I think in science the term replication normally just means re-running the study exactly as described. It doesn't mean arguing with the authors over their definitions or the logical basis of the study itself. I've seen a bunch of cases where a replication fails and the original study team basically rejects the whole exercise by saying "They didn't follow our instructions so of course they got different results". And I mean, that's kind of fair, right? If you say "I did X and saw Y", and then someone else says it's not true but they didn't actually do X, surely the scientists have every right to be annoyed and reject the exercise? So good replications are exactly what they sound like - a replication of the original process. They aren't generalized adversarial funded peer reviews. And that's why it's important to highlight that replication is only one aspect. If Congress or whomever goes and earmarks money for replication, then someone with money will eventually ask you to replicate a study with a flawed design. What do you do? You could: a. Say no: the study is flawed. Replication would be useless because you'd just be repeating a nonsensical procedure. You get no money. b. Take the money and re-run it. The study conclusions are still wrong but now you got a paper published, and the original is successfully replicated so journalists/professors will use that as a stamp of approval. You may even want the study to replicate because it's professionally or ideologically useful. c. Try to 'fix' the original design and do an improved study. This just starts the process over from scratch, it doesn't yield evidence. It'll be (b) or (c) of course, every time. That's just how the incentives are set up and besides, doing another study can be done without involving the original authors. The moment you go down route (a) you're going to get pushback. They won't agree their definitions are illogical. So, it'll devolve into an exchange of letters that nobody ever sees or cares about, and granting agencies won't know how to value it. > 2. Effect sizes varying in replicated studies is still useful information. Not all "does/doesn't replicate" questions need a binary yes/no answer. But the bureaucracy needs actionable outcomes, otherwise there's no point. If someone can point at a replication and say "ah yes, we claimed our educational intervention would boost grades by 2x in 15 year olds and governments spent money on that basis, but a replicated 0.1% improvement nonetheless proves us right" then nobody outside the system will consider this a valid replication. There must be actual outcomes from a failure to replicate meaning you'd have to draw the line at how much delta is allowable from the original numbers, but nobody is even having that discussion let alone doing anything about it. For (5), it was more of a question than anything else. Replication using originally collected data might be useful sometimes but the question is whether the general public would consider this to be a genuine cross-check. My guess is no. Science has no formal mechanisms to detect made up or fraudulent data sets, which is a real problem, so if you allow re-analysis of originally collected data you'll get a steady stream of situations where a study is announced, some skeptics say "uhhh that sounds wrong", the media/academic institutions beat up on them claiming it's peer reviewed and replicated which is gold standard so you've got to believe, and then it turns out the whole original study + replications were all based on fraud. What could be more damaging than that? A replication is a type of audit. These have to take into account the possibility of people playing games for profit.

by qchris 5y ago

mike_hearn 4y ago

To be clear, I'm not against funding replication studies. That's a great thing that should be done. The risk is that if it were done people would think - great! There was a problem but science is fixed now. That wouldn't be true. The clear majority of papers I've read that were bad/unusable in some way in the past 10 years wouldn't have been helped by funding replication studies. > (1) (3) (4) you seem to be making the assumption that replication studies are basically done by robots blindly following an instruction manual of someone else's research. I guess we're using the word replication differently. You seem to be using it to mean a general re-review and re-analysis of everything - basically a funded more aggressive peer review followed by an actual re-running of the study sometimes, if it makes sense, whereas I'm using it to just mean re-running the study exactly as originally described to check the results are the same. I think in science the term replication normally just means re-running the study exactly as described. It doesn't mean arguing with the authors over their definitions or the logical basis of the study itself. I've seen a bunch of cases where a replication fails and the original study team basically rejects the whole exercise by saying "They didn't follow our instructions so of course they got different results". And I mean, that's kind of fair, right? If you say "I did X and saw Y", and then someone else says it's not true but they didn't actually do X, surely the scientists have every right to be annoyed and reject the exercise? So good replications are exactly what they sound like - a replication of the original process. They aren't generalized adversarial funded peer reviews. And that's why it's important to highlight that replication is only one aspect. If Congress or whomever goes and earmarks money for replication, then someone with money will eventually ask you to replicate a study with a flawed design. What do you do? You could: a. Say no: the study is flawed. Replication would be useless because you'd just be repeating a nonsensical procedure. You get no money. b. Take the money and re-run it. The study conclusions are still wrong but now you got a paper published, and the original is successfully replicated so journalists/professors will use that as a stamp of approval. You may even want the study to replicate because it's professionally or ideologically useful. c. Try to 'fix' the original design and do an improved study. This just starts the process over from scratch, it doesn't yield evidence. It'll be (b) or (c) of course, every time. That's just how the incentives are set up and besides, doing another study can be done without involving the original authors. The moment you go down route (a) you're going to get pushback. They won't agree their definitions are illogical. So, it'll devolve into an exchange of letters that nobody ever sees or cares about, and granting agencies won't know how to value it. > 2. Effect sizes varying in replicated studies is still useful information. Not all "does/doesn't replicate" questions need a binary yes/no answer. But the bureaucracy needs actionable outcomes, otherwise there's no point. If someone can point at a replication and say "ah yes, we claimed our educational intervention would boost grades by 2x in 15 year olds and governments spent money on that basis, but a replicated 0.1% improvement nonetheless proves us right" then nobody outside the system will consider this a valid replication. There must be actual outcomes from a failure to replicate meaning you'd have to draw the line at how much delta is allowable from the original numbers, but nobody is even having that discussion let alone doing anything about it. For (5), it was more of a question than anything else. Replication using originally collected data might be useful sometimes but the question is whether the general public would consider this to be a genuine cross-check. My guess is no. Science has no formal mechanisms to detect made up or fraudulent data sets, which is a real problem, so if you allow re-analysis of originally collected data you'll get a steady stream of situations where a study is announced, some skeptics say "uhhh that sounds wrong", the media/academic institutions beat up on them claiming it's peer reviewed and replicated which is gold standard so you've got to believe, and then it turns out the whole original study + replications were all based on fraud. What could be more damaging than that? A replication is a type of audit. These have to take into account the possibility of people playing games for profit.