3 ms·
To be clear, I'm not against funding replication studies. That's a great thing that should be done. The risk is that if it were done people would think - great!
by mike_hearn 5y ago
To be clear, I'm not against funding replication studies. That's a great thing that should be done. The risk is that if it were done people would think - great! There was a problem but science is fixed now. That wouldn't be true. The clear majority of papers I've read that were bad/unusable in some way in the past 10 years wouldn't have been helped by funding replication studies.
> (1) (3) (4) you seem to be making the assumption that replication studies are basically done by robots blindly following an instruction manual of someone else's research.
I guess we're using the word replication differently. You seem to be using it to mean a general re-review and re-analysis of everything - basically a funded more aggressive peer review followed by an actual re-running of the study sometimes, if it makes sense, whereas I'm using it to just mean re-running the study exactly as originally described to check the results are the same.
I think in science the term replication normally just means re-running the study exactly as described. It doesn't mean arguing with the authors over their definitions or the logical basis of the study itself. I've seen a bunch of cases where a replication fails and the original study team basically rejects the whole exercise by saying "They didn't follow our instructions so of course they got different results". And I mean, that's kind of fair, right? If you say "I did X and saw Y", and then someone else says it's not true but they didn't actually do X, surely the scientists have every right to be annoyed and reject the exercise? So good replications are exactly what they sound like - a replication of the original process. They aren't generalized adversarial funded peer reviews.
And that's why it's important to highlight that replication is only one aspect. If Congress or whomever goes and earmarks money for replication, then someone with money will eventually ask you to replicate a study with a flawed design. What do you do? You could:
a. Say no: the study is flawed. Replication would be useless because you'd just be repeating a nonsensical procedure. You get no money.
b. Take the money and re-run it. The study conclusions are still wrong but now you got a paper published, and the original is successfully replicated so journalists/professors will use that as a stamp of approval. You may even want the study to replicate because it's professionally or ideologically useful.
c. Try to 'fix' the original design and do an improved study. This just starts the process over from scratch, it doesn't yield evidence.
It'll be (b) or (c) of course, every time. That's just how the incentives are set up and besides, doing another study can be done without involving the original authors. The moment you go down route (a) you're going to get pushback. They won't agree their definitions are illogical. So, it'll devolve into an exchange of letters that nobody ever sees or cares about, and granting agencies won't know how to value it.
> 2. Effect sizes varying in replicated studies is still useful information. Not all "does/doesn't replicate" questions need a binary yes/no answer.
But the bureaucracy needs actionable outcomes, otherwise there's no point. If someone can point at a replication and say "ah yes, we claimed our educational intervention would boost grades by 2x in 15 year olds and governments spent money on that basis, but a replicated 0.1% improvement nonetheless proves us right" then nobody outside the system will consider this a valid replication. There must be actual outcomes from a failure to replicate meaning you'd have to draw the line at how much delta is allowable from the original numbers, but nobody is even having that discussion let alone doing anything about it.
For (5), it was more of a question than anything else. Replication using originally collected data might be useful sometimes but the question is whether the general public would consider this to be a genuine cross-check. My guess is no. Science has no formal mechanisms to detect made up or fraudulent data sets, which is a real problem, so if you allow re-analysis of originally collected data you'll get a steady stream of situations where a study is announced, some skeptics say "uhhh that sounds wrong", the media/academic institutions beat up on them claiming it's peer reviewed and replicated which is gold standard so you've got to believe, and then it turns out the whole original study + replications were all based on fraud. What could be more damaging than that? A replication is a type of audit. These have to take into account the possibility of people playing games for profit.
- qchris 5y ago> I guess we're using the word replication differently. You seem to be using it to mean a general re-review and re-analysis of everything - basically a funded more aggressive peer review followed by an actual re-running of the study sometimes, if it makes sense, whereas I'm using it to just mean re-running the study exactly as originally described to check the results are the same. Yes, I think that this is the crux of the matter here, and I agree that merely funding direct, vanilla replication is not the optimal solution here and would not solve many of the problems we're facing. I'm really talking about the idea from the thread I linked in my original comment--the idea of "red teams" for science. There's probably a number of ways to accomplish this, and direct replication could play a role, but I imagine that a funding agency tasked with funding red teams for other government-funded research would lean towards a more diverse and holistic toolkit. You'd certainly need to have some mechanisms for "watching the watchers," but again, very few researchers I've met are stupid, many of them actually care pretty deeply about ethics and scientific integrity, and would probably jump at the chance for a steady paycheck while helping to advance that. > Science has no formal mechanisms to detect made up or fraudulent data sets, which is a real problem [...] This isn't strictly true (I'd like encourage people not to use "Science" as a proper noun; I think it's actually a bit unhelpful). I think a common approach to performing research using the scientific method often (almost always?) involves using observation across multiple instances, then using statistics to draw a generalized conclusions based on the group of observations. And statistics does have formal methods for identifying potentially fraudulent datasets, some of which a Dr. Hilgard does a great job of explaining in an actual case of research fraud he found in this article[1]. [1] https://crystalprisonzone.blogspot.com/2021/01/i-tried-to-report-scientific-misconduct.html https://crystalprisonzone.blogspot.com/2021/01/i-tried-to-re...
- mike_hearn 5y agoAh yes, Hilgard's article is great. I link to an article that links to it elsewhere in this thread. SPRITE and such are very clever but such ammo is very limited. As Hilgard himself points out, the first thing institutions do when faced with a claim of fraudulent data is immediately tell the accused everything. Then there's either no investigation at all or a useless one. All this approach does is teach fraudsters how to avoid detection, and let's face it, the moment there are any consequences, fraudulent scientists will just start running SPRITE and GRIM on their own papers before publication. Fighting fraud requires massive consequences for fraudsters, otherwise if they fail they'll just keep trying again until they learn to evade detection. Full blown red teams would be great, but seem very hard to set up. I've actually talked to British MPs about this and made some concrete proposals along these lines, but there are several blockers: 1. General apathy / lack of political will. Some of the more savvy MPs know their scientific advisors weren't reliable in COVID and understand the underlying issues, but the vast majority do not and would rather not think about it. 2. The moment you say let's set up a red team, you face the question of how to find high integrity well trained people that have exactly the same idea of what good science is as you do. Particularly problematic: do you have field-specific red teams? If so what stops Ferguson-style problems where the entire field agrees that it's unreasonable to even expect mathematical models to be replicable? You really need outsiders with the highest standards, who won't accept justifications like "We rewrote our dataset with a model because that's just how we do things in our field and who are you to argue with us?". I think there's a meme about how physicists are always telling other fields they're doing it wrong. Maybe red teams need to be made of physicists :) 3. What exactly is Science, proper noun intended? To red team you need to have a very clear definition of what the scientific method actually is. Who should come up with this? I ended up suggesting a committee of MPs should do it even though that sounds wrong, because ultimately they're the ones authorizing the flow of money, and they're the ultimate outsiders. The moment you start picking insiders it's going to turn into a huge fight (who are these physics nerds to tell us we can't do experiments with 30 undergrads? etc)