3 ms·
Ah yes, Hilgard's article is great. I link to an article that links to it elsewhere in this thread. SPRITE and such are very clever but such ammo is very limite
by mike_hearn 5y ago
Ah yes, Hilgard's article is great. I link to an article that links to it elsewhere in this thread. SPRITE and such are very clever but such ammo is very limited. As Hilgard himself points out, the first thing institutions do when faced with a claim of fraudulent data is immediately tell the accused everything. Then there's either no investigation at all or a useless one. All this approach does is teach fraudsters how to avoid detection, and let's face it, the moment there are any consequences, fraudulent scientists will just start running SPRITE and GRIM on their own papers before publication. Fighting fraud requires massive consequences for fraudsters, otherwise if they fail they'll just keep trying again until they learn to evade detection.
Full blown red teams would be great, but seem very hard to set up. I've actually talked to British MPs about this and made some concrete proposals along these lines, but there are several blockers:
1. General apathy / lack of political will. Some of the more savvy MPs know their scientific advisors weren't reliable in COVID and understand the underlying issues, but the vast majority do not and would rather not think about it.
2. The moment you say let's set up a red team, you face the question of how to find high integrity well trained people that have exactly the same idea of what good science is as you do. Particularly problematic: do you have field-specific red teams? If so what stops Ferguson-style problems where the entire field agrees that it's unreasonable to even expect mathematical models to be replicable? You really need outsiders with the highest standards, who won't accept justifications like "We rewrote our dataset with a model because that's just how we do things in our field and who are you to argue with us?".
I think there's a meme about how physicists are always telling other fields they're doing it wrong. Maybe red teams need to be made of physicists :)
3. What exactly is Science, proper noun intended? To red team you need to have a very clear definition of what the scientific method actually is. Who should come up with this? I ended up suggesting a committee of MPs should do it even though that sounds wrong, because ultimately they're the ones authorizing the flow of money, and they're the ultimate outsiders. The moment you start picking insiders it's going to turn into a huge fight (who are these physics nerds to tell us we can't do experiments with 30 undergrads? etc)