4 ms·
That sneering misses the mark, IMO. Other fields definitely have similar problems: Amgen found that they could only replicate about half of the cancer biology
by mattkrause 4y ago
That sneering misses the mark, IMO.
Other fields definitely have similar problems: Amgen found that they could only replicate about half of the cancer biology papers they tested.
The "replication crisis" showed up in psychology first for a few reasons. First, psychology probably runs the most replications of any field. The experiments themselves are easy to reproduce: all you need to redo a survey project is a Qualtrics account—--or a photocopier. They can often be done fairly quickly, making replications a good "warm-up" project for a new grad student. The field also seems to value replications, at least a little.
Second, psychology is actually quite hard! Some failed replications surely reflect sloppy or sketchy research practices, but many of them are probably due to reflect uncontrolled variables/variance. A physicist can produce an endless stream of photons, all of which are exact copies and totally independent of each other. Psychologists have nothing like that. People's reactions are shaped by their own idiosyncratic biases, their past history, and even what they think the experimenter wants to see. It's often not clear what variables actually matter, and in that sense, "failed" replications are sometimes more interesting than successes.
- 3pt14159 4y agoNot going to comment on psychology directly, because I don't know enough about it, but there are some fields where there really is no replication crisis. In structural engineering we literally tore metals apart to find their physical properties. Concrete was mixed. Giant crushing machines were used. Etc. Sometimes I think that half the reason why some fields don't have a replication crisis is because literal lives are on the line if you get it wrong, so people actually do the work to make the testing gear and align the incentives that need to be aligned.
- MichaelZuo 4y agoThere's definitely no replication crisis in jet turbine design research also.
- mattkrause 4y agoahem https://retractionwatch.com/?s=turbine https://retractionwatch.com/?s=turbine A lot of these appear to be cases of flat-out fraud or dubious research practices. These all get muddled together under the heading of "replication crisis" but I do think they're a bit of a different beast.
- MichaelZuo 4y agoYou linked to something about wind turbines. Did you link to the wrong article?
- thelamest 4y agoYeah, yet this bonduary traces a fuzzy and surprising route. Lives are on the line in microbiology, yet Bik uncovered thousands of suspect image errors there (https://en.wikipedia.org/wiki/Elisabeth_Bik https://en.wikipedia.org/wiki/Elisabeth_Bik). Semiconductor physics is relatively tangible and close to production, yet Hsu&Loo and others found that Schön faked his award and attention grabbing recipes (https://en.wikipedia.org/wiki/Sch%C3%B6n_scandal https://en.wikipedia.org/wiki/Sch%C3%B6n_scandal).
- mattkrause 4y agoRight--but that's half the problem! You can't tear people apart to figure out why they're depressed. Not only will no one volunteer to be torn apart, but it won't tell you much anyway: depression (probably) isn' a property of a single protein or cell. In a sense, this is actually my day job: I work with animals where we can dig down to the level of a single neuron or sometimes even a single molecule....but it's still different enough that you still need to go back to intact, living people to see if your proposed treatment works.
- mhuffman 4y ago>Sometimes I think that half the reason why some fields don't have a replication crisis is because literal lives are on the line if you get it wrong, so people actually do the work to make the testing gear and align the incentives that need to be aligned. You don't think lives are on the line with psychology? The replication crisis there should be terrifying but because of the extra degree of separation between theory and result, it is, for some reason, just hand-waved over. You are right that if people died because of a structural engineering mistake due to incorrect information people would be sued and possibly sent to prison. In mental health, however, people can die in droves and nothing! Or public policy that destroys the lives of millions can be made on bad research and just shrugged off later.
- 3pt14159 4y agoYes, I think your way of phrasing it is more correct. Lives are on the line with psychology, but the feedback loop is usually broken.