6 ms·

Pre-registration is becoming more common but unfortunately, it isn't enough. For it to work there's an implied second stage: journals and other funding gatekeep

Pre-registration is becoming more common but unfortunately, it isn't enough. For it to work there's an implied second stage: journals and other funding gatekeepers have to commit to blocking research if it doesn't actually implement the pre-registered plan. This does not seem to be happening reliably, or at least it's relatively easy to find counter-examples. There's also a lot of scope for drama and problems at that second stage, because there can be a lot of wiggle room in whether what was done meets or doesn't meet the pre-registered plan. It might sound easy to follow a pre-written plan but for instance if what you're doing turns out to need adjustment due to discovering something you hadn't anticipated, then technically you might violate your pre-registration. In turn that can yield perverse incentives to not fix things. Having been through a few rounds of "here's a new procedure that will yield better science, oh no it isn't really working" I'm not really enthused anymore by this kind of tactical fix. From a software dev perspective it's sort of like constantly hot-patching a system in prod to try and cover up bugs when the underlying problem is actually bad architecture that creates too many bugs in the first place. Getting overly fixated on P-hacking or non-replicability distracts from the deeper underlying problem of a mis-specified utility function. Example: replication was meant to be the big hammer but most studies still aren't being replicated and when they are they sometimes use lax definitions of "replicate" that wasn't what people had in mind, e.g. they'll get an effect size 10% of what was originally reported and still label that study "replicated" or they'll get totally different numbers but claim that because the trend is sort of similar, that's replication.

uniqueuid 4y ago
(I'm answering here even though the story has already dropped, because I think you raise a very important point.) All that is true, and we should realize that there are no silver bullets anywhere. Cohen/Gigerenzer's painful papers on power were some 40 years apart, yet nothing happened in between. There needs to be a systemic answer, you're completely right about that. But then, It's important to consider what the assumed model of change is. I believe deeply that influential individual cases can often push progress more than gradual toiling. Having a replication crisis with Daniel Kahneman writing "fucking solve it" emails definitely helps. That's because progress here is a ratchet (new methodological standards set a new lower bound). And more and more people realize that your research today is what you may be judged on tomorrow. So it's a dangerous game to intentionally deceive. In the end, science needs to be eventually correct in finite time. The steeper the slope, the better. The one thing we must not allow is for the slope to become negative.
- origin_path 4y ago
  I want to be as optimistic as you, I do. At the moment I'm not though. My perception of things like the push for replication or pre-registration is not as a one-way ratchet towards progress, however slow moving, but more like pushing down on a blob of mercury. The problems aren't actually solved but simply get moved around by an academic community that deep down doesn't want to improve, only to create enough of an appearance of improving to dispel criticism. For example, there was a big push to give people access to the raw data. Researchers were made to sign data availability statements. The appearance of improvement was created. Then one day someone decided to test if these statements meant anything and discovered they did not; compliance was extremely low [1]. This seems to be how it goes. After over a decade of noise about P-hacking and low replicability in the social sciences, estimated replicability of studies has hardly changed, nor has statistical power improved much [2]. And these are the subtle problems that people don't mind talking about because the problems are sufficiently opaque enough that the general public doesn't pay much attention. During COVID we saw just how deep and serious bad science can get. Climatology isn't better. P-hacking is an irrelevant concern if you aren't even comparing your hypothesis to actual data to begin with (c.f. computational epidemiology), or if your response to a hypothesis/data mismatch is to edit the data (c.f. surface temperature datasets). Poor replicability is likewise irrelevant if the supposedly top researchers in a field can't implement a model that can even replicate its own outputs, and literally nobody anywhere within the institutions cares! The core problem here just doesn't seem to be something you can fix with this kind of incremental approach on top of the academic basis. You get the temporary appearance of progress in one or two fields where noise is being made, but it's fake and falls apart the moment someone examines it deeply or looks at a slightly different field. Ultimately, academia doesn't institutionally care if the output of its professors is correct, only that output happens and gets attention. You can't fix this by requiring more forms to be filled out; these people are smart and have all day to find workarounds. For as long as their position and status in society revolves around creating mass/public perception that they're doing intelligent work, vs actual hard-nosed evaluation of said work by independent but deeply interested outsiders (e.g. companies that want to build things), we will continue to be disappointed by the failure of reform efforts. [1] https://www.jclinepi.com/article/S0895-4356(22)00141-X/pdf https://www.jclinepi.com/article/S0895-4356(22)00141-X/pdf [2] https://fantasticanachronism.com/2020/09/11/whats-wrong-with-social-science-and-how-to-fix-it/#Things-Are-Not-Getting-Better https://fantasticanachronism.com/2020/09/11/whats-wrong-with...

uniqueuid 4y ago

(I'm answering here even though the story has already dropped, because I think you raise a very important point.) All that is true, and we should realize that there are no silver bullets anywhere. Cohen/Gigerenzer's painful papers on power were some 40 years apart, yet nothing happened in between. There needs to be a systemic answer, you're completely right about that. But then, It's important to consider what the assumed model of change is. I believe deeply that influential individual cases can often push progress more than gradual toiling. Having a replication crisis with Daniel Kahneman writing "fucking solve it" emails definitely helps. That's because progress here is a ratchet (new methodological standards set a new lower bound). And more and more people realize that your research today is what you may be judged on tomorrow. So it's a dangerous game to intentionally deceive. In the end, science needs to be eventually correct in finite time. The steeper the slope, the better. The one thing we must not allow is for the slope to become negative.

origin_path 4y ago
I want to be as optimistic as you, I do. At the moment I'm not though. My perception of things like the push for replication or pre-registration is not as a one-way ratchet towards progress, however slow moving, but more like pushing down on a blob of mercury. The problems aren't actually solved but simply get moved around by an academic community that deep down doesn't want to improve, only to create enough of an appearance of improving to dispel criticism. For example, there was a big push to give people access to the raw data. Researchers were made to sign data availability statements. The appearance of improvement was created. Then one day someone decided to test if these statements meant anything and discovered they did not; compliance was extremely low [1]. This seems to be how it goes. After over a decade of noise about P-hacking and low replicability in the social sciences, estimated replicability of studies has hardly changed, nor has statistical power improved much [2]. And these are the subtle problems that people don't mind talking about because the problems are sufficiently opaque enough that the general public doesn't pay much attention. During COVID we saw just how deep and serious bad science can get. Climatology isn't better. P-hacking is an irrelevant concern if you aren't even comparing your hypothesis to actual data to begin with (c.f. computational epidemiology), or if your response to a hypothesis/data mismatch is to edit the data (c.f. surface temperature datasets). Poor replicability is likewise irrelevant if the supposedly top researchers in a field can't implement a model that can even replicate its own outputs, and literally nobody anywhere within the institutions cares! The core problem here just doesn't seem to be something you can fix with this kind of incremental approach on top of the academic basis. You get the temporary appearance of progress in one or two fields where noise is being made, but it's fake and falls apart the moment someone examines it deeply or looks at a slightly different field. Ultimately, academia doesn't institutionally care if the output of its professors is correct, only that output happens and gets attention. You can't fix this by requiring more forms to be filled out; these people are smart and have all day to find workarounds. For as long as their position and status in society revolves around creating mass/public perception that they're doing intelligent work, vs actual hard-nosed evaluation of said work by independent but deeply interested outsiders (e.g. companies that want to build things), we will continue to be disappointed by the failure of reform efforts. [1] https://www.jclinepi.com/article/S0895-4356(22)00141-X/pdf https://www.jclinepi.com/article/S0895-4356(22)00141-X/pdf [2] https://fantasticanachronism.com/2020/09/11/whats-wrong-with-social-science-and-how-to-fix-it/#Things-Are-Not-Getting-Better https://fantasticanachronism.com/2020/09/11/whats-wrong-with...