4 ms·

> the scientific community's reaction to the replication crisis has been to ignore it What can they do? It's an incredibly hard problem to solve. It's like ask

> the scientific community's reaction to the replication crisis has been to ignore it What can they do? It's an incredibly hard problem to solve. It's like asking why the buisness community has done nothing to adress the housing crisis. Large scale culture changes, or the entire structure of the way science is conducted and funded, would be the only solutions

colechristensen 3y ago
If a university was actually in the business of seeking truth, there is much they could do to solve this. * working groups "red teams" whose whole job it is to find weaknesses in papers published at the university * post mortems after finding papers with serious flaws exposing the problem and coming up with constructive corrective actions * funding / forcing researchers to devote a certain amount of time to replicating significant results * working groups of experts in statistics and study design available to consult * systems to register studies, methodologies, data sets, etc. with the aim of eliminating error and preventing post-hoc fishing expeditions for results The whole-ass purpose of a university is seeking knowledge. They are fully capable of doing a better job of it but they don't because what they actually focus on are things like fundraising, professional sports teams, constructing attractive buildings, and advancing ranking. Most universities would be better off just firing and not replacing 90% of their administration.
- jltsiren 3y ago
  Universities can't do that, because they lack the expertise. Most of the time, the only people in a university capable of judging a study are the people who did it. That's because universities are in the business of teaching. Apart from a few rare exceptions, universities don't have the money to hire redundant people. Instead of hiring many experts in the same topic, they prefer hiring a wider range of expertise, in order to provide better learning opportunities for the students.
  - mkl 3y ago
    > Universities can't do that, because they lack the expertise. Most of the time, the only people in a university capable of judging a study are the people who did it. Statisticians can find faults in many studies. Universities even employ statisticians to act as consultants for research in other departments, as that raises the quality of the institution's publications.
    - jltsiren 3y ago
      Statisticians can find faults in many studies, but without a sufficient background in the topic, they can rarely tell if those faults actually matter. And the statisticians who have the background are usually already involved in the study anyway.
some_random 3y ago
Maybe brush up on undergraduate statistics and experimental design so that they don't execute and publish obviously flawed studies? Perhaps they could apply such basic knowledge to the peer review process which is supposed to catch these things. They could stop defending their peers who publish bogus research, stop teaching about obviously flawed nonsense like the Stanford Prison "Experiment" and hold "scientists" like Philip Zimbardo in contempt. It's like asking how individual cops can make things better, of course they can't fix systematic problems but that doesn't mean there's nothing they can do.
LudwigNagasena 3y ago
It doesn’t have to be large scale. It can start with a single journal, with a single university, with a single scientist. Make peer review public. Weight replication studies more. Make conducted peer reviews and replications an important metric for tenure and grants. Publish data and code alongside papers.
- mike_hearn 3y ago
  Some of those things were tried already. Some peer reviews are public, I've read them. Data and code are sometimes published too. It doesn't help. It just means outsiders get to read bad peer reviews and bad code, but the right people within the system don't actually care. There's no way for the public to hold anyone to account, science is a law unto itself because politicians are generally not scientifically minded. They can be easily bullied with technobabble. The few that aren't susceptible to this don't tend to rise to the top.
JacobThreeThree 3y ago
>What can they do? There's literally thousands of different things they could do. But why do anything when the real business of academia is in the tax-free hedge funds they operate and the government-subsidized international student gravy train? There's no short-term incentive to change anything.

colechristensen 3y ago

If a university was actually in the business of seeking truth, there is much they could do to solve this. * working groups "red teams" whose whole job it is to find weaknesses in papers published at the university * post mortems after finding papers with serious flaws exposing the problem and coming up with constructive corrective actions * funding / forcing researchers to devote a certain amount of time to replicating significant results * working groups of experts in statistics and study design available to consult * systems to register studies, methodologies, data sets, etc. with the aim of eliminating error and preventing post-hoc fishing expeditions for results The whole-ass purpose of a university is seeking knowledge. They are fully capable of doing a better job of it but they don't because what they actually focus on are things like fundraising, professional sports teams, constructing attractive buildings, and advancing ranking. Most universities would be better off just firing and not replacing 90% of their administration.

jltsiren 3y ago
Universities can't do that, because they lack the expertise. Most of the time, the only people in a university capable of judging a study are the people who did it. That's because universities are in the business of teaching. Apart from a few rare exceptions, universities don't have the money to hire redundant people. Instead of hiring many experts in the same topic, they prefer hiring a wider range of expertise, in order to provide better learning opportunities for the students.
- mkl 3y ago
  > Universities can't do that, because they lack the expertise. Most of the time, the only people in a university capable of judging a study are the people who did it. Statisticians can find faults in many studies. Universities even employ statisticians to act as consultants for research in other departments, as that raises the quality of the institution's publications.
  - jltsiren 3y ago
    Statisticians can find faults in many studies, but without a sufficient background in the topic, they can rarely tell if those faults actually matter. And the statisticians who have the background are usually already involved in the study anyway.

some_random 3y ago

Maybe brush up on undergraduate statistics and experimental design so that they don't execute and publish obviously flawed studies? Perhaps they could apply such basic knowledge to the peer review process which is supposed to catch these things. They could stop defending their peers who publish bogus research, stop teaching about obviously flawed nonsense like the Stanford Prison "Experiment" and hold "scientists" like Philip Zimbardo in contempt. It's like asking how individual cops can make things better, of course they can't fix systematic problems but that doesn't mean there's nothing they can do.

LudwigNagasena 3y ago

It doesn’t have to be large scale. It can start with a single journal, with a single university, with a single scientist. Make peer review public. Weight replication studies more. Make conducted peer reviews and replications an important metric for tenure and grants. Publish data and code alongside papers.

mike_hearn 3y ago
Some of those things were tried already. Some peer reviews are public, I've read them. Data and code are sometimes published too. It doesn't help. It just means outsiders get to read bad peer reviews and bad code, but the right people within the system don't actually care. There's no way for the public to hold anyone to account, science is a law unto itself because politicians are generally not scientifically minded. They can be easily bullied with technobabble. The few that aren't susceptible to this don't tend to rise to the top.

JacobThreeThree 3y ago

>What can they do? There's literally thousands of different things they could do. But why do anything when the real business of academia is in the tax-free hedge funds they operate and the government-subsidized international student gravy train? There's no short-term incentive to change anything.