3 ms·

> The replication crisis has cast into doubt important research findings, such as the Stanford prison experiment. What other socially important findings have be

by hhs 6y ago

> The replication crisis has cast into doubt important research findings, such as the Stanford prison experiment. What other socially important findings have been undermined? How should we interpret scientific literature post-crisis? This should start early in middle school and high school, too. Surprised this idea of replication crisis fails to get more attention.

mlthoughts2018 6y ago
I’m not surprised. Science is subject to political control like anything else. Why does a study use this or that data? What flaws or assumptions do we explain away as permitted trade-offs rather than rejecting as lacking enough basic fidelity to even proceed. There are Overton Windows in science (when’s the last time a scientific paper suggested that a bloody revolution should be used to address climate change?) and ultimately what gets published is a media channel that only exists to suit purposes of those funding it or permitting it - wealthy think tanks with a particular agenda, political parties or specific administrations or legislators who need reelection or need to foment support for their favored plan or outrage for their opponent’s plan. A replication crisis is by design - the system doesn’t exist to publish the best available science, it exists to create credential and status hierarchy in which to control scientific authority and shape it as a political tool. To address this issue you’d have to first address general plutocracy which even 80000 Hours itself is enmeshed in.
- SQueeeeeL 6y ago
  Yes, academia is no better suited to handling a major internal crisis then any other institution. The intensives are just too skewed towards publishing. Every major player out there almost certainly has an un-reproducible paper they published under duress.
- hhs 6y ago
  Indeed, there should also be better data and research literacy. On the hopeful side: looking at how the field of medicine evolved over the centuries from outright quackery to better data and ethical disclosures from those who do the research. It’s still not perfect but it’s getting better.
rmah 6y ago
99.9% of teens do not have the knowledge, skills or capability to critically examine academic research. Hell, most secondary and high school teachers don't have those skills. Thus, all you can say is "be skeptical". But that is next to useless if the student has no basis to make an assessment -- all it will do is erode confidence in the process of science. Instead, IMO, students should be 1) given a firm foundation of well accepted knowledge and 2) taught how the process of science works. As an aside, I'd ask those who promote the idea that children should be taught "how to think" instead of "facts they can just look up" to think a bit more deeply about what they are saying. IMO, you cannot reach valid conclusions without facts. Facts are the foundation of knowledge, synthesis and analysis. Without facts, one cannot understand a topic or even know what questions to ask about it. Note, I'm not saying that children should be force fed facts via rote learning. But consider, what happens when someone believes they can think for themselves but is ignorant of topic they're thinking about?
- SpicyLemonZest 6y ago
  I'd expect them to iteratively learn the facts of the topic, just as the people who first discovered those facts would have had to. Many areas of knowledge are traditionally introduced that way. Of course it can't be the only teaching strategy, since many things couldn't be learned within a human lifespan without explicit instruction on the facts, but it plays an important pedagogical role. The replication crisis indicates, frankly, that confidence in the process of science should be eroded a bit. Many people I know treat scientific studies as true until proven otherwise, and the replication crisis demonstrates this isn't an accurate assessment.
  - scarmig 6y ago
    There's two different kinds of confidence in science. There's the type that uncritically accepts an article at face value (oranges cause/cure cancer! Time to ban/require orange consumption!), and there's the type that says "the scientific community thinks evolution is true, so even if individual articles can be criticized, on the whole I shouldn't discount it as a theory based on a single critique or mistake." The former is good to undermine; the latter bad. But the boundary between them can be murky.
deleted 6y ago
[deleted]
UncleMeat 6y ago
The prison experiment is a terrible example here because it is exactly the sort of thing that experts have discounted for ages but held sway in popular discourse. It isn't an example of the replication crisis, since experts correctly diagnosed its flaws. It is an example of failing to listen to current academics in social science, which is the opposite of what most people who complain about the replication crisis would tell us to do.

by hhs 6y ago

mlthoughts2018 6y ago

I’m not surprised. Science is subject to political control like anything else. Why does a study use this or that data? What flaws or assumptions do we explain away as permitted trade-offs rather than rejecting as lacking enough basic fidelity to even proceed. There are Overton Windows in science (when’s the last time a scientific paper suggested that a bloody revolution should be used to address climate change?) and ultimately what gets published is a media channel that only exists to suit purposes of those funding it or permitting it - wealthy think tanks with a particular agenda, political parties or specific administrations or legislators who need reelection or need to foment support for their favored plan or outrage for their opponent’s plan. A replication crisis is by design - the system doesn’t exist to publish the best available science, it exists to create credential and status hierarchy in which to control scientific authority and shape it as a political tool. To address this issue you’d have to first address general plutocracy which even 80000 Hours itself is enmeshed in.

SQueeeeeL 6y ago
Yes, academia is no better suited to handling a major internal crisis then any other institution. The intensives are just too skewed towards publishing. Every major player out there almost certainly has an un-reproducible paper they published under duress.
hhs 6y ago
Indeed, there should also be better data and research literacy. On the hopeful side: looking at how the field of medicine evolved over the centuries from outright quackery to better data and ethical disclosures from those who do the research. It’s still not perfect but it’s getting better.

rmah 6y ago

99.9% of teens do not have the knowledge, skills or capability to critically examine academic research. Hell, most secondary and high school teachers don't have those skills. Thus, all you can say is "be skeptical". But that is next to useless if the student has no basis to make an assessment -- all it will do is erode confidence in the process of science. Instead, IMO, students should be 1) given a firm foundation of well accepted knowledge and 2) taught how the process of science works. As an aside, I'd ask those who promote the idea that children should be taught "how to think" instead of "facts they can just look up" to think a bit more deeply about what they are saying. IMO, you cannot reach valid conclusions without facts. Facts are the foundation of knowledge, synthesis and analysis. Without facts, one cannot understand a topic or even know what questions to ask about it. Note, I'm not saying that children should be force fed facts via rote learning. But consider, what happens when someone believes they can think for themselves but is ignorant of topic they're thinking about?

SpicyLemonZest 6y ago
I'd expect them to iteratively learn the facts of the topic, just as the people who first discovered those facts would have had to. Many areas of knowledge are traditionally introduced that way. Of course it can't be the only teaching strategy, since many things couldn't be learned within a human lifespan without explicit instruction on the facts, but it plays an important pedagogical role. The replication crisis indicates, frankly, that confidence in the process of science should be eroded a bit. Many people I know treat scientific studies as true until proven otherwise, and the replication crisis demonstrates this isn't an accurate assessment.
- scarmig 6y ago
  There's two different kinds of confidence in science. There's the type that uncritically accepts an article at face value (oranges cause/cure cancer! Time to ban/require orange consumption!), and there's the type that says "the scientific community thinks evolution is true, so even if individual articles can be criticized, on the whole I shouldn't discount it as a theory based on a single critique or mistake." The former is good to undermine; the latter bad. But the boundary between them can be murky.

deleted 6y ago

[deleted]

UncleMeat 6y ago

The prison experiment is a terrible example here because it is exactly the sort of thing that experts have discounted for ages but held sway in popular discourse. It isn't an example of the replication crisis, since experts correctly diagnosed its flaws. It is an example of failing to listen to current academics in social science, which is the opposite of what most people who complain about the replication crisis would tell us to do.