12 ms·

In an age when nothing seems to replicate, this wouldn't surprise me. But 1. There is an obvious conflict of interest in having a fact checking organization cl

by rsj_hn 7y ago

In an age when nothing seems to replicate, this wouldn't surprise me. But 1. There is an obvious conflict of interest in having a fact checking organization claim to "debunk" studies showing that fact checking is ineffective. 2. This "refutation" is a press briefing, not a peer reviewed study, and doesn't have a lot of facts to back up the claim, other than looking at the N difference in only 7 handpicked studies. This is an amusing example of what happens when so called authorities are held to account. Let's say someone reports that fact checking organizations are a waste of taxpayer dollars. Then the fact checking organization comes out with a 'Fact Check: False'. It seems there should be some prohibition about being an authority on criticisms of yourself.

avs733 7y ago
I've always struggled with the phrase 'replication crisis.' Really we have a 'mindset to approach science crisis.' The replication crisis is a couple of things to me... 1) A problem with how research is published and treated as 'new knowledge' - some of which is being solved (slowly) by efforts such as the OSF. Other efforts include registered reports, journals for nonsignificant findings, etc. The problem is journals seek to publish a 'contribution' and define contribution in a way that misrepresents science. 2) A problem with the promotion and rewarding of academics...see 1 and also the drive to produce work in academia results in rewarding minuscule steps that don't have larger coherence because the paper not the impact is rewarded. People get tenure for publications not scientific discoveries. This obviously links to 1 3) Often replication failures are a feature not a bug - but fundamentally we think about and talk about science in way to absolutist of a frame. This is especially a problem in social sciences which try and perform 'science' to gain equal treatment to 'hard science.' They aren't worse or better...they are different, their science is harder because context matters - but it is still science. There are myriad studies in psychology performed on white men from Harvard in the 40's (forgive my facetiousness). If I rerun the experiment and it 'fails to replicate' is the theory 'wrong', 'invalid', 'incomplete', or context specific? The choice of term there is critical to a set of philosophical beliefs about science that can actually imperil science by making different experimental results into an inherently bad thing.

by rsj_hn 7y ago

avs733 7y ago

I've always struggled with the phrase 'replication crisis.' Really we have a 'mindset to approach science crisis.' The replication crisis is a couple of things to me... 1) A problem with how research is published and treated as 'new knowledge' - some of which is being solved (slowly) by efforts such as the OSF. Other efforts include registered reports, journals for nonsignificant findings, etc. The problem is journals seek to publish a 'contribution' and define contribution in a way that misrepresents science. 2) A problem with the promotion and rewarding of academics...see 1 and also the drive to produce work in academia results in rewarding minuscule steps that don't have larger coherence because the paper not the impact is rewarded. People get tenure for publications not scientific discoveries. This obviously links to 1 3) Often replication failures are a feature not a bug - but fundamentally we think about and talk about science in way to absolutist of a frame. This is especially a problem in social sciences which try and perform 'science' to gain equal treatment to 'hard science.' They aren't worse or better...they are different, their science is harder because context matters - but it is still science. There are myriad studies in psychology performed on white men from Harvard in the 40's (forgive my facetiousness). If I rerun the experiment and it 'fails to replicate' is the theory 'wrong', 'invalid', 'incomplete', or context specific? The choice of term there is critical to a set of philosophical beliefs about science that can actually imperil science by making different experimental results into an inherently bad thing.