3 ms·

FWIW, I think it already has, actually, multiple times. My guess there are few factors in place -- people recognized motivated reasoning -- there are many stud

FWIW, I think it already has, actually, multiple times. My guess there are few factors in place -- people recognized motivated reasoning -- there are many studies the authors clearly want to be true, e.g. "People I disagree with are evil ugly dum-dums" is a recurring one. There's also the studies that go against our experience, or what seems obviously right "Mormons are more likely to be alcoholics". These have a tendency to grab headlines (and likely grant money), and while occasionally are true, generally seem not to be.

vlovich123 5y ago
This is the line I'm betting on. > most people can successfully predict which psychological/behavioral studies will replicate Here's the underlying claim: > Results showed that these laypeople predicted replication success with above-chance accuracy (i.e., 59%). In addition, when participants were informed about the strength of evidence from the original studies, this boosted their prediction performance to 67% My problem is with what does "above chance" really mean. > For instance, several systematic high-powered replication projects have demonstrated successful replication rates ranging from 36% (Open Science Collaboration, 2015) to 50% (Klein et al., 2018), 62% (Camerer et al., 2018), and 85% (Klein et al., 2014) It sounds like we don't actually know what our ground truth replication rate is. I think what we're actually above-chance predicting is not the replication rate but just the baseline uncertainty challenge of scientific funding ("I have $X and I need to spend it on $SCIENCE - here go do some $SCIENCE things because my experts tell me you're good at $SCIENCE") + the low-level corruption it brings which blinds us to reality. > If laypeople can indeed make accurate predictions about replicability, these predictions may supplement theoretical considerations concerning the selection of candidate studies for replication projects. Given limited resources, laypeople’s predictions concerning replicability could be used to define the subset of studies for which one can expect to learn the most from the data. In other words, researchers could use laypeople’s predictions as input to assess information gain in a quantitative decision-making framework for replication (Hardwicke, Tessler, Peloquin, & Frank, 2018; MacKay, 1992). This framework follows the intuition that—for original studies with surprising effects (i.e., low plausibility) or small sample sizes (i.e., little evidence)—replications can bring about considerable informational gain. I'm betting against ^this as being the replicatable result (that this produces better scientific outcomes). That's just saying "humans have confirmation bias". This is a useful tool to understand our how we should remove confirmation bias from our experiment selection/setup NOT as a way to guide the selection/setup itself blindly. Imagine if you said "we should let the majority of people weigh in on surprising mathematical/biotech/chemistry/biology/drug discovery results". That's an INSANE statement for anyone in the business of scientific discovery. To me what it signals is that the people doing "social science" isn't building good theoretical models of how humans work and therefore the baseline quality of research is insanely low. This makes sense when you consider that the financial incentives and difficulty of the problem interplay with each other. There's some very good people who are doing the work of wading through the garbage, but they can't keep up with the volume of bullshit that's getting generated. So if you're willing to place a bet on that specific hypothesis, let me know.

vlovich123 5y ago

This is the line I'm betting on. > most people can successfully predict which psychological/behavioral studies will replicate Here's the underlying claim: > Results showed that these laypeople predicted replication success with above-chance accuracy (i.e., 59%). In addition, when participants were informed about the strength of evidence from the original studies, this boosted their prediction performance to 67% My problem is with what does "above chance" really mean. > For instance, several systematic high-powered replication projects have demonstrated successful replication rates ranging from 36% (Open Science Collaboration, 2015) to 50% (Klein et al., 2018), 62% (Camerer et al., 2018), and 85% (Klein et al., 2014) It sounds like we don't actually know what our ground truth replication rate is. I think what we're actually above-chance predicting is not the replication rate but just the baseline uncertainty challenge of scientific funding ("I have $X and I need to spend it on $SCIENCE - here go do some $SCIENCE things because my experts tell me you're good at $SCIENCE") + the low-level corruption it brings which blinds us to reality. > If laypeople can indeed make accurate predictions about replicability, these predictions may supplement theoretical considerations concerning the selection of candidate studies for replication projects. Given limited resources, laypeople’s predictions concerning replicability could be used to define the subset of studies for which one can expect to learn the most from the data. In other words, researchers could use laypeople’s predictions as input to assess information gain in a quantitative decision-making framework for replication (Hardwicke, Tessler, Peloquin, & Frank, 2018; MacKay, 1992). This framework follows the intuition that—for original studies with surprising effects (i.e., low plausibility) or small sample sizes (i.e., little evidence)—replications can bring about considerable informational gain. I'm betting against ^this as being the replicatable result (that this produces better scientific outcomes). That's just saying "humans have confirmation bias". This is a useful tool to understand our how we should remove confirmation bias from our experiment selection/setup NOT as a way to guide the selection/setup itself blindly. Imagine if you said "we should let the majority of people weigh in on surprising mathematical/biotech/chemistry/biology/drug discovery results". That's an INSANE statement for anyone in the business of scientific discovery. To me what it signals is that the people doing "social science" isn't building good theoretical models of how humans work and therefore the baseline quality of research is insanely low. This makes sense when you consider that the financial incentives and difficulty of the problem interplay with each other. There's some very good people who are doing the work of wading through the garbage, but they can't keep up with the volume of bullshit that's getting generated. So if you're willing to place a bet on that specific hypothesis, let me know.