3 ms·

It's also wrong. There have now been large-scale initiatives to reproduce papers, and they're getting higher failure rates than 14%. The statistics underlying

by timr 4y ago

It's also wrong. There have now been large-scale initiatives to reproduce papers, and they're getting higher failure rates than 14%. The statistics underlying all of this make a number of implicit assumptions that may or may not be true IRL (for example: that journals publish papers independent of the salaciousness of claims made within them). If Science picks out only the top-5% most-sensational claims for publication, then you can't assume that a 95% CI is a safe threshold. You've probably got to increase it to a much higher value to have any prayer of getting past the inherent bias in such a process.

genewitch 4y ago
Wait until you start looking in to, say, psychology papers, vis a vis reproducibility. I think that a lot of these studies should be reviewed during the actual research and study part by at least a sociologist or cultural anthropologist, and a mathematician with expertise in probability. When nearly every policy can be backed by "some papers published in <some journal>", stuff like this becomes important. For a recent example, the US White house is engaging in a 5 year study to determine if releasing sulfur dioxide into the stratosphere would accomplish enough "cooling" to offset the "climate crisis". the "climate crisis" is something that exists in computer models, few of the models agree, and if they do, it behooves us to look at when they started agreeing and find out why that change happened. a more personal example: I can't reproduce getting CaCO out of seawater, yet.
- tgv 4y ago
  I think you've never done psych research. First of all, statisticians and methodologists are frequently included, also in the design phase, if it's not a simple adaptation of an existing study. Nonetheless, many papers contain design errors. Second, sociology and anthropology are among the least empirical of academic disciplines. What reason would you have to involve them? Third, many studies are of the type "my theory predicts X and my experiment shows p < 0.05". That's almost wrong by definition. Bayesian theory explains why. Fourth, many studies have conclusions/claims that cannot be inferred from the actual finding. > When nearly every policy can be backed by "some papers published in <some journal>", stuff like this becomes important. It is important, but as long as academia is a "turn out papers for tenure" industry, it won't be fixed. Don't take the conclusions of a paper for true. They're meant for other academics.
  - genewitch 4y ago
    it seems that psychology papers extrapolate too much, and even "correcting for confounding factors" isn't enough, if the researchers don't know enough confounding factors, for one. And i think that R and python being popular in the academic fields belies the fact that no one wants to talk to statisticians about their data. It's a lot easier to massage inputs to a machine than a human. I was under the impression we were talking about reproducibility in this thread; therefore my point about policy being influenced by something you suggest is "meant for other academics"

by timr 4y ago

genewitch 4y ago

Wait until you start looking in to, say, psychology papers, vis a vis reproducibility. I think that a lot of these studies should be reviewed during the actual research and study part by at least a sociologist or cultural anthropologist, and a mathematician with expertise in probability. When nearly every policy can be backed by "some papers published in <some journal>", stuff like this becomes important. For a recent example, the US White house is engaging in a 5 year study to determine if releasing sulfur dioxide into the stratosphere would accomplish enough "cooling" to offset the "climate crisis". the "climate crisis" is something that exists in computer models, few of the models agree, and if they do, it behooves us to look at when they started agreeing and find out why that change happened. a more personal example: I can't reproduce getting CaCO out of seawater, yet.

tgv 4y ago
I think you've never done psych research. First of all, statisticians and methodologists are frequently included, also in the design phase, if it's not a simple adaptation of an existing study. Nonetheless, many papers contain design errors. Second, sociology and anthropology are among the least empirical of academic disciplines. What reason would you have to involve them? Third, many studies are of the type "my theory predicts X and my experiment shows p < 0.05". That's almost wrong by definition. Bayesian theory explains why. Fourth, many studies have conclusions/claims that cannot be inferred from the actual finding. > When nearly every policy can be backed by "some papers published in <some journal>", stuff like this becomes important. It is important, but as long as academia is a "turn out papers for tenure" industry, it won't be fixed. Don't take the conclusions of a paper for true. They're meant for other academics.
- genewitch 4y ago
  it seems that psychology papers extrapolate too much, and even "correcting for confounding factors" isn't enough, if the researchers don't know enough confounding factors, for one. And i think that R and python being popular in the academic fields belies the fact that no one wants to talk to statisticians about their data. It's a lot easier to massage inputs to a machine than a human. I was under the impression we were talking about reproducibility in this thread; therefore my point about policy being influenced by something you suggest is "meant for other academics"