3 ms·

Not being replicated isn't the same as not being replicable. There might simply be no interest at the time. But that is no the same as the results not being rig

Not being replicated isn't the same as not being replicable. There might simply be no interest at the time. But that is no the same as the results not being right or even valuable at some point. Replication is not a be all and end all, because things could replicate even though understanding is wrong (i.e., observation is right in the publication but everything else is wrong). Even the replication itself can be wrong, if certain materials share on unknown contaminant, for example. It addresses some issues, but not all. For some areas it also not clear what replication means. What is it for theology, for example?

naasking 3y ago
> But that is no the same as the results not being right or even valuable at some point. I can't imagine anyone who truly understands the current replication failure rates thinking that a single non-replicated study is valuable for anything other than informing what replications should be attempted. Just think about it: replication rate is generally less than 50%. That means you'll have a better chance of determining the truth on any question posed by a study by flipping a coin than by actually reading the study. > Replication is not a be all and end all, because things could replicate even though understanding is wrong (i.e., observation is right in the publication but everything else is wrong). The observations are the only things that we have to get right. Interpretations can be subject to decades of debate (some QM debates are ongoing a century later), but if the data isn't reliable then you're just wasting time debating falsehoods. Replications are critical to ensuring we have reliable empirical data. > Even the replication itself can be wrong, if certain materials share on unknown contaminant, for example. Indeed, and literally the only way to figure out that such variables exist and are affecting the results are by independent replications. The more replications, the better. One replication is a bare minimum threshold to demonstrate that the process of gathering the data is at least repeatable, in principle. > For some areas it also not clear what replication means. What is it for theology, for example? It makes perfect sense in any context where you're gathering empirical data. For instance, if you're surveying people's interpretation of "free will" [1], then the process via which you probe their views should be replicable. This means being clear about the specific phrasing of the questions asked, the environment in which they were asked, the makeup of the cohort, and so on. [1] https://www.researchgate.net/publication/274892120_Why_Compatibilist_Intuitions_are_not_Mistaken_A_Reply_to_Feltz_and_Millan https://www.researchgate.net/publication/274892120_Why_Compa...
- RandomLensman 3y ago
  This is just back to the old problem: most research is irrelevant to anyone beyond a few researchers and largely inconsequential to the world. This means, there will be no money for replication for most things. Anything truly critical will (eventually) go through some replication/control of sorts (but it can take a long time). You can either shut down most of research and then place your bets on what to keep and replicate, or you run broad but with a lot of incorrect stuff in it. If you go for the former, you run the risk that you keep the wrong things, though. You have to have a way to quantify the direct and indirect costs of all the bad research and see if that trades off vs a much smaller research surface. Not sure if that is the case - empirical data matters much less for a lot of big decisions than people often make it out.
  - naasking 3y ago
    > This is just back to the old problem: most research is irrelevant to anyone beyond a few researchers and largely inconsequential to the world. This means, there will be no money for replication for most things. If it's inconsequential, then wouldn't that money be better spent on replications or other research that is consequential? I'm not really clear on what you're suggesting. Although maybe I wasn't really clear on what I've been suggesting. Edit: to clarify, there are multiple ways to reorganize research. Consider an approach similar to physics, where there's an informal division between theoreticians and experimentalists. What if we have two different kinds of publications in social sciences, one that's proposing and/or refining experimental designs to correct possible sources of bias, and another type of publication that is publishing the results of conducting experiments that have been proposed. The experimentalists simply read proposals and apply for grants to conduct experiments, and multiple groups can do so completely independently. Conducting the experiment must strictly adhere to the proposed experimental design, no deviations can be permitted as is so common in social science when they find uninteresting results, otherwise this breaks the reliability of the results. A proposal should probably undergo a few rounds of refinement before experimentalists should feel confident in conducting the experiment, but I think the overall approach could work.
- notahacker 3y ago

naasking 3y ago

> But that is no the same as the results not being right or even valuable at some point. I can't imagine anyone who truly understands the current replication failure rates thinking that a single non-replicated study is valuable for anything other than informing what replications should be attempted. Just think about it: replication rate is generally less than 50%. That means you'll have a better chance of determining the truth on any question posed by a study by flipping a coin than by actually reading the study. > Replication is not a be all and end all, because things could replicate even though understanding is wrong (i.e., observation is right in the publication but everything else is wrong). The observations are the only things that we have to get right. Interpretations can be subject to decades of debate (some QM debates are ongoing a century later), but if the data isn't reliable then you're just wasting time debating falsehoods. Replications are critical to ensuring we have reliable empirical data. > Even the replication itself can be wrong, if certain materials share on unknown contaminant, for example. Indeed, and literally the only way to figure out that such variables exist and are affecting the results are by independent replications. The more replications, the better. One replication is a bare minimum threshold to demonstrate that the process of gathering the data is at least repeatable, in principle. > For some areas it also not clear what replication means. What is it for theology, for example? It makes perfect sense in any context where you're gathering empirical data. For instance, if you're surveying people's interpretation of "free will" [1], then the process via which you probe their views should be replicable. This means being clear about the specific phrasing of the questions asked, the environment in which they were asked, the makeup of the cohort, and so on. [1] https://www.researchgate.net/publication/274892120_Why_Compatibilist_Intuitions_are_not_Mistaken_A_Reply_to_Feltz_and_Millan https://www.researchgate.net/publication/274892120_Why_Compa...

RandomLensman 3y ago
This is just back to the old problem: most research is irrelevant to anyone beyond a few researchers and largely inconsequential to the world. This means, there will be no money for replication for most things. Anything truly critical will (eventually) go through some replication/control of sorts (but it can take a long time). You can either shut down most of research and then place your bets on what to keep and replicate, or you run broad but with a lot of incorrect stuff in it. If you go for the former, you run the risk that you keep the wrong things, though. You have to have a way to quantify the direct and indirect costs of all the bad research and see if that trades off vs a much smaller research surface. Not sure if that is the case - empirical data matters much less for a lot of big decisions than people often make it out.
- naasking 3y ago
  > This is just back to the old problem: most research is irrelevant to anyone beyond a few researchers and largely inconsequential to the world. This means, there will be no money for replication for most things. If it's inconsequential, then wouldn't that money be better spent on replications or other research that is consequential? I'm not really clear on what you're suggesting. Although maybe I wasn't really clear on what I've been suggesting. Edit: to clarify, there are multiple ways to reorganize research. Consider an approach similar to physics, where there's an informal division between theoreticians and experimentalists. What if we have two different kinds of publications in social sciences, one that's proposing and/or refining experimental designs to correct possible sources of bias, and another type of publication that is publishing the results of conducting experiments that have been proposed. The experimentalists simply read proposals and apply for grants to conduct experiments, and multiple groups can do so completely independently. Conducting the experiment must strictly adhere to the proposed experimental design, no deviations can be permitted as is so common in social science when they find uninteresting results, otherwise this breaks the reliability of the results. A proposal should probably undergo a few rounds of refinement before experimentalists should feel confident in conducting the experiment, but I think the overall approach could work.
notahacker 3y ago