6 ms·

If that is the case, why did the replication crisis occur despite peer review? And why don’t most studies publish code or data? If you look closely most papers

If that is the case, why did the replication crisis occur despite peer review? And why don’t most studies publish code or data? If you look closely most papers today are not reproducible. Consensus is a horrible way to judge science. The whole point of basing knowledge on empirical data is so that you don’t need to defer to authorities or consensus (nullius in verba). Consensus based science is just a synonym for Kuhnian paradigms (aka a scientific popularity contest). See Lakatos and Feyerabend for why this is not a logical or useful way to judge science. Imagine a field where the consensus methods are later found to be in error. In this case the only valuable thing to do is to challenge the consensus so the field adopts better methods. In the meantime, following the consensus guarantees following falsehoods because everyone is wrong in the same way. In many cases the minority heterodox views are the most correct.

Guvante 3y ago
Replication crisis occurred because nobody replicated. It wasn't that you couldn't (although I slightly overgeneralized the ease of replication I will admit). Pretending that the tools we have aren't useful because they aren't perfect isn't valuable. Heliocentric models were good for science even if they were wrong. You aren't saying anything that moves the line towards a better situation just pointlessly complaining that it isn't good enough.
- brutusborn 3y ago
  I think you should reread the original replication crisis paper, a huge proportion were simply not able to be replicated based on information in the original papers. I agree heliocentric models were useful, but if scientists had followed the consensus methods that produced heliocentrism then we would not have gone past it. It was heterodox theorists who resisted Catholic authority who progressed science. The Vatican are not good peer-reviewers. If you want a better solution look to Lakatos’ Methodology of Scientific Research Programmes. It is much better than the status quo of relying on peer review. It is rarely used due to inertia of the status quo, but methods like it will become essential as scientific failures mount. I don’t think my complaints are pointless, we need to take scientific failures very seriously because they eventually become policy failures and cause all kinds of suffering. Also, I’m not pretending that tools like peer review are useless, I’m arguing that they are useless and providing alternatives that have been developed over decades of epistemological progress.
  - Guvante 3y ago
    My understanding of the paper is less "we couldn't reproduce this paper" and more "insufficient data is available to guarantee any replication is identical". Basically how can you be sure your difference points to a mistake in the original vs a mistake in repetition. Additionally I think it was also a little "replication would be cheaper if we did this". However I think calling it fundamental is a bit of a stress. After all such a paper is going to heavily focus on such details by necessity. You have to categorize to perform such a study and blockers become glaringly obvious. However IMHO I think that a failed replication caused by inaccurate replication isn't a lost cause and instead acts as a new lens to view the original in. "You forgot to mention whether you controlled for X so I did but it didn't work" is valuable but certainly harder to catch in peer review (where the explicit mention might have caused questions). Put another way we can replicate as well as we need to if we prioritize it and while better steps are important they aren't as important as actually replicating. You only once mentioned a single theory which only touches on a part of the replication crisis and not the core issue of "no one can replicate with no funds to do so"...
    - brutusborn 3y ago
      I completely agree that replication is essential, but am arguing that peer review and consensus based science do not help improve replication and thus do not produce better science (generally). We put huge amounts of (unpaid) scientific resources towards peer review and it has hardly improved anything. It’s the opposite: “it’s peer reviewed” has become a reflexive defence of published papers with the incorrect implicit assumption that peer review makes the science better. We should focus on the initiatives that have a proven track record of improving things (like open science) and abandon peer review and reliance on consensus. As you point out, financial limits prevent replication. So why do we spend so much time and money on fancy journals and reviewers who have such a terrible track record of improving things? We could be replicating instead. For a recent example of consensus failures, see Nobel Prize winner Katalin Karikó whos seminal paper was rejected by Nature and whos research programme was rejected by her University for being too incremental. Her peers failed because they relied on flawed heuristics like consensus.
- absrec 3y ago
  I have read a lot of papers that are vague with the details of implementation. Some release code that isn't portable at all and probably works only on their machine with 0 documentation. A lot of obfuscation in general. Poor documentation and omission of critical implementation details means it's way too hard to replicate a paper. And even if you do replicate there's no way of finding out if the output is wrong because of your implementation or the idea itself is flawed. Doing proper documentation should be mandatory. I can only talk about CS but yeah more information.
jltsiren 3y ago
Without consensus-building there is only madness. Either you accept that some authority (possibly you) is the ultimate arbitrer of truth. Anyone who disagrees with them is automatically wrong. Or everyone who believes that their opinions are scientifically justified is equally right, as long as they stick to their beliefs. Or you choose to believe that you can always trust your own judgment, ignoring the evidence that often points to the contrary. Consensus-building is about crowdsourcing the truth. You reject both authorities and relativism. You believe that, in many cases, the objective truth exists, even if it can be difficult to reach by fallible humans. You are also optimistic enough to believe that experts who share your worldview will accept true arguments and reject false arguments. Eventually, on the average, and with a high probability.
- brutusborn 3y ago
  Basing truth on consensus is madness. Science _is_ crowdsourcing the truth, by using ideas that other people generate and then testing them empirically to verify them. This does not require consensus; rather consensus causes problems which make science worse, disagreements are always better for progress. We don’t have to accept current problems when there are better alternatives (see Lakatos for a good example).
  - jltsiren 3y ago
    But how do you know that the idea was verified successfully? It's common that some people manage to replicate a result, while others fail at it. No matter how many levels of indirection you add, you are still facing the same problem. Do you choose to believe in an authority, or is everyone equally right? Do you believe in your own infallibility? Or do you resort to consensus-building? Disagreements have a key role in building a consensus. But those who disagree with the consensus are usually wrong, because the world is full of capable people with contrarian tendencies and weird ideas.
    - brutusborn 3y ago
      You will never “know” with certainty because of the limits of inductive reasoning to prove theories, but you _can_ choose demarcation methods that don’t rely on a consensus. The consensus might be correct more of the time, but that shouldn’t be a scientific reason to follow the consensus. You should follow the consensus because you have some empirical reason to agree with it and disagree with it if you have reason to believe a better theory. The only person who needs to defer to an authority reflexively is someone who is completely ignorant of science. They don’t have the sophistication to judge, but thinking that this extends to sophisticated actors is an error. Consensus is for the ignorant.

Guvante 3y ago

Replication crisis occurred because nobody replicated. It wasn't that you couldn't (although I slightly overgeneralized the ease of replication I will admit). Pretending that the tools we have aren't useful because they aren't perfect isn't valuable. Heliocentric models were good for science even if they were wrong. You aren't saying anything that moves the line towards a better situation just pointlessly complaining that it isn't good enough.

brutusborn 3y ago
I think you should reread the original replication crisis paper, a huge proportion were simply not able to be replicated based on information in the original papers. I agree heliocentric models were useful, but if scientists had followed the consensus methods that produced heliocentrism then we would not have gone past it. It was heterodox theorists who resisted Catholic authority who progressed science. The Vatican are not good peer-reviewers. If you want a better solution look to Lakatos’ Methodology of Scientific Research Programmes. It is much better than the status quo of relying on peer review. It is rarely used due to inertia of the status quo, but methods like it will become essential as scientific failures mount. I don’t think my complaints are pointless, we need to take scientific failures very seriously because they eventually become policy failures and cause all kinds of suffering. Also, I’m not pretending that tools like peer review are useless, I’m arguing that they are useless and providing alternatives that have been developed over decades of epistemological progress.
- Guvante 3y ago
  My understanding of the paper is less "we couldn't reproduce this paper" and more "insufficient data is available to guarantee any replication is identical". Basically how can you be sure your difference points to a mistake in the original vs a mistake in repetition. Additionally I think it was also a little "replication would be cheaper if we did this". However I think calling it fundamental is a bit of a stress. After all such a paper is going to heavily focus on such details by necessity. You have to categorize to perform such a study and blockers become glaringly obvious. However IMHO I think that a failed replication caused by inaccurate replication isn't a lost cause and instead acts as a new lens to view the original in. "You forgot to mention whether you controlled for X so I did but it didn't work" is valuable but certainly harder to catch in peer review (where the explicit mention might have caused questions). Put another way we can replicate as well as we need to if we prioritize it and while better steps are important they aren't as important as actually replicating. You only once mentioned a single theory which only touches on a part of the replication crisis and not the core issue of "no one can replicate with no funds to do so"...
  - brutusborn 3y ago
    I completely agree that replication is essential, but am arguing that peer review and consensus based science do not help improve replication and thus do not produce better science (generally). We put huge amounts of (unpaid) scientific resources towards peer review and it has hardly improved anything. It’s the opposite: “it’s peer reviewed” has become a reflexive defence of published papers with the incorrect implicit assumption that peer review makes the science better. We should focus on the initiatives that have a proven track record of improving things (like open science) and abandon peer review and reliance on consensus. As you point out, financial limits prevent replication. So why do we spend so much time and money on fancy journals and reviewers who have such a terrible track record of improving things? We could be replicating instead. For a recent example of consensus failures, see Nobel Prize winner Katalin Karikó whos seminal paper was rejected by Nature and whos research programme was rejected by her University for being too incremental. Her peers failed because they relied on flawed heuristics like consensus.
absrec 3y ago
I have read a lot of papers that are vague with the details of implementation. Some release code that isn't portable at all and probably works only on their machine with 0 documentation. A lot of obfuscation in general. Poor documentation and omission of critical implementation details means it's way too hard to replicate a paper. And even if you do replicate there's no way of finding out if the output is wrong because of your implementation or the idea itself is flawed. Doing proper documentation should be mandatory. I can only talk about CS but yeah more information.

jltsiren 3y ago

Without consensus-building there is only madness. Either you accept that some authority (possibly you) is the ultimate arbitrer of truth. Anyone who disagrees with them is automatically wrong. Or everyone who believes that their opinions are scientifically justified is equally right, as long as they stick to their beliefs. Or you choose to believe that you can always trust your own judgment, ignoring the evidence that often points to the contrary. Consensus-building is about crowdsourcing the truth. You reject both authorities and relativism. You believe that, in many cases, the objective truth exists, even if it can be difficult to reach by fallible humans. You are also optimistic enough to believe that experts who share your worldview will accept true arguments and reject false arguments. Eventually, on the average, and with a high probability.

brutusborn 3y ago
Basing truth on consensus is madness. Science _is_ crowdsourcing the truth, by using ideas that other people generate and then testing them empirically to verify them. This does not require consensus; rather consensus causes problems which make science worse, disagreements are always better for progress. We don’t have to accept current problems when there are better alternatives (see Lakatos for a good example).
- jltsiren 3y ago
  But how do you know that the idea was verified successfully? It's common that some people manage to replicate a result, while others fail at it. No matter how many levels of indirection you add, you are still facing the same problem. Do you choose to believe in an authority, or is everyone equally right? Do you believe in your own infallibility? Or do you resort to consensus-building? Disagreements have a key role in building a consensus. But those who disagree with the consensus are usually wrong, because the world is full of capable people with contrarian tendencies and weird ideas.
  - brutusborn 3y ago
    You will never “know” with certainty because of the limits of inductive reasoning to prove theories, but you _can_ choose demarcation methods that don’t rely on a consensus. The consensus might be correct more of the time, but that shouldn’t be a scientific reason to follow the consensus. You should follow the consensus because you have some empirical reason to agree with it and disagree with it if you have reason to believe a better theory. The only person who needs to defer to an authority reflexively is someone who is completely ignorant of science. They don’t have the sophistication to judge, but thinking that this extends to sophisticated actors is an error. Consensus is for the ignorant.