4 ms·

Of course you will still need to trust others to reproduce results in areas with which you are unfamiliar, but this is still better than implicitly trusting pee

Of course you will still need to trust others to reproduce results in areas with which you are unfamiliar, but this is still better than implicitly trusting peer reviewers. Same thing with software security: if software is open source then I still need to trust someone else to determine if the source code is malicious, but it is better than implicitly trusting software because e.g. “a contractor has done a security audit.” Basically I don’t believe you can truly trust closed-anything, especially scientific analysis. You need _less_ trust when the thing is reproducible, which is the best you can ask for.

Guvante 3y ago
Science is very reproducibile by design, half the goal of peer review is to ensure that. The problem is the incentive structure is not aligned with that and so reproduction is never done. I would instead argue that consensus is the best bar we have to judge science. If the majority of author papers align that means either there is a conspiracy or some fraction of things was reproducibile.
- brutusborn 3y ago
  If that is the case, why did the replication crisis occur despite peer review? And why don’t most studies publish code or data? If you look closely most papers today are not reproducible. Consensus is a horrible way to judge science. The whole point of basing knowledge on empirical data is so that you don’t need to defer to authorities or consensus (nullius in verba). Consensus based science is just a synonym for Kuhnian paradigms (aka a scientific popularity contest). See Lakatos and Feyerabend for why this is not a logical or useful way to judge science. Imagine a field where the consensus methods are later found to be in error. In this case the only valuable thing to do is to challenge the consensus so the field adopts better methods. In the meantime, following the consensus guarantees following falsehoods because everyone is wrong in the same way. In many cases the minority heterodox views are the most correct.
  - Guvante 3y ago
    Replication crisis occurred because nobody replicated. It wasn't that you couldn't (although I slightly overgeneralized the ease of replication I will admit). Pretending that the tools we have aren't useful because they aren't perfect isn't valuable. Heliocentric models were good for science even if they were wrong. You aren't saying anything that moves the line towards a better situation just pointlessly complaining that it isn't good enough.
    - brutusborn 3y ago
      I think you should reread the original replication crisis paper, a huge proportion were simply not able to be replicated based on information in the original papers. I agree heliocentric models were useful, but if scientists had followed the consensus methods that produced heliocentrism then we would not have gone past it. It was heterodox theorists who resisted Catholic authority who progressed science. The Vatican are not good peer-reviewers. If you want a better solution look to Lakatos’ Methodology of Scientific Research Programmes. It is much better than the status quo of relying on peer review. It is rarely used due to inertia of the status quo, but methods like it will become essential as scientific failures mount. I don’t think my complaints are pointless, we need to take scientific failures very seriously because they eventually become policy failures and cause all kinds of suffering. Also, I’m not pretending that tools like peer review are useless, I’m arguing that they are useless and providing alternatives that have been developed over decades of epistemological progress.
      - Guvante 3y ago
        My understanding of the paper is less "we couldn't reproduce this paper" and more "insufficient data is available to guarantee any replication is identical". Basically how can you be sure your difference points to a mistake in the original vs a mistake in repetition. Additionally I think it was also a little "replication would be cheaper if we did this". However I think calling it fundamental is a bit of a stress. After all such a paper is going to heavily focus on such details by necessity. You have to categorize to perform such a study and blockers become glaringly obvious. However IMHO I think that a failed replication caused by inaccurate replication isn't a lost cause and instead acts as a new lens to view the original in. "You forgot to mention whether you controlled for X so I did but it didn't work" is valuable but certainly harder to catch in peer review (where the explicit mention might have caused questions). Put another way we can replicate as well as we need to if we prioritize it and while better steps are important they aren't as important as actually replicating. You only once mentioned a single theory which only touches on a part of the replication crisis and not the core issue of "no one can replicate with no funds to do so"...
- _aavaa_ 3y ago
  This is a pretty rosy interpretation of how the science sausage is made. 1. The scientific method might be reproducible by design, but there can be (and often is) a very large gap between "Science" and what gets published as a paper. 2. Peer review is a grab bag. Sometimes it's obvious the reviewers barely read or understood what was written. Other times they provide feedback that is asinine. Or simply they provide commends and you tell the editor you're ignoring those comments and the paper still gets published. 3. Only a fraction of reviewers (depending on field) will check that the paper includes enough information to be reproduced. This is easy to verify, just go ask different academics how many papers they read that don't include enough information to reproduce. And only a fraction of a fraction make any attempt at reproducing the paper to any degree before signing off on it. (and if they say that they couldn't reproduce it, then see the end of point 2 above) > If the majority of author papers align that means either there is a conspiracy or some fraction of things was reproducibile. This is a false dichotomy. It misses some very real dynamics, none of which are conspiratorial. 1. It's hard to publish negative results, especially if the paper written by someone influential in the field. Which means that there's a great disincentive to even try. 2. Competing theories are difficult to study not just because of peer review issues but also because of financial and repetitional issues. Getting funding is significantly harder if your work doesn't use the methods that the funding agencies expect. Same idea with reputation, hard to get your career started if you want to work on theories that go against the prevailing theory or if you have work that contradicts it. This is the whole idea behind the concept of paradigm shifts.
  - Guvante 3y ago
    The former issue will be easier to fix than the latter. After all going against the establishment generally results in failure. Quantum Mechanics and General Relativity may not agree with each other but they both have destroyed mountains of attempts to disprove them. I am not being rosy I am on the side of "we have to do something" contrasted against "all science is bad". Not saying you are saying that but some posters are. We can certainly improve things on many vectors but throwing away all existing science isn't the way forward either.
    - 3y ago
- explaininjs 3y ago
  Peer review has little to nothing to do with reproducibility. In fact I’d say is has absolutely nothing at all to do with it. Why? Because exactly as you say: you can get a paper “peer reviewed” despite it being totally bogus and irreproducible. But, as you also say, it’d be a Really Good Thing if it was about reproducibility! Imagine a world where instead of some people writing an essay, their “peers” giving it meaningless comments, and some editors at a paper selecting it to be enshrined as “valid”, we totally flipped the script: People perform a scientific observation. They record their methods and results, and put it into some freely accessible store of data regarding the question at hand. Anyone is free to consult the store for any question, and observe how many entries it has, and how their results compare. If an entry has very few results, the person consulting it with the question would be encouraged to create a reproduction of their own, and share the results they derived as a sibiling of the original paper.
  - brutusborn 3y ago
    It’s related because peer reviewers have the power to sink or float a paper based on its ability to reproduce, but they don’t currently. Thus there are not many incentives to try to make a paper reproducible. I couldn’t understand the last couple paragraphs. sarcasm?
    - explaininjs 3y ago
      An institution which fails to use a power is precisely equivalent to one that doesn’t hold that power at all. As for the last few, I’m basically just saying we should dismantle the current “journal” concept entirely (it stands only to benefit those who receive the fees it takes and those who derive self-worth from being published in a “prestigious” journal), and replace it with a system by which for any given scientifically testable hypothesis, a collection of many different reproduction attempts and their respective methodologies and results are immediately available all side-by-side. With that in place, no scientific result would derive any credibility from being “peer reviewed” or not, but rather from the quantity and diversity of reproduction attempts it has faced. This database should be free to query and free to insert into. Individual papers may support community comments to serve as the weak “peer review” we currently have, but at no point should these comments be considered anywhere near equivalent to a full reproduction attempt.
wolverine876 3y ago
How much research have you replicated?
- brutusborn 3y ago
  A handful of papers. Lots more if you include high school and undergrad lab assignments. In my field replication usually involves a lot of guesswork since papers don't usually have enough information to be certain of their exact methods (no code or data unless the planets align or the authors are friendly) and yet peer reviewers happily apply their rubber stamp.

Guvante 3y ago

Science is very reproducibile by design, half the goal of peer review is to ensure that. The problem is the incentive structure is not aligned with that and so reproduction is never done. I would instead argue that consensus is the best bar we have to judge science. If the majority of author papers align that means either there is a conspiracy or some fraction of things was reproducibile.

brutusborn 3y ago
If that is the case, why did the replication crisis occur despite peer review? And why don’t most studies publish code or data? If you look closely most papers today are not reproducible. Consensus is a horrible way to judge science. The whole point of basing knowledge on empirical data is so that you don’t need to defer to authorities or consensus (nullius in verba). Consensus based science is just a synonym for Kuhnian paradigms (aka a scientific popularity contest). See Lakatos and Feyerabend for why this is not a logical or useful way to judge science. Imagine a field where the consensus methods are later found to be in error. In this case the only valuable thing to do is to challenge the consensus so the field adopts better methods. In the meantime, following the consensus guarantees following falsehoods because everyone is wrong in the same way. In many cases the minority heterodox views are the most correct.
- Guvante 3y ago
  Replication crisis occurred because nobody replicated. It wasn't that you couldn't (although I slightly overgeneralized the ease of replication I will admit). Pretending that the tools we have aren't useful because they aren't perfect isn't valuable. Heliocentric models were good for science even if they were wrong. You aren't saying anything that moves the line towards a better situation just pointlessly complaining that it isn't good enough.
  - brutusborn 3y ago
    I think you should reread the original replication crisis paper, a huge proportion were simply not able to be replicated based on information in the original papers. I agree heliocentric models were useful, but if scientists had followed the consensus methods that produced heliocentrism then we would not have gone past it. It was heterodox theorists who resisted Catholic authority who progressed science. The Vatican are not good peer-reviewers. If you want a better solution look to Lakatos’ Methodology of Scientific Research Programmes. It is much better than the status quo of relying on peer review. It is rarely used due to inertia of the status quo, but methods like it will become essential as scientific failures mount. I don’t think my complaints are pointless, we need to take scientific failures very seriously because they eventually become policy failures and cause all kinds of suffering. Also, I’m not pretending that tools like peer review are useless, I’m arguing that they are useless and providing alternatives that have been developed over decades of epistemological progress.
    - Guvante 3y ago
      My understanding of the paper is less "we couldn't reproduce this paper" and more "insufficient data is available to guarantee any replication is identical". Basically how can you be sure your difference points to a mistake in the original vs a mistake in repetition. Additionally I think it was also a little "replication would be cheaper if we did this". However I think calling it fundamental is a bit of a stress. After all such a paper is going to heavily focus on such details by necessity. You have to categorize to perform such a study and blockers become glaringly obvious. However IMHO I think that a failed replication caused by inaccurate replication isn't a lost cause and instead acts as a new lens to view the original in. "You forgot to mention whether you controlled for X so I did but it didn't work" is valuable but certainly harder to catch in peer review (where the explicit mention might have caused questions). Put another way we can replicate as well as we need to if we prioritize it and while better steps are important they aren't as important as actually replicating. You only once mentioned a single theory which only touches on a part of the replication crisis and not the core issue of "no one can replicate with no funds to do so"...
_aavaa_ 3y ago
This is a pretty rosy interpretation of how the science sausage is made. 1. The scientific method might be reproducible by design, but there can be (and often is) a very large gap between "Science" and what gets published as a paper. 2. Peer review is a grab bag. Sometimes it's obvious the reviewers barely read or understood what was written. Other times they provide feedback that is asinine. Or simply they provide commends and you tell the editor you're ignoring those comments and the paper still gets published. 3. Only a fraction of reviewers (depending on field) will check that the paper includes enough information to be reproduced. This is easy to verify, just go ask different academics how many papers they read that don't include enough information to reproduce. And only a fraction of a fraction make any attempt at reproducing the paper to any degree before signing off on it. (and if they say that they couldn't reproduce it, then see the end of point 2 above) > If the majority of author papers align that means either there is a conspiracy or some fraction of things was reproducibile. This is a false dichotomy. It misses some very real dynamics, none of which are conspiratorial. 1. It's hard to publish negative results, especially if the paper written by someone influential in the field. Which means that there's a great disincentive to even try. 2. Competing theories are difficult to study not just because of peer review issues but also because of financial and repetitional issues. Getting funding is significantly harder if your work doesn't use the methods that the funding agencies expect. Same idea with reputation, hard to get your career started if you want to work on theories that go against the prevailing theory or if you have work that contradicts it. This is the whole idea behind the concept of paradigm shifts.
- Guvante 3y ago
  The former issue will be easier to fix than the latter. After all going against the establishment generally results in failure. Quantum Mechanics and General Relativity may not agree with each other but they both have destroyed mountains of attempts to disprove them. I am not being rosy I am on the side of "we have to do something" contrasted against "all science is bad". Not saying you are saying that but some posters are. We can certainly improve things on many vectors but throwing away all existing science isn't the way forward either.
  - 3y ago
explaininjs 3y ago
Peer review has little to nothing to do with reproducibility. In fact I’d say is has absolutely nothing at all to do with it. Why? Because exactly as you say: you can get a paper “peer reviewed” despite it being totally bogus and irreproducible. But, as you also say, it’d be a Really Good Thing if it was about reproducibility! Imagine a world where instead of some people writing an essay, their “peers” giving it meaningless comments, and some editors at a paper selecting it to be enshrined as “valid”, we totally flipped the script: People perform a scientific observation. They record their methods and results, and put it into some freely accessible store of data regarding the question at hand. Anyone is free to consult the store for any question, and observe how many entries it has, and how their results compare. If an entry has very few results, the person consulting it with the question would be encouraged to create a reproduction of their own, and share the results they derived as a sibiling of the original paper.
- brutusborn 3y ago
  It’s related because peer reviewers have the power to sink or float a paper based on its ability to reproduce, but they don’t currently. Thus there are not many incentives to try to make a paper reproducible. I couldn’t understand the last couple paragraphs. sarcasm?
  - explaininjs 3y ago
    An institution which fails to use a power is precisely equivalent to one that doesn’t hold that power at all. As for the last few, I’m basically just saying we should dismantle the current “journal” concept entirely (it stands only to benefit those who receive the fees it takes and those who derive self-worth from being published in a “prestigious” journal), and replace it with a system by which for any given scientifically testable hypothesis, a collection of many different reproduction attempts and their respective methodologies and results are immediately available all side-by-side. With that in place, no scientific result would derive any credibility from being “peer reviewed” or not, but rather from the quantity and diversity of reproduction attempts it has faced. This database should be free to query and free to insert into. Individual papers may support community comments to serve as the weak “peer review” we currently have, but at no point should these comments be considered anywhere near equivalent to a full reproduction attempt.

wolverine876 3y ago

How much research have you replicated?

brutusborn 3y ago
A handful of papers. Lots more if you include high school and undergrad lab assignments. In my field replication usually involves a lot of guesswork since papers don't usually have enough information to be certain of their exact methods (no code or data unless the planets align or the authors are friendly) and yet peer reviewers happily apply their rubber stamp.