11 ms·

In my experience, Linus's law [1] is false in scientific research. There's a popular view that peer review is a strong filter that allows other researchers to h

In my experience, Linus's law [1] is false in scientific research. There's a popular view that peer review is a strong filter that allows other researchers to have confidence that a claim is correct. My guess is that this might contribute to Linus's law being false, because many researchers actually do seem to think "it's been checked, so it must be okay, I don't need to check anything myself". Or, perhaps, people don't do much checking in the first place. My view is that peer review is a fairly low (and random) bar to pass that doesn't necessarily say much about the validity of a work. More people should check things more carefully. Recently I got reviews for a paper I wrote back, and a reviewer was skeptical of one claim I made because (paraphrasing) "If this were true, that would mean all previous researchers were wrong." They're exaggerating as it would only mean many researchers were wrong; I think some were skeptical of this for a long but kept quiet about it. But the idea that "everyone seemed to accept it, therefore you are probably wrong" independent of any arguments made seems popular, despite it being obviously fallacious. I have some arguments. The reviewer should engage with those arguments rather than make some appeal to popularity. Edit: To be clear, what I was claiming here would overturn something dating back to the 1930s that last received a well-accepted revision about 40 years ago. [1] https://en.wikipedia.org/wiki/Linus%27s_law https://en.wikipedia.org/wiki/Linus%27s_law

rtkwe 7y ago
I think the bigger reason it's false in science while somewhat more true in CS is that it's basically free to check the source code for something and setup a small localhost test where replication in science is tough and expensive. Even if someone does have the funds to check experimental descriptions don't actually cover all the details usually and there's some combination of small but important details left out and well meaning but imperfect directions. Also there's just the fact that in CS we're working on a defined system. From top to bottom it's possible to definitively say when I execute X the following Y things happen. The details may be buried deep within the hardware for the truly eldritch bugs but fundamentally it's all human defined and the bits that do exist outside human design are well understood enough they can largely be ignored (no one is reasonably going to find a bug that traces down to some unknown quirk of semiconductor physics). In the sciences we're poking at the edges to see what the system actually is.
- btrettel 7y ago
  While I agree that checking source code and running software is relatively easier than running experiments, there's more to science than experiments. Many wrong things in science are even easier to identify than the "easy" case you identify, but you still don't see them checked. In fact, the example I gave was for a math error that should be much easier to check than looking at someone's software or source code. All you needed to do was go through the claimed derivation to see that something was off. No fancy math was involved either, just algebra. I think problems like this are much more common than is appreciated.
  - rtkwe 7y ago
    Yeah the issue is probably partially weird results can be lazily explained as a number of sources; genuinely new science, experimental design problems, or equipment errors to list a few, so if you're quickly reviewing a paper it's easy to just gloss over checking all the math because it's easy to trust on that. With computers we know exactly what we expect a piece of code to do and if it doesn't we're not discovering some new truth about the universe it's just that something was done incorrectly or our assumptions were wrong if it's someone else's code/program/library.
ColanR 7y ago
I've been seeing discussions about the need for more transparent peer review, and more replication. It strikes me that if the peer reviews of an article were published with the article, like the commentary next to court cases in a law textbook, we would have a much more accurate idea of exactly what the peer review validated about the published article.
- bumby 7y ago
  I think fixing the peer review process may be easier than the replication crisis. All the incentives seem to align against performing work to replicated previously accepted science.
  - inetknght 7y ago
    > I think fixing the peer review process may be easier than the replication crisis. I think fixing the two are almost one and the same. Peer review without replication is just a place for ivory tower intellectuals to review from an armchair. Replication helps to demonstrate that other people have not only "reviewed" the work but have gone through the trouble of setting up the problem and re-identifying the (potentially undocumented) pain points.
    - bumby 7y ago
      I wish the downvotes would be accompanied with comments but I don't think your stance is completely unreasonable. I do think, however, it may not be tenable in the current structure. I think there is too much cost (both monetarily and opportunity) to expect replication to be part of the peer review process. But somehow, we still need to incentive replication.
    - PeterisP 7y ago
      Replication often (depends on the kind of science you're doing) takes a comparable amount of work to making the paper in the first place. If I'm a reviewer, then that means that I'm asked to volunteer a bit of my spare time to facilitate the evaluation of my peers by helping the editor (who's expected to be competent but not knowledgeable in every niche) to read the paper, and answer the following questions: (a) is this topic relevant and novel in my sub-niche; (b) is the content adequately described in a way that the target audience (i.e. people in my sub-niche like me) can understand clearly; and (c) does it have any obvious flaws or ignorance of things and previous research that are (or should be) well-known in my sub-niche. So I provide my niche-expert opinion about these questions to help the editor make their decision on which papers to include and what changes to mandate. That is peer review - in some sense, establishing whether the author is "behaving as a peer". Replication is something that comes after the study is published. If the system would require me to replicate your experiment for a review (assuming that I'm interested enough in the particular topic - I review papers that are tangentially interesting, but not the direction that I'd want to do myself), then that would be unreasonable to have it done as a volunteer service, it would require months of full time work to make a review, and who would pay for that work? If the authors would have to fund a replication before publishing, that places an enormous barrier on publishing and, frankly, I would want to read results that have not passed that barrier because I can't assume that everything that's useful for my research will be able to pass it. As the reader in the target audience for scientific publishing (i.e. other researchers actively working in that field - scientific journals and conferences are made by a niche of scientists for that niche of scientists, it's how we communicate between ourselves, disseminating knowledge to outsiders is not the primary goal for the vast majority of academic publication venues, that's what textbooks and monographies are for) I want authors to publish their results without a prolonged vetting, because I'd prefer to read them sooner rather than later.
bumby 7y ago
>"everyone seemed to accept it, therefore you are probably wrong" independent of any arguments made seems popular I've run into some of the same and it often reminds me of Kahneman's "theory induced blindness": "we trust a theory so much that we search for reasoning to reinforce it even when a specific situation arises that our model doesn’t suit well, when we can’t think about other options, it becomes inconceivable that the model is incorrect or not suitable in a given context." On the other hand, I've had items make it through the peer review filter that became obvious from the comments that the reviewers probably had little business reviewing that topic at all. It makes one rethink the value of the process.
moultano 7y ago
It seems like the peer review process is more likely to establish importance than correctness.
- btrettel 7y ago
  Maybe, but my fellow researchers' evaluations of importance seem to differ significantly from my own, so I'm not sure this is a good thing. I think it mostly helps improve the presentation of the paper by clarifying unclear parts.
hannob 7y ago
I think by now it should be pretty obvious what peer review is and isn't. I see it as a low bar to filter out some (yet not all) obvious bullshit. Wakefield's "Vaccines cause Autism" study was peer reviewed. All the fancy powerposing/social priming/etc. studies that created the psychology replication crisis were peer reviewed. Bem's precognition research was peer reviewed. As soon as you go into empirical research there are things that peer reviewers just can't check, e.g. whether the study author created the hypothesis before or after collecting the data. There's also things that could be checked, but usually aren't, like pretty much most of the software created for research.
- BrandoElFollito 7y ago
  Why is creating the hypothesis after having collected the data wrong?
majos 7y ago
Are you talking about math/CS theory or a more empirical field like economics or physics? Peer review is not perfect, but to the best of my knowledge long-standing errors in published work are still pretty rare. Especially because incorrect claims, when used, often generate incorrect results. Whereas the "models" aspect of economics and physics is less amenable to "verifying" things.
- btrettel 7y ago
  Empirical. I work in fluid dynamics. While I think purely formal fields like math have their advantages (as you point out), I would not be surprised if that literature is less reliable than most people believe. I know from my own experience that even highly cited papers with theoretical models can have surprisingly obvious major problems. Sometimes no empirical data is needed to know that something isn't right. The exact same reasons one might be suspicious of a pure math result can apply here.
  - atq2119 7y ago
    Having worked in discrete math and theoretical computer science myself, I agree with you especially when it comes to less well known / niche statements. Results that many other people build on tend to be solid though, because building on those results often requires the users to at least partially "reprove" them.
salty_biscuits 7y ago
"If this were true, that would mean all previous researchers were wrong." It's poor phrasing, but in a review I would interpret this statement as "I made a strong claim, and my discussion wasn't strong enough to address the concerns of a reviewer who has only interacted with my work via this document." not as "you must be wrong because everyone else says something different". Reviews are a weird social process though.
- btrettel 7y ago
  To be honest, I'm not sure what the reviewer is arguing, but your suggestion seems reasonable. Perhaps it's bad English. On a reread I get the impression that they were trying to say that I was being rude. Maybe I've misinterpreted them.
  - MereInterest 7y ago
    To add to that, it is better to explicitly point out disagreements with previous experiments. That shows that you have read the literature, seen that it is in disagreement, and are willing to stand by your measurement. Not pointing it out comes across as there being something to hide, and that the disagreement in the measurements is being snuck past the reviewers. Overall, you are absolutely correct that reviews are a really weird social context.
    - heavenlyblue 7y ago
      Aren’t you supposed to review the work, rather than the implications of said work?
      - atq2119 7y ago
        The two go hand in hand. After all, one way of proving that a statement is false is to prove that it leads to a contradiction.
      - setr 7y ago
        Yes, but it's so likely that your work is somehow wrong (in that it was misimplemented), that hueristics are employed to fail it early. If you can't easily point out why your work comes to a different conclusion than prior work, then it's likely you've failed to actually look into the prior work effectively. And it's very unlikely you've managed to upend a body of work you didn't even know existed, and has probably run into several errors before (which you probably didn't avoid, because you didn't look into it). If you did read through the previous literature, it shouldn't be difficult to specify and compare. It's possible for you to come from no background and upend everything.. but the chances are simply (and largely) in favor of a mistake being made. The onus is really on you to double-check and triple check your radical conclusions

rtkwe 7y ago

I think the bigger reason it's false in science while somewhat more true in CS is that it's basically free to check the source code for something and setup a small localhost test where replication in science is tough and expensive. Even if someone does have the funds to check experimental descriptions don't actually cover all the details usually and there's some combination of small but important details left out and well meaning but imperfect directions. Also there's just the fact that in CS we're working on a defined system. From top to bottom it's possible to definitively say when I execute X the following Y things happen. The details may be buried deep within the hardware for the truly eldritch bugs but fundamentally it's all human defined and the bits that do exist outside human design are well understood enough they can largely be ignored (no one is reasonably going to find a bug that traces down to some unknown quirk of semiconductor physics). In the sciences we're poking at the edges to see what the system actually is.

btrettel 7y ago
While I agree that checking source code and running software is relatively easier than running experiments, there's more to science than experiments. Many wrong things in science are even easier to identify than the "easy" case you identify, but you still don't see them checked. In fact, the example I gave was for a math error that should be much easier to check than looking at someone's software or source code. All you needed to do was go through the claimed derivation to see that something was off. No fancy math was involved either, just algebra. I think problems like this are much more common than is appreciated.
- rtkwe 7y ago
  Yeah the issue is probably partially weird results can be lazily explained as a number of sources; genuinely new science, experimental design problems, or equipment errors to list a few, so if you're quickly reviewing a paper it's easy to just gloss over checking all the math because it's easy to trust on that. With computers we know exactly what we expect a piece of code to do and if it doesn't we're not discovering some new truth about the universe it's just that something was done incorrectly or our assumptions were wrong if it's someone else's code/program/library.

ColanR 7y ago

I've been seeing discussions about the need for more transparent peer review, and more replication. It strikes me that if the peer reviews of an article were published with the article, like the commentary next to court cases in a law textbook, we would have a much more accurate idea of exactly what the peer review validated about the published article.

bumby 7y ago
I think fixing the peer review process may be easier than the replication crisis. All the incentives seem to align against performing work to replicated previously accepted science.
- inetknght 7y ago
  > I think fixing the peer review process may be easier than the replication crisis. I think fixing the two are almost one and the same. Peer review without replication is just a place for ivory tower intellectuals to review from an armchair. Replication helps to demonstrate that other people have not only "reviewed" the work but have gone through the trouble of setting up the problem and re-identifying the (potentially undocumented) pain points.
  - bumby 7y ago
    I wish the downvotes would be accompanied with comments but I don't think your stance is completely unreasonable. I do think, however, it may not be tenable in the current structure. I think there is too much cost (both monetarily and opportunity) to expect replication to be part of the peer review process. But somehow, we still need to incentive replication.
  - PeterisP 7y ago
    Replication often (depends on the kind of science you're doing) takes a comparable amount of work to making the paper in the first place. If I'm a reviewer, then that means that I'm asked to volunteer a bit of my spare time to facilitate the evaluation of my peers by helping the editor (who's expected to be competent but not knowledgeable in every niche) to read the paper, and answer the following questions: (a) is this topic relevant and novel in my sub-niche; (b) is the content adequately described in a way that the target audience (i.e. people in my sub-niche like me) can understand clearly; and (c) does it have any obvious flaws or ignorance of things and previous research that are (or should be) well-known in my sub-niche. So I provide my niche-expert opinion about these questions to help the editor make their decision on which papers to include and what changes to mandate. That is peer review - in some sense, establishing whether the author is "behaving as a peer". Replication is something that comes after the study is published. If the system would require me to replicate your experiment for a review (assuming that I'm interested enough in the particular topic - I review papers that are tangentially interesting, but not the direction that I'd want to do myself), then that would be unreasonable to have it done as a volunteer service, it would require months of full time work to make a review, and who would pay for that work? If the authors would have to fund a replication before publishing, that places an enormous barrier on publishing and, frankly, I would want to read results that have not passed that barrier because I can't assume that everything that's useful for my research will be able to pass it. As the reader in the target audience for scientific publishing (i.e. other researchers actively working in that field - scientific journals and conferences are made by a niche of scientists for that niche of scientists, it's how we communicate between ourselves, disseminating knowledge to outsiders is not the primary goal for the vast majority of academic publication venues, that's what textbooks and monographies are for) I want authors to publish their results without a prolonged vetting, because I'd prefer to read them sooner rather than later.

bumby 7y ago

>"everyone seemed to accept it, therefore you are probably wrong" independent of any arguments made seems popular I've run into some of the same and it often reminds me of Kahneman's "theory induced blindness": "we trust a theory so much that we search for reasoning to reinforce it even when a specific situation arises that our model doesn’t suit well, when we can’t think about other options, it becomes inconceivable that the model is incorrect or not suitable in a given context." On the other hand, I've had items make it through the peer review filter that became obvious from the comments that the reviewers probably had little business reviewing that topic at all. It makes one rethink the value of the process.

moultano 7y ago

It seems like the peer review process is more likely to establish importance than correctness.

btrettel 7y ago
Maybe, but my fellow researchers' evaluations of importance seem to differ significantly from my own, so I'm not sure this is a good thing. I think it mostly helps improve the presentation of the paper by clarifying unclear parts.

hannob 7y ago

I think by now it should be pretty obvious what peer review is and isn't. I see it as a low bar to filter out some (yet not all) obvious bullshit. Wakefield's "Vaccines cause Autism" study was peer reviewed. All the fancy powerposing/social priming/etc. studies that created the psychology replication crisis were peer reviewed. Bem's precognition research was peer reviewed. As soon as you go into empirical research there are things that peer reviewers just can't check, e.g. whether the study author created the hypothesis before or after collecting the data. There's also things that could be checked, but usually aren't, like pretty much most of the software created for research.

BrandoElFollito 7y ago
Why is creating the hypothesis after having collected the data wrong?

majos 7y ago

Are you talking about math/CS theory or a more empirical field like economics or physics? Peer review is not perfect, but to the best of my knowledge long-standing errors in published work are still pretty rare. Especially because incorrect claims, when used, often generate incorrect results. Whereas the "models" aspect of economics and physics is less amenable to "verifying" things.

btrettel 7y ago
Empirical. I work in fluid dynamics. While I think purely formal fields like math have their advantages (as you point out), I would not be surprised if that literature is less reliable than most people believe. I know from my own experience that even highly cited papers with theoretical models can have surprisingly obvious major problems. Sometimes no empirical data is needed to know that something isn't right. The exact same reasons one might be suspicious of a pure math result can apply here.
- atq2119 7y ago
  Having worked in discrete math and theoretical computer science myself, I agree with you especially when it comes to less well known / niche statements. Results that many other people build on tend to be solid though, because building on those results often requires the users to at least partially "reprove" them.

salty_biscuits 7y ago

"If this were true, that would mean all previous researchers were wrong." It's poor phrasing, but in a review I would interpret this statement as "I made a strong claim, and my discussion wasn't strong enough to address the concerns of a reviewer who has only interacted with my work via this document." not as "you must be wrong because everyone else says something different". Reviews are a weird social process though.

btrettel 7y ago
To be honest, I'm not sure what the reviewer is arguing, but your suggestion seems reasonable. Perhaps it's bad English. On a reread I get the impression that they were trying to say that I was being rude. Maybe I've misinterpreted them.
- MereInterest 7y ago
  To add to that, it is better to explicitly point out disagreements with previous experiments. That shows that you have read the literature, seen that it is in disagreement, and are willing to stand by your measurement. Not pointing it out comes across as there being something to hide, and that the disagreement in the measurements is being snuck past the reviewers. Overall, you are absolutely correct that reviews are a really weird social context.
  - heavenlyblue 7y ago
    Aren’t you supposed to review the work, rather than the implications of said work?
    - atq2119 7y ago
      The two go hand in hand. After all, one way of proving that a statement is false is to prove that it leads to a contradiction.
    - setr 7y ago
      Yes, but it's so likely that your work is somehow wrong (in that it was misimplemented), that hueristics are employed to fail it early. If you can't easily point out why your work comes to a different conclusion than prior work, then it's likely you've failed to actually look into the prior work effectively. And it's very unlikely you've managed to upend a body of work you didn't even know existed, and has probably run into several errors before (which you probably didn't avoid, because you didn't look into it). If you did read through the previous literature, it shouldn't be difficult to specify and compare. It's possible for you to come from no background and upend everything.. but the chances are simply (and largely) in favor of a mistake being made. The onus is really on you to double-check and triple check your radical conclusions