8 ms·

It is not a good way of doing science, but it is the best we have. All the alternatives, including the ones you proposed, have their own serious downsides, whi

by n144q 2y ago

It is not a good way of doing science, but it is the best we have. All the alternatives, including the ones you proposed, have their own serious downsides, which is why we kept the status quo for the past few decades.

fastball 2y ago
What is the serious downside of open internet centric review?
- reilly3000 2y ago
  The open internet. i.e. trolls, brigades, spammers, bots, and all manner of uninformed voices.
  - bruce511 2y ago
    To expand on this - because if the barrier to publishing is zero, then the "reputation" of the publisher is also zero. (Actually, we already have the "open publishing" you are suggesting - it's called Blogging or social media.) In other words, if we have open publishing, then someone like me (with zero understanding of a topic) can publish a very authentic-looking pile of nonsense with exactly the same weight as someone who, you know, has actually done some science and knows what they're talking about. The common "solution" to this is voting - like with StackOverflow answers. But that is clearly trivial to game and would quickly become meaningless. So human review it is - combined with the reputation that a journal brings. The author gains reputation because some reviewers (with reputation) reviewed the paper, and the journal (with reputation) accepted it. Yes, this system is cumbersome, prone to failure, and subject to outside influences. It's not perfect. Just the best we have right now.
    - eru 2y ago
      > To expand on this - because if the barrier to publishing is zero, then the "reputation" of the publisher is also zero. That's fine. I don't read eg Astral Codex Ten because I think the reputation of Substack is great. The blog can stand entirely on its own reputation (and the reputation of its author), no need for the publisher to rent out their reputation. See also Gwern.net for a similar example. No need for any voting.
    - ricksunny 2y ago
      Reviewers could themselves have reputation levels that weight how visible their review is. This would make brigading more costly. There might still be a pseudoscientific brigade trying to take down (or boost) a particular paper, one that clusters so much that it builds its own competing reputatation, but that's okay. The casual reader can decide which high-vote reviews to follow on their own merits.
- daemontus 2y ago
  As others have mentioned, the main problem is that open systems are more vulnerable to low-cost, coordinated external attacks. This is less of an issue with systems where there is little monetary value attached (I don't know anyone whose mortgage is paid for by their Stack Overflow reputation). Now imagine that the future prospects of a national lab with multi-million yearly budget are tied to a system that can be (relatively easily) gamed with a Chinese or Russian bot farm for a few thousand dollars. There are already players that are trying hard to game the current system, and it sometimes sort of works, but not quite, exactly because of how hard it is to get into the "high reputation" club (on the other hand, once you're in, you can often publish a lot of lower quality stuff just because of your reputation, so I'm not saying this is a perfect system either). In other words, I don't think anyone reasonable is seriously against making peer review more transparent, but for better or worse, the current system (with all of its other downsides) is relatively robust to outside interference. So, unless we (a) make "being a scientist" much more financially accessible, or (b), untangle funding from this new "open" measure of "scientific achievement", the open system would probably not be very impactful. Of course, (a) is unlikely, at least in most high-impact fields; CS was an outlier for a long time, not so much today. And (b) would mean that funding agencies would still need something else to judge your research, which would most likely still be some closed, reputation-based system. Edit TL;DR: Describe how the open science peer-review system should be used to distribute funding among researchers while begin reasonably robust to coordinated attacks. Then we can talk :)
- Al-Khwarizmi 2y ago
  If by "open" you mean that the paper is there and people just voluntarily choose to review it, rather than having some top-down coordinated assignment process, the problem is that papers by the superstars would get hundreds of reviews while papers from unknown labs would get zero. You could of course make it double blind, but that seems hard to enforce in practice in such an open setup, and still, hyped papers in fashionable topics would get many reviews while papers that are hardcore theoretical, in an underdog domain, etc. would get zero. Finally, it also becomes much more difficult to handle conflicts of interest, and the system is highly vulnerable to reviewer collusion.
Panoramix 2y ago
We kept that mostly due to inertia and because it's the most profitable for the journals (everybody does their work for free and they don't have to invest in new systems), not because it's the best for science and scientists.
eru 2y ago
> It is not a good way of doing science, but it is the best we have. What makes you think so? We already have and had plenty of other ways. Eg you can see how science is done in corporations or for the military or for fun (see those old gentlemen scientists, or amateurs these days), and you can also just publish things on your own these days. The only real function of these old fashioned journals is as gatekeepers for funding and career decisions.
- n144q 2y ago
  I heard first hand accounts from multiple people of running into a different set of problems (from academia) publishing papers in corporations. Publishing is never simple or easy. If you have concrete examples, or better, generally recognized studies that show there is an objectively better way to do research, I'd very like to know that. Because, as an PhD who knows dozens of other PhDs in both academia and industry, and who has never heard of this magic new approach to doing science, it would be quite a surprise.
  - bumby 2y ago
    I think the distinction in the examples given (corporations, military), science is being done but much less open.
  - eru 2y ago
    I wasn't talking about publishing, I was talking about doing science. Publishing can be one part of doing science, but it's not the end-and-be-all. And yes, I have no idea how great corporate research or military research etc are, I just brought them up as examples of research outside of academia that we can look to and perhaps learn something from. (And I also strongly suspect research at TSMC will be very different from research at Johnson & Johnson and that's very different from how Jane Street does research. So not all corporate research is the same.) > Because, as an PhD who knows dozens of other PhDs in both academia and industry, and who has never heard of this magic new approach to doing science, it would be quite a surprise. And why would you expect your PhD friends to hear from that? PhD's are very much in academia, and very much embedded in academia's publish-or-perish.
eeeeeeehio 2y ago
Peer review is not designed for science. Many papers are not rejected because of an issue with the science -- in fact, reviewers seldom have the time to actually check the science! As a CS-centric example: you'll almost never find a reviewer who reads a single line of code (if code is submitted with the paper at all). There is artifact review, but this is never tied to the acceptance of the paper. Reviewers focus on ideas, presentation, and the presented results. (And the current system is a good filter for this! Most accepted papers are well-written and the results always look good on paper.) However, reviewers never take the time to actually verify that the experiment code matches the ideas described in the paper, and that the results reproduce. Ask any CS/engineering PhD student how many papers (in top venues) they've seen with a critical implementation flaw that invalidates the results -- and you might begin to understand the problem. At least in CS, the system can be fixed, but those in power are unable and unwilling to fix it. Authors don't want to be held accountable ("if we submit the code with the paper -- someone might find a critical bug and reject the paper!"), and reviewers are both unqualified (i.e. haven't written a line of code in 25 years) and unwilling to take on more responsibility ("I don't have the time to make sure their experiment code is fair!"). So we are left with an obviously broken system where junior PhD students review artifacts for "reproducibility" and this evaluation has no bearing whatsoever on whether a paper gets accepted. It's too easy to cook up positive results in almost any field (intentionally, or unintentionally), and we have a system with little accountability. It's not "the best we have", it's "the best those in power will allow". Those in power do not want consequences for publishing bad research, and also don't want the reviewing load required to keep bad research out.
- DiogenesKynikos 2y ago
  > It's not "the best we have", it's "the best those in power will allow". Those in power do not want consequences for publishing bad research, and also don't want the reviewing load required to keep bad research out. This is a very conspiratorial view of things. The simple and true answer is your last suggestion: doing a more thorough review takes more time than anyone has available. Reviewers work for free. Applying the level of scrutiny you're requesting would require far more work than reviewers currently do, and maybe even something approaching the amount of work required to write the paper in the first place. The more work it takes to review an article, the less willing reviewers are to volunteer their time, and the harder it is for editors to find reviewers. The current level of scrutiny that papers get at the peer-review stage is a result of how much time reviewers can realistically volunteer. Peer review is a very low standard. It's only an initial filter to remove the garbage and to bring papers up to some basic quality standard. The real test of a paper is whether it is cited and built upon by other scientists after publication. Many papers are published and then forgotten, or found to be flawed and not used any more.
tuyiown 2y ago
> It is not a good way of doing science, but it is the best we have. It may have been for some time, but there is human social dynamics in play.
psychoslave 2y ago
So the lesson is there is not a single good way to do science (or anything really), as whatever the approach retained, there will be human biases involved. So the less brittle option obviously might be to go through all possible approaches, but this is obviously more resources demanding, plus we still have the issue of creating some synthesis of all the accumulated insights from various approaches which itself might be taken into various approaches. That’s more of a indefinitely deep spiral, under that perspective An other perspective is to consider, what are the expected outcomes of the stakeholders maybe. A shiny academic career? An attempt to bring some enlightenment on deep cognitive patterns to the luckiest follows that have the resources at end to follow your high level intellectual gymnastic? A pursuit of ways to improve humanity condition through relevant and sound knowledge bodies? There are definitely many others.

by n144q 2y ago

fastball 2y ago

What is the serious downside of open internet centric review?

reilly3000 2y ago
The open internet. i.e. trolls, brigades, spammers, bots, and all manner of uninformed voices.
- bruce511 2y ago
  To expand on this - because if the barrier to publishing is zero, then the "reputation" of the publisher is also zero. (Actually, we already have the "open publishing" you are suggesting - it's called Blogging or social media.) In other words, if we have open publishing, then someone like me (with zero understanding of a topic) can publish a very authentic-looking pile of nonsense with exactly the same weight as someone who, you know, has actually done some science and knows what they're talking about. The common "solution" to this is voting - like with StackOverflow answers. But that is clearly trivial to game and would quickly become meaningless. So human review it is - combined with the reputation that a journal brings. The author gains reputation because some reviewers (with reputation) reviewed the paper, and the journal (with reputation) accepted it. Yes, this system is cumbersome, prone to failure, and subject to outside influences. It's not perfect. Just the best we have right now.
  - eru 2y ago
    > To expand on this - because if the barrier to publishing is zero, then the "reputation" of the publisher is also zero. That's fine. I don't read eg Astral Codex Ten because I think the reputation of Substack is great. The blog can stand entirely on its own reputation (and the reputation of its author), no need for the publisher to rent out their reputation. See also Gwern.net for a similar example. No need for any voting.
  - ricksunny 2y ago
    Reviewers could themselves have reputation levels that weight how visible their review is. This would make brigading more costly. There might still be a pseudoscientific brigade trying to take down (or boost) a particular paper, one that clusters so much that it builds its own competing reputatation, but that's okay. The casual reader can decide which high-vote reviews to follow on their own merits.
daemontus 2y ago
As others have mentioned, the main problem is that open systems are more vulnerable to low-cost, coordinated external attacks. This is less of an issue with systems where there is little monetary value attached (I don't know anyone whose mortgage is paid for by their Stack Overflow reputation). Now imagine that the future prospects of a national lab with multi-million yearly budget are tied to a system that can be (relatively easily) gamed with a Chinese or Russian bot farm for a few thousand dollars. There are already players that are trying hard to game the current system, and it sometimes sort of works, but not quite, exactly because of how hard it is to get into the "high reputation" club (on the other hand, once you're in, you can often publish a lot of lower quality stuff just because of your reputation, so I'm not saying this is a perfect system either). In other words, I don't think anyone reasonable is seriously against making peer review more transparent, but for better or worse, the current system (with all of its other downsides) is relatively robust to outside interference. So, unless we (a) make "being a scientist" much more financially accessible, or (b), untangle funding from this new "open" measure of "scientific achievement", the open system would probably not be very impactful. Of course, (a) is unlikely, at least in most high-impact fields; CS was an outlier for a long time, not so much today. And (b) would mean that funding agencies would still need something else to judge your research, which would most likely still be some closed, reputation-based system. Edit TL;DR: Describe how the open science peer-review system should be used to distribute funding among researchers while begin reasonably robust to coordinated attacks. Then we can talk :)
Al-Khwarizmi 2y ago
If by "open" you mean that the paper is there and people just voluntarily choose to review it, rather than having some top-down coordinated assignment process, the problem is that papers by the superstars would get hundreds of reviews while papers from unknown labs would get zero. You could of course make it double blind, but that seems hard to enforce in practice in such an open setup, and still, hyped papers in fashionable topics would get many reviews while papers that are hardcore theoretical, in an underdog domain, etc. would get zero. Finally, it also becomes much more difficult to handle conflicts of interest, and the system is highly vulnerable to reviewer collusion.

Panoramix 2y ago

We kept that mostly due to inertia and because it's the most profitable for the journals (everybody does their work for free and they don't have to invest in new systems), not because it's the best for science and scientists.

eru 2y ago

> It is not a good way of doing science, but it is the best we have. What makes you think so? We already have and had plenty of other ways. Eg you can see how science is done in corporations or for the military or for fun (see those old gentlemen scientists, or amateurs these days), and you can also just publish things on your own these days. The only real function of these old fashioned journals is as gatekeepers for funding and career decisions.

n144q 2y ago
I heard first hand accounts from multiple people of running into a different set of problems (from academia) publishing papers in corporations. Publishing is never simple or easy. If you have concrete examples, or better, generally recognized studies that show there is an objectively better way to do research, I'd very like to know that. Because, as an PhD who knows dozens of other PhDs in both academia and industry, and who has never heard of this magic new approach to doing science, it would be quite a surprise.
- bumby 2y ago
  I think the distinction in the examples given (corporations, military), science is being done but much less open.
- eru 2y ago
  I wasn't talking about publishing, I was talking about doing science. Publishing can be one part of doing science, but it's not the end-and-be-all. And yes, I have no idea how great corporate research or military research etc are, I just brought them up as examples of research outside of academia that we can look to and perhaps learn something from. (And I also strongly suspect research at TSMC will be very different from research at Johnson & Johnson and that's very different from how Jane Street does research. So not all corporate research is the same.) > Because, as an PhD who knows dozens of other PhDs in both academia and industry, and who has never heard of this magic new approach to doing science, it would be quite a surprise. And why would you expect your PhD friends to hear from that? PhD's are very much in academia, and very much embedded in academia's publish-or-perish.

eeeeeeehio 2y ago

Peer review is not designed for science. Many papers are not rejected because of an issue with the science -- in fact, reviewers seldom have the time to actually check the science! As a CS-centric example: you'll almost never find a reviewer who reads a single line of code (if code is submitted with the paper at all). There is artifact review, but this is never tied to the acceptance of the paper. Reviewers focus on ideas, presentation, and the presented results. (And the current system is a good filter for this! Most accepted papers are well-written and the results always look good on paper.) However, reviewers never take the time to actually verify that the experiment code matches the ideas described in the paper, and that the results reproduce. Ask any CS/engineering PhD student how many papers (in top venues) they've seen with a critical implementation flaw that invalidates the results -- and you might begin to understand the problem. At least in CS, the system can be fixed, but those in power are unable and unwilling to fix it. Authors don't want to be held accountable ("if we submit the code with the paper -- someone might find a critical bug and reject the paper!"), and reviewers are both unqualified (i.e. haven't written a line of code in 25 years) and unwilling to take on more responsibility ("I don't have the time to make sure their experiment code is fair!"). So we are left with an obviously broken system where junior PhD students review artifacts for "reproducibility" and this evaluation has no bearing whatsoever on whether a paper gets accepted. It's too easy to cook up positive results in almost any field (intentionally, or unintentionally), and we have a system with little accountability. It's not "the best we have", it's "the best those in power will allow". Those in power do not want consequences for publishing bad research, and also don't want the reviewing load required to keep bad research out.

DiogenesKynikos 2y ago
> It's not "the best we have", it's "the best those in power will allow". Those in power do not want consequences for publishing bad research, and also don't want the reviewing load required to keep bad research out. This is a very conspiratorial view of things. The simple and true answer is your last suggestion: doing a more thorough review takes more time than anyone has available. Reviewers work for free. Applying the level of scrutiny you're requesting would require far more work than reviewers currently do, and maybe even something approaching the amount of work required to write the paper in the first place. The more work it takes to review an article, the less willing reviewers are to volunteer their time, and the harder it is for editors to find reviewers. The current level of scrutiny that papers get at the peer-review stage is a result of how much time reviewers can realistically volunteer. Peer review is a very low standard. It's only an initial filter to remove the garbage and to bring papers up to some basic quality standard. The real test of a paper is whether it is cited and built upon by other scientists after publication. Many papers are published and then forgotten, or found to be flawed and not used any more.

tuyiown 2y ago

> It is not a good way of doing science, but it is the best we have. It may have been for some time, but there is human social dynamics in play.

psychoslave 2y ago

So the lesson is there is not a single good way to do science (or anything really), as whatever the approach retained, there will be human biases involved. So the less brittle option obviously might be to go through all possible approaches, but this is obviously more resources demanding, plus we still have the issue of creating some synthesis of all the accumulated insights from various approaches which itself might be taken into various approaches. That’s more of a indefinitely deep spiral, under that perspective An other perspective is to consider, what are the expected outcomes of the stakeholders maybe. A shiny academic career? An attempt to bring some enlightenment on deep cognitive patterns to the luckiest follows that have the resources at end to follow your high level intellectual gymnastic? A pursuit of ways to improve humanity condition through relevant and sound knowledge bodies? There are definitely many others.