6 ms·

I think this is good, and may also improve reviews. I doubt that Nature has an issue with substandard reviews, but I have definitely seen it in some of the comp

I think this is good, and may also improve reviews. I doubt that Nature has an issue with substandard reviews, but I have definitely seen it in some of the computer science conferences I submit to and have served on [1]. If reviewers know that their review may be published, I think they may turn in more substantial reviews. [1] In computer science, conference publications are peer-reviewed and highly competitive. We also have journals, but most novel work appears in conferences. The more theory oriented parts of computer science will tend to publish in journals more, but they still publish in conferences, too.

nextos 7y ago
Indeed. In my experience and other researchers I know, often rival groups will try to block your research from getting published in Nature or Science by asking for unreasonable follow-up experiments. It's sadly very political. Hopefully this will improve things. I personally know the current Nature chief editor and she totally agrees with my points above. In contrast, I have always found math & CS journal reviews way more fair and rigorous.
- BurningFrog 7y ago
  > It's sadly very political This is the predictable consequence of pre publication review by competitors. The solution is not to hope for better reviewers. It's to abolish the failed peer review system.
  - leereeves 7y ago
    > It's to abolish the failed peer review system. And replace with...?
    - jsmith45 7y ago
      In many ways the real peer review comes after publication, when more people have seen the work, and can react to it. In some fields it is not terribly rare for flaws in the work get discovered at that point, and rebuttals get written and circulated (or possibly even published in the journal.) The prepublication peer review is not always a terribly high bar to pass. It does help to weed out some of the crap, but lots still get published, especially in less competitive journals.
    - chr1 7y ago
      Publish your article with accompanying data and programs on sci-hub (github for science). Tell your colleagues about it if you normally hold seminars before publishing papers, or post to mailing list. As interested people read your paper they will post comments, open issues, or star your repository. If it gets many stars Nature will include a link to it in their monthly list of important works, which would help even more people to find your repository.
      - cozzyd 7y ago
        So that Nature will be full of papers from researchers who hire the most clickfarmers to star their work?
        chr1 7y ago
        Nature would not assign the same value to every star. Star from someone who works in a university and has large impact factor in the related field will have much higher value than a star from newly created account without any published work.
        rusticpenn 7y ago
        That is then a circular problem. Most people who have high impact right now, know how to play the publishing game.
      - meej 7y ago
        > Publish your article with accompanying data and programs on sci-hub (github for science). How does one publish on Sci-Hub? I would suggest sites like https://zenodo.org/ https://zenodo.org/ or https://osf.io/ https://osf.io/ for self-publishing research products. Or one's own institution's repository, if they have one (many do, see http://v2.sherpa.ac.uk/opendoar/ http://v2.sherpa.ac.uk/opendoar/).
        matwood 7y ago
        Publish the paper and put the protocols on something like https://www.protocols.io https://www.protocols.io so the results can be reproduced.
    - BurningFrog 7y ago
      The way things were done before the current system took over (in the 1980s, I believe). Not sure what it was exactly. I heard rumors that the journal often did its own review. Either way, if it was good enough for Einstein and Feynman, it might work for others. The important thing is that the people whose careers might be upended by revolutionary findings can't stop or steal the ideas.
      - Cyph0n 7y ago
        So you want to replace the current peer review system with one you don’t really understand? Have you tried looking into why the system was changed in the first place? The fact that it worked for Einstein doesn’t make it a better system. Do you think Einstein wouldn’t have gotten his revolutionary work published with the current system?
        petschge 7y ago
        Btw: The old system was the editor reading and reviewing all submissions himself. The new system of only a quick glance by the editor and then further scrutiny by a peer was instituted for two reasons: 1.) to many submissions for the editor to read all and 2.) increasing specialization. So going back to the old system would either lead to very cursory and potentially incompetent reviews or the increase in the number of editors to a point where all current senior reviewers are now editors. The former is completely intractable and the latter is still worse then the current status quo, especially since it removes the practice of having two or more reviews that specialize at different aspects (e.g. having on reviewer focus on the numerical side of the simulation and another on the statistical side of the comparison with observation data).
        BurningFrog 7y ago
        > So going back to the old system would either lead to very cursory and potentially incompetent reviews... Yeah, this is side I'd pick. I'd rather have many mediocre submissions published fast than the current system. Let the "real" review happens after after the results are public. I see how reasonable people can pick the other side.
    - JMTQp8lwXL 7y ago
      The only reason the review process is politicized is because academic institutions compete for research money. Preventing your peer institutions from publishing helps you get money for your own. You need to change how research is funded. Follow the money.
      - BurningFrog 7y ago
        It's not the only reason. There is also the glory of being the genius leader in your field, and all the prizes, titles and lovers that come with it. > Preventing your peer institutions from publishing helps you get money for your own. You need to change how research is funded. Surely removing people's opportunity to veto their competitors publications is a more direct way!
      - thaumasiotes 7y ago
        > The only reason the review process is politicized is because academic institutions compete for research money. Well, publication is also how researchers get promoted by people who can't evaluate their research. There's a within-department need outside of just attracting grants.
      - slavik81 7y ago
        Politics have been part of the peer review process for a very, very long time. It's not all about the money, either: > Fourier was interested in heat propagation, and presented a paper in 1807 to the Institut de France on the use of sinusoids to represent temperature distributions. The paper contained the controversial claim that any continuous periodic signal could be represented as the sum of properly chosen sinusoidal waves. Among the reviewers were two of history's most famous mathematicians, Joseph Louis Lagrange (1736-1813), and Pierre Simon de Laplace (1749-1827). > While Laplace and the other reviewers voted to publish the paper, Lagrange adamantly protested. For nearly 50 years, Lagrange had insisted that such an approach could not be used to represent signals with corners, i.e., discontinuous slopes, such as in square waves. The Institut de France bowed to the prestige of Lagrange, and rejected Fourier's work. It was only after Lagrange died that the paper was finally published, some 15 years later. https://www.dspguide.com/ch8/1.htm https://www.dspguide.com/ch8/1.htm
- t_serpico 7y ago
  Jesus. It's unbelievable that people can be so petty. This is why if I see an academic with tons of high profile publications, it generally signals that they merely know how to play the publication game (and they of course have the desire to do so). This behavior is a red flag imo if it occurs post-tenure.
  - brazzy 7y ago
    "Petty" isn't really the right word, because it's not about inconsequential rivalries, it's about real money and careers.
    - acollins1331 7y ago
      Yes it is. It's petty to cheat to elevate oneself above ones position, or knock someone else down below theirs. Even if it's about money. Especially when it's about money.
biomcgary 7y ago
Published reviews will just make reviews political in new ways. As a graduate student, I helped my advisor review a paper submitted to Nature that was publishing a large biological data set. The internal metrics that the authors provided made it clear that the data was 95% noise. The review was direct and to the point. The paper was rightfully rejected, but ended up in another high profile journal after the authors removed the damning evidence. The corresponding author was one of the most famous and powerful people in a very large field. My advisor was still a relatively junior professor at the time, even if well known. I don't think publication of the review would have changed things for him, but for many others I have known, it might have.
- monadic2 7y ago
  Surely this was happening already without published reviews...?
  - biomcgary 7y ago
    Yes, but criticizing someone's work in a single-blind and private review leaves less motivation and opportunity for the powerful to be vindictive.
    - gindely 7y ago
      But outside of physics, there's many fields of science where double and even single blind is meaningless. If you get access to the review directly, you can tell by what they say and how they say it who they are. If you're a vindictive powerful person, who cares that you might be wrong?
    - bakuninsbart 7y ago
      Just have to publish reviews anonymously, and if there is outcry about it journals can have internal investigations on the matter.
- beerandt 7y ago
  I'd say this is actually a stronger argument for publishing reviews, as a persuasive argument might prevent venue shopping, potentially in multiple ways, depending on the when.
cozzyd 7y ago
At the very least, people will be more likely to decline to be a referee for a paper they're less of an expert on or don't have time for if they knew the report would be public.
- bumby 7y ago
  After seeing some review comments that make it clear the reviewer either had little knowledge of the field or comments that could have been rebutted by cursory view of the data, this seems like a an area that sorely needs improvement.
  - mr__y 7y ago
    >area that sorely needs improvement. the simplest idea would be to improve peer-reviewing with moving it to something similiar to hacker-news discussion board (but possibly not open to everyone), where reviewers would comment, authors could respond to questions or clarify something.
    - bumby 7y ago
      I’m not convinced that would work. If the problem is the quality of the review it seems like the reviewers need more vetting, not less. I could see the HN version devolving into a lot of wasted effort responding. It would be even worse if it was allowed to be anonymous. If it wasn’t anonymous and the reviewers were vetted with respect to the field they were commenting on, it might work
ThePhysicist 7y ago
In my (very limited) experience there are still large differences in the quality of the reviews. As researchers do this basically on the side and for free, the amount of time they devote to a review varies greatly depending on their time budget and motivation. While some reviewers seem to go through the paper character by character and point out even the tiniest flaws, others are contend with a superficial reading and miss even glaring mistakes. I'd say publishing the reviews is good because it allows you to see how much effort went into the review process. It will also (anonymously) shame the second type of reviewer into being more thorough, which I think is also good. As a side note, I think articles should all be available for free and the work of the reviewers should be compensated or appreciated somehow. Publishing reviews with the reviewers names (only for people that want it) can help to create such appreciation, because it is a lot of work.
Fishysoup 7y ago
They'll only publish reviews for already accepted papers. This may help reviewers be (or at least appear) less obviously biased against the authors in the review process if the paper has a chance of being accepted. But it won't do anything for papers that 2 or more reviewers want to shut down. Note I'm not talking about low quality research but about papers that are shut down for political reasons (happens often enough in neuroscience at least) so that the reviewers can then do the same experiments and publish before you.
- dhimes 7y ago
  I don't understand how they can get away with this blatant idea-stealing. Isn't there something in the grant process that can punish them for misbehavior? If one team has a laboratory notebook showing they did the work, it seems like a paper trail to the fraudster could easily be constructed.

nextos 7y ago

Indeed. In my experience and other researchers I know, often rival groups will try to block your research from getting published in Nature or Science by asking for unreasonable follow-up experiments. It's sadly very political. Hopefully this will improve things. I personally know the current Nature chief editor and she totally agrees with my points above. In contrast, I have always found math & CS journal reviews way more fair and rigorous.

BurningFrog 7y ago
> It's sadly very political This is the predictable consequence of pre publication review by competitors. The solution is not to hope for better reviewers. It's to abolish the failed peer review system.
- leereeves 7y ago
  > It's to abolish the failed peer review system. And replace with...?
  - jsmith45 7y ago
    In many ways the real peer review comes after publication, when more people have seen the work, and can react to it. In some fields it is not terribly rare for flaws in the work get discovered at that point, and rebuttals get written and circulated (or possibly even published in the journal.) The prepublication peer review is not always a terribly high bar to pass. It does help to weed out some of the crap, but lots still get published, especially in less competitive journals.
  - chr1 7y ago
    Publish your article with accompanying data and programs on sci-hub (github for science). Tell your colleagues about it if you normally hold seminars before publishing papers, or post to mailing list. As interested people read your paper they will post comments, open issues, or star your repository. If it gets many stars Nature will include a link to it in their monthly list of important works, which would help even more people to find your repository.
    - cozzyd 7y ago
      So that Nature will be full of papers from researchers who hire the most clickfarmers to star their work?
      - chr1 7y ago
        Nature would not assign the same value to every star. Star from someone who works in a university and has large impact factor in the related field will have much higher value than a star from newly created account without any published work.
        rusticpenn 7y ago
        That is then a circular problem. Most people who have high impact right now, know how to play the publishing game.
    - meej 7y ago
      > Publish your article with accompanying data and programs on sci-hub (github for science). How does one publish on Sci-Hub? I would suggest sites like https://zenodo.org/ https://zenodo.org/ or https://osf.io/ https://osf.io/ for self-publishing research products. Or one's own institution's repository, if they have one (many do, see http://v2.sherpa.ac.uk/opendoar/ http://v2.sherpa.ac.uk/opendoar/).
      - matwood 7y ago
        Publish the paper and put the protocols on something like https://www.protocols.io https://www.protocols.io so the results can be reproduced.
  - BurningFrog 7y ago
    The way things were done before the current system took over (in the 1980s, I believe). Not sure what it was exactly. I heard rumors that the journal often did its own review. Either way, if it was good enough for Einstein and Feynman, it might work for others. The important thing is that the people whose careers might be upended by revolutionary findings can't stop or steal the ideas.
    - Cyph0n 7y ago
      So you want to replace the current peer review system with one you don’t really understand? Have you tried looking into why the system was changed in the first place? The fact that it worked for Einstein doesn’t make it a better system. Do you think Einstein wouldn’t have gotten his revolutionary work published with the current system?
      - petschge 7y ago
        Btw: The old system was the editor reading and reviewing all submissions himself. The new system of only a quick glance by the editor and then further scrutiny by a peer was instituted for two reasons: 1.) to many submissions for the editor to read all and 2.) increasing specialization. So going back to the old system would either lead to very cursory and potentially incompetent reviews or the increase in the number of editors to a point where all current senior reviewers are now editors. The former is completely intractable and the latter is still worse then the current status quo, especially since it removes the practice of having two or more reviews that specialize at different aspects (e.g. having on reviewer focus on the numerical side of the simulation and another on the statistical side of the comparison with observation data).
        BurningFrog 7y ago
        > So going back to the old system would either lead to very cursory and potentially incompetent reviews... Yeah, this is side I'd pick. I'd rather have many mediocre submissions published fast than the current system. Let the "real" review happens after after the results are public. I see how reasonable people can pick the other side.
  - JMTQp8lwXL 7y ago
    The only reason the review process is politicized is because academic institutions compete for research money. Preventing your peer institutions from publishing helps you get money for your own. You need to change how research is funded. Follow the money.
    - BurningFrog 7y ago
      It's not the only reason. There is also the glory of being the genius leader in your field, and all the prizes, titles and lovers that come with it. > Preventing your peer institutions from publishing helps you get money for your own. You need to change how research is funded. Surely removing people's opportunity to veto their competitors publications is a more direct way!
    - thaumasiotes 7y ago
      > The only reason the review process is politicized is because academic institutions compete for research money. Well, publication is also how researchers get promoted by people who can't evaluate their research. There's a within-department need outside of just attracting grants.
    - slavik81 7y ago
      Politics have been part of the peer review process for a very, very long time. It's not all about the money, either: > Fourier was interested in heat propagation, and presented a paper in 1807 to the Institut de France on the use of sinusoids to represent temperature distributions. The paper contained the controversial claim that any continuous periodic signal could be represented as the sum of properly chosen sinusoidal waves. Among the reviewers were two of history's most famous mathematicians, Joseph Louis Lagrange (1736-1813), and Pierre Simon de Laplace (1749-1827). > While Laplace and the other reviewers voted to publish the paper, Lagrange adamantly protested. For nearly 50 years, Lagrange had insisted that such an approach could not be used to represent signals with corners, i.e., discontinuous slopes, such as in square waves. The Institut de France bowed to the prestige of Lagrange, and rejected Fourier's work. It was only after Lagrange died that the paper was finally published, some 15 years later. https://www.dspguide.com/ch8/1.htm https://www.dspguide.com/ch8/1.htm
t_serpico 7y ago
Jesus. It's unbelievable that people can be so petty. This is why if I see an academic with tons of high profile publications, it generally signals that they merely know how to play the publication game (and they of course have the desire to do so). This behavior is a red flag imo if it occurs post-tenure.
- brazzy 7y ago
  "Petty" isn't really the right word, because it's not about inconsequential rivalries, it's about real money and careers.
  - acollins1331 7y ago
    Yes it is. It's petty to cheat to elevate oneself above ones position, or knock someone else down below theirs. Even if it's about money. Especially when it's about money.

biomcgary 7y ago

Published reviews will just make reviews political in new ways. As a graduate student, I helped my advisor review a paper submitted to Nature that was publishing a large biological data set. The internal metrics that the authors provided made it clear that the data was 95% noise. The review was direct and to the point. The paper was rightfully rejected, but ended up in another high profile journal after the authors removed the damning evidence. The corresponding author was one of the most famous and powerful people in a very large field. My advisor was still a relatively junior professor at the time, even if well known. I don't think publication of the review would have changed things for him, but for many others I have known, it might have.

monadic2 7y ago
Surely this was happening already without published reviews...?
- biomcgary 7y ago
  Yes, but criticizing someone's work in a single-blind and private review leaves less motivation and opportunity for the powerful to be vindictive.
  - gindely 7y ago
    But outside of physics, there's many fields of science where double and even single blind is meaningless. If you get access to the review directly, you can tell by what they say and how they say it who they are. If you're a vindictive powerful person, who cares that you might be wrong?
  - bakuninsbart 7y ago
    Just have to publish reviews anonymously, and if there is outcry about it journals can have internal investigations on the matter.
beerandt 7y ago
I'd say this is actually a stronger argument for publishing reviews, as a persuasive argument might prevent venue shopping, potentially in multiple ways, depending on the when.

cozzyd 7y ago

At the very least, people will be more likely to decline to be a referee for a paper they're less of an expert on or don't have time for if they knew the report would be public.

bumby 7y ago
After seeing some review comments that make it clear the reviewer either had little knowledge of the field or comments that could have been rebutted by cursory view of the data, this seems like a an area that sorely needs improvement.
- mr__y 7y ago
  >area that sorely needs improvement. the simplest idea would be to improve peer-reviewing with moving it to something similiar to hacker-news discussion board (but possibly not open to everyone), where reviewers would comment, authors could respond to questions or clarify something.
  - bumby 7y ago
    I’m not convinced that would work. If the problem is the quality of the review it seems like the reviewers need more vetting, not less. I could see the HN version devolving into a lot of wasted effort responding. It would be even worse if it was allowed to be anonymous. If it wasn’t anonymous and the reviewers were vetted with respect to the field they were commenting on, it might work

ThePhysicist 7y ago

In my (very limited) experience there are still large differences in the quality of the reviews. As researchers do this basically on the side and for free, the amount of time they devote to a review varies greatly depending on their time budget and motivation. While some reviewers seem to go through the paper character by character and point out even the tiniest flaws, others are contend with a superficial reading and miss even glaring mistakes. I'd say publishing the reviews is good because it allows you to see how much effort went into the review process. It will also (anonymously) shame the second type of reviewer into being more thorough, which I think is also good. As a side note, I think articles should all be available for free and the work of the reviewers should be compensated or appreciated somehow. Publishing reviews with the reviewers names (only for people that want it) can help to create such appreciation, because it is a lot of work.

Fishysoup 7y ago

They'll only publish reviews for already accepted papers. This may help reviewers be (or at least appear) less obviously biased against the authors in the review process if the paper has a chance of being accepted. But it won't do anything for papers that 2 or more reviewers want to shut down. Note I'm not talking about low quality research but about papers that are shut down for political reasons (happens often enough in neuroscience at least) so that the reviewers can then do the same experiments and publish before you.

dhimes 7y ago
I don't understand how they can get away with this blatant idea-stealing. Isn't there something in the grant process that can punish them for misbehavior? If one team has a laboratory notebook showing they did the work, it seems like a paper trail to the fraudster could easily be constructed.