25 ms·

Nature will publish peer review reports as a trial

239 points by aaavl2821 7y ago

scott_s 7y ago
I think this is good, and may also improve reviews. I doubt that Nature has an issue with substandard reviews, but I have definitely seen it in some of the computer science conferences I submit to and have served on [1]. If reviewers know that their review may be published, I think they may turn in more substantial reviews. [1] In computer science, conference publications are peer-reviewed and highly competitive. We also have journals, but most novel work appears in conferences. The more theory oriented parts of computer science will tend to publish in journals more, but they still publish in conferences, too.
- nextos 7y ago
  Indeed. In my experience and other researchers I know, often rival groups will try to block your research from getting published in Nature or Science by asking for unreasonable follow-up experiments. It's sadly very political. Hopefully this will improve things. I personally know the current Nature chief editor and she totally agrees with my points above. In contrast, I have always found math & CS journal reviews way more fair and rigorous.
  - BurningFrog 7y ago
    > It's sadly very political This is the predictable consequence of pre publication review by competitors. The solution is not to hope for better reviewers. It's to abolish the failed peer review system.
    - leereeves 7y ago
      > It's to abolish the failed peer review system. And replace with...?
      - jsmith45 7y ago
        In many ways the real peer review comes after publication, when more people have seen the work, and can react to it. In some fields it is not terribly rare for flaws in the work get discovered at that point, and rebuttals get written and circulated (or possibly even published in the journal.) The prepublication peer review is not always a terribly high bar to pass. It does help to weed out some of the crap, but lots still get published, especially in less competitive journals.
      - chr1 7y ago
        Publish your article with accompanying data and programs on sci-hub (github for science). Tell your colleagues about it if you normally hold seminars before publishing papers, or post to mailing list. As interested people read your paper they will post comments, open issues, or star your repository. If it gets many stars Nature will include a link to it in their monthly list of important works, which would help even more people to find your repository.
        cozzyd 7y ago
        So that Nature will be full of papers from researchers who hire the most clickfarmers to star their work?
        chr1 7y ago
        Nature would not assign the same value to every star. Star from someone who works in a university and has large impact factor in the related field will have much higher value than a star from newly created account without any published work.
        rusticpenn 7y ago
        That is then a circular problem. Most people who have high impact right now, know how to play the publishing game.
        meej 7y ago
        > Publish your article with accompanying data and programs on sci-hub (github for science). How does one publish on Sci-Hub? I would suggest sites like https://zenodo.org/ https://zenodo.org/ or https://osf.io/ https://osf.io/ for self-publishing research products. Or one's own institution's repository, if they have one (many do, see http://v2.sherpa.ac.uk/opendoar/ http://v2.sherpa.ac.uk/opendoar/).
  - t_serpico 7y ago
    Jesus. It's unbelievable that people can be so petty. This is why if I see an academic with tons of high profile publications, it generally signals that they merely know how to play the publication game (and they of course have the desire to do so). This behavior is a red flag imo if it occurs post-tenure.
    - brazzy 7y ago
      "Petty" isn't really the right word, because it's not about inconsequential rivalries, it's about real money and careers.
      - acollins1331 7y ago
        Yes it is. It's petty to cheat to elevate oneself above ones position, or knock someone else down below theirs. Even if it's about money. Especially when it's about money.
- biomcgary 7y ago
  Published reviews will just make reviews political in new ways. As a graduate student, I helped my advisor review a paper submitted to Nature that was publishing a large biological data set. The internal metrics that the authors provided made it clear that the data was 95% noise. The review was direct and to the point. The paper was rightfully rejected, but ended up in another high profile journal after the authors removed the damning evidence. The corresponding author was one of the most famous and powerful people in a very large field. My advisor was still a relatively junior professor at the time, even if well known. I don't think publication of the review would have changed things for him, but for many others I have known, it might have.
  - monadic2 7y ago
    Surely this was happening already without published reviews...?
    - biomcgary 7y ago
      Yes, but criticizing someone's work in a single-blind and private review leaves less motivation and opportunity for the powerful to be vindictive.
      - gindely 7y ago
        But outside of physics, there's many fields of science where double and even single blind is meaningless. If you get access to the review directly, you can tell by what they say and how they say it who they are. If you're a vindictive powerful person, who cares that you might be wrong?
      - bakuninsbart 7y ago
        Just have to publish reviews anonymously, and if there is outcry about it journals can have internal investigations on the matter.
  - beerandt 7y ago
    I'd say this is actually a stronger argument for publishing reviews, as a persuasive argument might prevent venue shopping, potentially in multiple ways, depending on the when.
- cozzyd 7y ago
  At the very least, people will be more likely to decline to be a referee for a paper they're less of an expert on or don't have time for if they knew the report would be public.
  - bumby 7y ago
    After seeing some review comments that make it clear the reviewer either had little knowledge of the field or comments that could have been rebutted by cursory view of the data, this seems like a an area that sorely needs improvement.
    - mr__y 7y ago
      >area that sorely needs improvement. the simplest idea would be to improve peer-reviewing with moving it to something similiar to hacker-news discussion board (but possibly not open to everyone), where reviewers would comment, authors could respond to questions or clarify something.
      - bumby 7y ago
        I’m not convinced that would work. If the problem is the quality of the review it seems like the reviewers need more vetting, not less. I could see the HN version devolving into a lot of wasted effort responding. It would be even worse if it was allowed to be anonymous. If it wasn’t anonymous and the reviewers were vetted with respect to the field they were commenting on, it might work
- ThePhysicist 7y ago
  In my (very limited) experience there are still large differences in the quality of the reviews. As researchers do this basically on the side and for free, the amount of time they devote to a review varies greatly depending on their time budget and motivation. While some reviewers seem to go through the paper character by character and point out even the tiniest flaws, others are contend with a superficial reading and miss even glaring mistakes. I'd say publishing the reviews is good because it allows you to see how much effort went into the review process. It will also (anonymously) shame the second type of reviewer into being more thorough, which I think is also good. As a side note, I think articles should all be available for free and the work of the reviewers should be compensated or appreciated somehow. Publishing reviews with the reviewers names (only for people that want it) can help to create such appreciation, because it is a lot of work.
- Fishysoup 7y ago
  They'll only publish reviews for already accepted papers. This may help reviewers be (or at least appear) less obviously biased against the authors in the review process if the paper has a chance of being accepted. But it won't do anything for papers that 2 or more reviewers want to shut down. Note I'm not talking about low quality research but about papers that are shut down for political reasons (happens often enough in neuroscience at least) so that the reviewers can then do the same experiments and publish before you.
  - dhimes 7y ago
    I don't understand how they can get away with this blatant idea-stealing. Isn't there something in the grant process that can punish them for misbehavior? If one team has a laboratory notebook showing they did the work, it seems like a paper trail to the fraudster could easily be constructed.
adamnemecek 7y ago
The peer review process has reached a point where it's more harmful than helpful. What I want has following properties: * Version control, I can contribute to other people's papers * I can link to a sentence in another paper at some point in time so that when I read another paper, I can go to the location immediately * Open, like arxiv (anyone can publish anything), however arxiv discourages uploading personal or class projects, which is a terrible idea, the best part about GitHub is finding someone's abandoned project that does something * Full-text search I think that most academic publishing startups approach the problem by indexing already published papers. I think that starting clean slate is achievable.
- Ar-Curunir 7y ago
  None of the points you raised are about peer review...
  - chr1 7y ago
    They are not about peer review, if by "peer review" you mean the abstract notion of someone reviewing your work. But peer review is a specific mechanism of getting comments about your work. And unfortunately it is a mechanism that has not been updated after invention of computers, so it is very inefficient at its job. Peer review in 21st century should be as easy and as open, as reviewing pull requests on github.
    - ISL 7y ago
      The comments are secondary. Filtering for error and importance are the primary goals of peer-review.
      - chr1 7y ago
        How does one "Filter for error and importance" without writing comments? I have been peer reviewer multiple times, and always had to write comments either to author pointing out errors and asking for clarifications or to the journal explaining why the paper is hopeless. It was not much different from reviewing pull requests by email. Did you have different experience reviewing papers for scientific journals?
yurlungur 7y ago
I wonder what is the impact of this change. From my limited experience peer reviews are treated as fairly private communications between the reviewers, editor and authors. We would often use it as an important clue on what to change even if we are going to another journal for publication. This makes the content of the reviews rather sensitive, especially in the cases where there is competition for publication on the same subject. There are also the cases where Reviewer X may ask authors to do a very specific analysis/improvement that may somewhat benefit their own future research. It's all par for the course. Another aspect of this that sometimes experienced professors have a fairly good sense on who's giving the review, even though they may be anonymous. Since this policy gives authors the choice of publishing reviews, I hope it does not degrade the quality of reviews and make people uncomfortable for writing "rude" feedback.
- prepend 7y ago
  I don’t publish or review too much, but I do a little. One of the recently frustrating events to me is that I got comments back from a reviewer that made an incorrect factual statement along the lines of “it’s against government policy to share x type of data.” And then discounted large parts of the conclusion as impossible or inappropriate. That was the only substantive comment from the reviewer. We responded back to the editors refuting it and showing the relevant policies and regulations showing how data sharing for x was allowed. The next message back was from editors declining. So further feedback. I would have really liked to learn more about why the reviewer thought what they thought and why what seemed like clear evidence to my co-authors and I wasn’t to the peer reviewer and editors. I’ll never know and I don’t spend many cycles contemplating it, but this new trial in Nature would help me understand the criticism better.
ISL 7y ago
Oh, man. I've been involved in peer-review discussions where the review correspondence is far more voluminous than the article itself. I wonder how the journals will handle the page-count.
- chr1 7y ago
  They should publish it only in electronic version.
  - petschge 7y ago
    That does not solve the problem that more pages take longer to read. Given the pace of science these days keeping up with papers just in your subfield is an actual problem.
    - gpm 7y ago
      Just because more text is published about a work doesn't mean you have to read it...
    - mr__y 7y ago
      Do you actually read whole papers? It's an honest question. I usually scan the introduction, quickly skim through charts/tables and jump directly to conclusions. If I find anything remotely interesting during that process, then and only then I would even consider reading the whole paper.
      - petschge 7y ago
        I probably read about three full papers per week. The rest I skim similar to you.
glofish 7y ago
The current scientific publishing method is pretty much dead. It is absurd, biased, produces terrible results. Promotes inept research, unimaginable waste. Yet dead as it might be most decision-makers and other leeches of science will pretend that it is just fine and alive. They will continue to do so for many years. Decades perhaps. It is like watching Weekend at Bernie's. This attempt of Nature is little more than holding up Bernie, waving his hand. Look, it is moving! It is not dead.
- coughupalung 7y ago
  How about you do something constructive instead and propose a better alternative. You're just "poisoning the well" with empty accusations and ignoring how successful the scientific method has been.
  - glofish 7y ago
    It is only a poisoning if it is not true. Nature is not the problem though, they just happen to be one of the most prominent and selective publishers - many/most? people hate them because they can't get in. IMHO the entire process of scientific review is wrong. Once millions of dollars in research grants depend on a paper being "accepted", once getting a grant is a zero-sum game, the scientific review is irrevocably corrupted.
    - coughupalung 7y ago
      > It is only a poisoning if it is not true. That is literally not the definition of the "poisoning the well" logical fallacy. And if you actually did publish in Nature, congrats. You still didn't offer an alternative. Edit: fixed typo
      - glofish 7y ago
        The main post is not about alternatives though - it is about a new thing that Nature is trying out. To which I posted that the whole thing is rotten from inside I don't think this is the right avenue or thread to talk about alternatives. If you want my opinion, it is the scientific publishing model the is broken, starting right with the structure of a scientific paper. The way research papers are written dates back to a time where your only source of information was the printed paper. Every single scientific publication today is overdone, exaggerated, overly verbose and almost impossible to comprehend. If you make a single, true and noteworthy observation that helps others, you cannot publish that alone. It has to be dressed up. You have to exaggerate, dress it up, bury it deep into "more" content to make it publication-worthy. The inablity to get credit for small but important research findings is the what, in my opinion is killing science from within. The big papers are a thing of a past, but scientists cannot let go, because that's what they were taught as being important.
kissickas 7y ago
> Research communities are unanimous in acknowledging the value of peer review > 82% agreed that standard peer review ensures high-quality work gets published How common is the view that peer review only serves as an institutional gatekeeping mechanism and harms innovation? (as I've heard Eric Weinstein charge on his podcast, The Portal)
- coughupalung 7y ago
  EW has a PhD but I can't seem to find any academic publications under his name. That doesn't outright mean he's wrong but it does mean that he actually has limited experience with the process. I have several and I've reviewed quite a few papers. I would say that peer review is a pretty good filter for garbage. Typically reviewers want to see shiny new ideas so if anything it over emphasizes innovation and discourages incremental improvements. As for gate keeping, I don't think so. Again, reviewers want to be WOW'd with new ideas. In general, I prefer papers from peer reviewed venues over the arXiv. Good work does get put on the arXiv though so I can't ignore it completely.
  - bumby 7y ago
    >Typically reviewers want to see shiny new ideas so if anything it over emphasizes innovation and discourages incremental improvements. I know publications fetishize novelty, but wasn’t there research showing a high percentage of publication is either non-replicable or derivative and subsequently of little value once you remove auto-citations?
mxcrossb 7y ago
We all know that publishers add little value to science, and nature here looks to be striving to add even less than usual. If the peer reviews are not of good enough quality, your editors should be handling that! Don’t rely on the readers to review the peer review. Dumping the peer reviews online is just a way to avoid the responsibility of quality control.
- chaos_emergent 7y ago
  Conversely, it's also a way for the general public to see criticisms and laudations of research that might otherwise be accepted with blind faith. While Nature has its pick of comparatively incontrovertible research, sending a signal to the rest of the industry that peer reviews ought to be published seems like a useful step toward better, more replicable, more honest science.
  - Vinnl 7y ago
    > While Nature has its pick of comparatively incontrovertible research Nature doesn't actually attempt to select for incontrovertible research, but rather the most groundbreaking/noteworthy research. Which is also why their retraction rate (and those of other "top" journals) is higher than average. (Disclosure: I'm part of an initiative that explicitly distinguishes between exciting and robust research for this reason.)
buboard 7y ago
Elife did this experiment long ago (one of their many innovations) https://elifesciences.org/inside-elife/e9091cea/peer-review-new-initiatives-to-enhance-the-value-of-elife-s-process https://elifesciences.org/inside-elife/e9091cea/peer-review-...
mmhsieh 7y ago
there should be a public and a private portion
cosmic_ape 7y ago
Some comments seem to miss the point that reviews will be published without naming the reviewers (unless they want to), and then only for accepted papers. So for the reviewer this should not change things much. Also, NeurIPS conference publishes reviews for a long time now. But I haven't heard of anyone talking about a review of a paper. And if you do look at these reviews, really many of them are of really dismal quality. Maybe things will be different for Nature, idk.
- captain_price7 7y ago
  Do you know any other ML conference/journal, doesn't have to be NeurIPS quality,that also publishes reviews? I know there is ICLR, but anything else? I really like reading them.
  - cosmic_ape 7y ago
    Not that I know of. But its seems a few other conferences use OpenReview, besides ICLR. UAI for example.
esfandia 7y ago
I like the idea in principle, but it raises the bar for the quality of the review that is going to be published, and might thus discourage people from accepting to review papers. It is hard already as it is to find peer reviewers! We're getting to the point where there really has to be a financial incentive to do a peer review (Springer gives you a free book, which is a start). If that were the case, I wouldn't mind doing it occasionally when I retire.
- captain_price7 7y ago
  I think it can be seen as quality vs quantity issue i.e Is 2 good reviews better than 4 ok reviews?
- yqx 7y ago
  > We're getting to the point where there really has to be a financial incentive to do a peer review IMHO, a free book is an insulting and scientists shouldn't accept to do this work for commercial publishers for free to begin with. I know many people feel it's their responsibility to provide "services to the community" which is great, but it's ridiculous that commercial publishers get to profit from this when they proceed to claim ownership of academic work and hide it from the public behind a paywall. If they want to make money off research, fine. But reviewing costs time and money. Academic publishing may well turn out to be not so profitable anymore if reviewers have to be paid. Maybe then we can finally move to university-hosted open-access publishing.
xvilka 7y ago
All research indeed will benefit from such an approach. Think of transitioning from the lack of version control systems to their appearance. But it is not enough for better scientific publishing. There was quite an interesting discussion[1] about creating something in between Overleaf[2], ArXiv[3], Git, and Wikipedia, moreover with the ability to do a peer-to-peer review, discussion, and social networking. See also the last[4] article in that series. There are a few implementations, albeit not covering all features, like Authorea[5] and MIT's PubPub[6] (it is the open source[7]). See also GitXiv[8]. See also the Publishing Reform[9] project. Moreover, there is quite an interesting initiative from DARPA, to create the scientific social network of a kind - Polyplexus[10]. [1] http://blog.jessriedel.com/2015/04/16/beyond-papers-gitwikxiv/ http://blog.jessriedel.com/2015/04/16/beyond-papers-gitwikxi... [2] https://www.overleaf.com/ https://www.overleaf.com/ [3] https://arxiv.org/ https://arxiv.org/ [4] http://blog.jessriedel.com/2015/05/20/gitwikxiv-follow-up-a-path-to-forkable-papers/ http://blog.jessriedel.com/2015/05/20/gitwikxiv-follow-up-a-... [5] https://authorea.com/ https://authorea.com/ [6] https://www.pubpub.org/ https://www.pubpub.org/ [7] https://github.com/pubpub https://github.com/pubpub [8] https://medium.com/@samim/gitxiv-collaborative-open-computer-science-e5fea734cd45 https://medium.com/@samim/gitxiv-collaborative-open-computer... [9] https://gitlab.com/publishing-reform/discussion https://gitlab.com/publishing-reform/discussion [10] https://polyplexus.com/ https://polyplexus.com/
Fomite 7y ago
It saddens me somewhat that in the push between open peer review and double-blind peer review, open peer review seems to be winning. I've had a couple bad experiences with it, and vastly prefer double-blind.
- timkam 7y ago
  In many communities, the problem with double-blind peer review is that it's pretty clear who the authors are, based on the exact line of research, writing style, et cetera. And in many cases, you can also guess who the reviewer is with some confidence. This means that double-blind is a nice feature for the "big guys" who dominate their communities to show how objectively strong their research is, although in fact oftentimes their friends review their work and know exactly who the authors are. Open peer reviews would at least make sure that there is some degree of accountability.
  - Fomite 7y ago
    That's potentially true. However, I will say that as a reviewer for several double-blind journals, I've often thought "I totally know who is writing this..." I've been wrong every time.
tcpekin 7y ago
As mentioned in the article, Nat Comm has been doing this for a few years now. I have found it immensely interesting and helpful for my own research to have access to the peer review files (usually in the Supplementary Information [1]). It is a great way to show younger students the process and formatting behind peer review and also have examples of what constitutes a positive review process. As an author, it does feel like a positive as well to have the peer review file out, so that readers who are interested can see how the work developed, and what points or weaknesses the reviewers focused on. [1] randomly chosen from homepage - https://www.nature.com/articles/s41467-020-14530-7#Sec11 https://www.nature.com/articles/s41467-020-14530-7#Sec11
Vinnl 7y ago
Note that eLife will also be publishing peer review reports, even independent of "accepting" or "rejecting" a paper (and might even abandon that concept altogether): https://twitter.com/mbeisen/status/1155286615721254912 https://twitter.com/mbeisen/status/1155286615721254912 (Tweets by eLife's Editor-in-Chief.)
bluenose69 7y ago
Start by publishing the name of the editor alongside each paper. Some editors barely look at papers or even the reviews. They basically just look at whether the reviewers ticked the 'accept' or 'reject' box, and base the decision on that (after possibly getting another review to break a tie). This can go undetected for quite a while, unless the journal has a chief editor who keeps on top of things.
kaliali 7y ago
Why not have a site where people can openly comment and discuss individual research papers? This would make it more accessible to the public and increase the transparency in the research. You can have a mod that keeps the conversation on topic and in any case it's better than nothing.
- joshvm 7y ago
  This is somewhat the point of openreview, except the reviewers are still anonymous. https://openreview.net/ https://openreview.net/
isaacyes 7y ago
The British Medical Journal (BMJ) have been doing this for a while - publishing both the peer-review comments and author responses. I have published a paper in the BMJ and thought the process worked well and tempered the responses on both sides.
nestlequ1k 7y ago
They've also announced two days ago that they will have a new journal dedicated to aging. This is big news for those of us who want longer, healthier lives.

239 points by aaavl2821 7y ago

I think this is good, and may also improve reviews. I doubt that Nature has an issue with substandard reviews, but I have definitely seen it in some of the computer science conferences I submit to and have served on [1]. If reviewers know that their review may be published, I think they may turn in more substantial reviews. [1] In computer science, conference publications are peer-reviewed and highly competitive. We also have journals, but most novel work appears in conferences. The more theory oriented parts of computer science will tend to publish in journals more, but they still publish in conferences, too.

nextos 7y ago
Indeed. In my experience and other researchers I know, often rival groups will try to block your research from getting published in Nature or Science by asking for unreasonable follow-up experiments. It's sadly very political. Hopefully this will improve things. I personally know the current Nature chief editor and she totally agrees with my points above. In contrast, I have always found math & CS journal reviews way more fair and rigorous.
- BurningFrog 7y ago
  > It's sadly very political This is the predictable consequence of pre publication review by competitors. The solution is not to hope for better reviewers. It's to abolish the failed peer review system.
  - leereeves 7y ago
    > It's to abolish the failed peer review system. And replace with...?
    - jsmith45 7y ago
      In many ways the real peer review comes after publication, when more people have seen the work, and can react to it. In some fields it is not terribly rare for flaws in the work get discovered at that point, and rebuttals get written and circulated (or possibly even published in the journal.) The prepublication peer review is not always a terribly high bar to pass. It does help to weed out some of the crap, but lots still get published, especially in less competitive journals.
    - chr1 7y ago
      Publish your article with accompanying data and programs on sci-hub (github for science). Tell your colleagues about it if you normally hold seminars before publishing papers, or post to mailing list. As interested people read your paper they will post comments, open issues, or star your repository. If it gets many stars Nature will include a link to it in their monthly list of important works, which would help even more people to find your repository.
      - cozzyd 7y ago
        So that Nature will be full of papers from researchers who hire the most clickfarmers to star their work?
        chr1 7y ago
        Nature would not assign the same value to every star. Star from someone who works in a university and has large impact factor in the related field will have much higher value than a star from newly created account without any published work.
        rusticpenn 7y ago
        That is then a circular problem. Most people who have high impact right now, know how to play the publishing game.
      - meej 7y ago
        > Publish your article with accompanying data and programs on sci-hub (github for science). How does one publish on Sci-Hub? I would suggest sites like https://zenodo.org/ https://zenodo.org/ or https://osf.io/ https://osf.io/ for self-publishing research products. Or one's own institution's repository, if they have one (many do, see http://v2.sherpa.ac.uk/opendoar/ http://v2.sherpa.ac.uk/opendoar/).
- t_serpico 7y ago
  Jesus. It's unbelievable that people can be so petty. This is why if I see an academic with tons of high profile publications, it generally signals that they merely know how to play the publication game (and they of course have the desire to do so). This behavior is a red flag imo if it occurs post-tenure.
  - brazzy 7y ago
    "Petty" isn't really the right word, because it's not about inconsequential rivalries, it's about real money and careers.
    - acollins1331 7y ago
      Yes it is. It's petty to cheat to elevate oneself above ones position, or knock someone else down below theirs. Even if it's about money. Especially when it's about money.
biomcgary 7y ago
Published reviews will just make reviews political in new ways. As a graduate student, I helped my advisor review a paper submitted to Nature that was publishing a large biological data set. The internal metrics that the authors provided made it clear that the data was 95% noise. The review was direct and to the point. The paper was rightfully rejected, but ended up in another high profile journal after the authors removed the damning evidence. The corresponding author was one of the most famous and powerful people in a very large field. My advisor was still a relatively junior professor at the time, even if well known. I don't think publication of the review would have changed things for him, but for many others I have known, it might have.
- monadic2 7y ago
  Surely this was happening already without published reviews...?
  - biomcgary 7y ago
    Yes, but criticizing someone's work in a single-blind and private review leaves less motivation and opportunity for the powerful to be vindictive.
    - gindely 7y ago
      But outside of physics, there's many fields of science where double and even single blind is meaningless. If you get access to the review directly, you can tell by what they say and how they say it who they are. If you're a vindictive powerful person, who cares that you might be wrong?
    - bakuninsbart 7y ago
      Just have to publish reviews anonymously, and if there is outcry about it journals can have internal investigations on the matter.
- beerandt 7y ago
  I'd say this is actually a stronger argument for publishing reviews, as a persuasive argument might prevent venue shopping, potentially in multiple ways, depending on the when.
cozzyd 7y ago
At the very least, people will be more likely to decline to be a referee for a paper they're less of an expert on or don't have time for if they knew the report would be public.
- bumby 7y ago
  After seeing some review comments that make it clear the reviewer either had little knowledge of the field or comments that could have been rebutted by cursory view of the data, this seems like a an area that sorely needs improvement.
  - mr__y 7y ago
    >area that sorely needs improvement. the simplest idea would be to improve peer-reviewing with moving it to something similiar to hacker-news discussion board (but possibly not open to everyone), where reviewers would comment, authors could respond to questions or clarify something.
    - bumby 7y ago
      I’m not convinced that would work. If the problem is the quality of the review it seems like the reviewers need more vetting, not less. I could see the HN version devolving into a lot of wasted effort responding. It would be even worse if it was allowed to be anonymous. If it wasn’t anonymous and the reviewers were vetted with respect to the field they were commenting on, it might work
ThePhysicist 7y ago
In my (very limited) experience there are still large differences in the quality of the reviews. As researchers do this basically on the side and for free, the amount of time they devote to a review varies greatly depending on their time budget and motivation. While some reviewers seem to go through the paper character by character and point out even the tiniest flaws, others are contend with a superficial reading and miss even glaring mistakes. I'd say publishing the reviews is good because it allows you to see how much effort went into the review process. It will also (anonymously) shame the second type of reviewer into being more thorough, which I think is also good. As a side note, I think articles should all be available for free and the work of the reviewers should be compensated or appreciated somehow. Publishing reviews with the reviewers names (only for people that want it) can help to create such appreciation, because it is a lot of work.
Fishysoup 7y ago
They'll only publish reviews for already accepted papers. This may help reviewers be (or at least appear) less obviously biased against the authors in the review process if the paper has a chance of being accepted. But it won't do anything for papers that 2 or more reviewers want to shut down. Note I'm not talking about low quality research but about papers that are shut down for political reasons (happens often enough in neuroscience at least) so that the reviewers can then do the same experiments and publish before you.
- dhimes 7y ago
  I don't understand how they can get away with this blatant idea-stealing. Isn't there something in the grant process that can punish them for misbehavior? If one team has a laboratory notebook showing they did the work, it seems like a paper trail to the fraudster could easily be constructed.

adamnemecek 7y ago

The peer review process has reached a point where it's more harmful than helpful. What I want has following properties: * Version control, I can contribute to other people's papers * I can link to a sentence in another paper at some point in time so that when I read another paper, I can go to the location immediately * Open, like arxiv (anyone can publish anything), however arxiv discourages uploading personal or class projects, which is a terrible idea, the best part about GitHub is finding someone's abandoned project that does something * Full-text search I think that most academic publishing startups approach the problem by indexing already published papers. I think that starting clean slate is achievable.

Ar-Curunir 7y ago
None of the points you raised are about peer review...
- chr1 7y ago
  They are not about peer review, if by "peer review" you mean the abstract notion of someone reviewing your work. But peer review is a specific mechanism of getting comments about your work. And unfortunately it is a mechanism that has not been updated after invention of computers, so it is very inefficient at its job. Peer review in 21st century should be as easy and as open, as reviewing pull requests on github.
  - ISL 7y ago
    The comments are secondary. Filtering for error and importance are the primary goals of peer-review.
    - chr1 7y ago
      How does one "Filter for error and importance" without writing comments? I have been peer reviewer multiple times, and always had to write comments either to author pointing out errors and asking for clarifications or to the journal explaining why the paper is hopeless. It was not much different from reviewing pull requests by email. Did you have different experience reviewing papers for scientific journals?

yurlungur 7y ago

I wonder what is the impact of this change. From my limited experience peer reviews are treated as fairly private communications between the reviewers, editor and authors. We would often use it as an important clue on what to change even if we are going to another journal for publication. This makes the content of the reviews rather sensitive, especially in the cases where there is competition for publication on the same subject. There are also the cases where Reviewer X may ask authors to do a very specific analysis/improvement that may somewhat benefit their own future research. It's all par for the course. Another aspect of this that sometimes experienced professors have a fairly good sense on who's giving the review, even though they may be anonymous. Since this policy gives authors the choice of publishing reviews, I hope it does not degrade the quality of reviews and make people uncomfortable for writing "rude" feedback.

prepend 7y ago
I don’t publish or review too much, but I do a little. One of the recently frustrating events to me is that I got comments back from a reviewer that made an incorrect factual statement along the lines of “it’s against government policy to share x type of data.” And then discounted large parts of the conclusion as impossible or inappropriate. That was the only substantive comment from the reviewer. We responded back to the editors refuting it and showing the relevant policies and regulations showing how data sharing for x was allowed. The next message back was from editors declining. So further feedback. I would have really liked to learn more about why the reviewer thought what they thought and why what seemed like clear evidence to my co-authors and I wasn’t to the peer reviewer and editors. I’ll never know and I don’t spend many cycles contemplating it, but this new trial in Nature would help me understand the criticism better.

ISL 7y ago

Oh, man. I've been involved in peer-review discussions where the review correspondence is far more voluminous than the article itself. I wonder how the journals will handle the page-count.

chr1 7y ago
They should publish it only in electronic version.
- petschge 7y ago
  That does not solve the problem that more pages take longer to read. Given the pace of science these days keeping up with papers just in your subfield is an actual problem.
  - gpm 7y ago
    Just because more text is published about a work doesn't mean you have to read it...
  - mr__y 7y ago
    Do you actually read whole papers? It's an honest question. I usually scan the introduction, quickly skim through charts/tables and jump directly to conclusions. If I find anything remotely interesting during that process, then and only then I would even consider reading the whole paper.
    - petschge 7y ago
      I probably read about three full papers per week. The rest I skim similar to you.

glofish 7y ago

The current scientific publishing method is pretty much dead. It is absurd, biased, produces terrible results. Promotes inept research, unimaginable waste. Yet dead as it might be most decision-makers and other leeches of science will pretend that it is just fine and alive. They will continue to do so for many years. Decades perhaps. It is like watching Weekend at Bernie's. This attempt of Nature is little more than holding up Bernie, waving his hand. Look, it is moving! It is not dead.

coughupalung 7y ago
How about you do something constructive instead and propose a better alternative. You're just "poisoning the well" with empty accusations and ignoring how successful the scientific method has been.
- glofish 7y ago
  It is only a poisoning if it is not true. Nature is not the problem though, they just happen to be one of the most prominent and selective publishers - many/most? people hate them because they can't get in. IMHO the entire process of scientific review is wrong. Once millions of dollars in research grants depend on a paper being "accepted", once getting a grant is a zero-sum game, the scientific review is irrevocably corrupted.
  - coughupalung 7y ago
    > It is only a poisoning if it is not true. That is literally not the definition of the "poisoning the well" logical fallacy. And if you actually did publish in Nature, congrats. You still didn't offer an alternative. Edit: fixed typo
    - glofish 7y ago
      The main post is not about alternatives though - it is about a new thing that Nature is trying out. To which I posted that the whole thing is rotten from inside I don't think this is the right avenue or thread to talk about alternatives. If you want my opinion, it is the scientific publishing model the is broken, starting right with the structure of a scientific paper. The way research papers are written dates back to a time where your only source of information was the printed paper. Every single scientific publication today is overdone, exaggerated, overly verbose and almost impossible to comprehend. If you make a single, true and noteworthy observation that helps others, you cannot publish that alone. It has to be dressed up. You have to exaggerate, dress it up, bury it deep into "more" content to make it publication-worthy. The inablity to get credit for small but important research findings is the what, in my opinion is killing science from within. The big papers are a thing of a past, but scientists cannot let go, because that's what they were taught as being important.

kissickas 7y ago

> Research communities are unanimous in acknowledging the value of peer review > 82% agreed that standard peer review ensures high-quality work gets published How common is the view that peer review only serves as an institutional gatekeeping mechanism and harms innovation? (as I've heard Eric Weinstein charge on his podcast, The Portal)

coughupalung 7y ago
EW has a PhD but I can't seem to find any academic publications under his name. That doesn't outright mean he's wrong but it does mean that he actually has limited experience with the process. I have several and I've reviewed quite a few papers. I would say that peer review is a pretty good filter for garbage. Typically reviewers want to see shiny new ideas so if anything it over emphasizes innovation and discourages incremental improvements. As for gate keeping, I don't think so. Again, reviewers want to be WOW'd with new ideas. In general, I prefer papers from peer reviewed venues over the arXiv. Good work does get put on the arXiv though so I can't ignore it completely.
- bumby 7y ago
  >Typically reviewers want to see shiny new ideas so if anything it over emphasizes innovation and discourages incremental improvements. I know publications fetishize novelty, but wasn’t there research showing a high percentage of publication is either non-replicable or derivative and subsequently of little value once you remove auto-citations?

mxcrossb 7y ago

We all know that publishers add little value to science, and nature here looks to be striving to add even less than usual. If the peer reviews are not of good enough quality, your editors should be handling that! Don’t rely on the readers to review the peer review. Dumping the peer reviews online is just a way to avoid the responsibility of quality control.

chaos_emergent 7y ago
Conversely, it's also a way for the general public to see criticisms and laudations of research that might otherwise be accepted with blind faith. While Nature has its pick of comparatively incontrovertible research, sending a signal to the rest of the industry that peer reviews ought to be published seems like a useful step toward better, more replicable, more honest science.
- Vinnl 7y ago
  > While Nature has its pick of comparatively incontrovertible research Nature doesn't actually attempt to select for incontrovertible research, but rather the most groundbreaking/noteworthy research. Which is also why their retraction rate (and those of other "top" journals) is higher than average. (Disclosure: I'm part of an initiative that explicitly distinguishes between exciting and robust research for this reason.)

buboard 7y ago

Elife did this experiment long ago (one of their many innovations) https://elifesciences.org/inside-elife/e9091cea/peer-review-new-initiatives-to-enhance-the-value-of-elife-s-process https://elifesciences.org/inside-elife/e9091cea/peer-review-...

mmhsieh 7y ago

there should be a public and a private portion

cosmic_ape 7y ago

Some comments seem to miss the point that reviews will be published without naming the reviewers (unless they want to), and then only for accepted papers. So for the reviewer this should not change things much. Also, NeurIPS conference publishes reviews for a long time now. But I haven't heard of anyone talking about a review of a paper. And if you do look at these reviews, really many of them are of really dismal quality. Maybe things will be different for Nature, idk.

captain_price7 7y ago
Do you know any other ML conference/journal, doesn't have to be NeurIPS quality,that also publishes reviews? I know there is ICLR, but anything else? I really like reading them.
- cosmic_ape 7y ago
  Not that I know of. But its seems a few other conferences use OpenReview, besides ICLR. UAI for example.

esfandia 7y ago

I like the idea in principle, but it raises the bar for the quality of the review that is going to be published, and might thus discourage people from accepting to review papers. It is hard already as it is to find peer reviewers! We're getting to the point where there really has to be a financial incentive to do a peer review (Springer gives you a free book, which is a start). If that were the case, I wouldn't mind doing it occasionally when I retire.

captain_price7 7y ago
I think it can be seen as quality vs quantity issue i.e Is 2 good reviews better than 4 ok reviews?
yqx 7y ago
> We're getting to the point where there really has to be a financial incentive to do a peer review IMHO, a free book is an insulting and scientists shouldn't accept to do this work for commercial publishers for free to begin with. I know many people feel it's their responsibility to provide "services to the community" which is great, but it's ridiculous that commercial publishers get to profit from this when they proceed to claim ownership of academic work and hide it from the public behind a paywall. If they want to make money off research, fine. But reviewing costs time and money. Academic publishing may well turn out to be not so profitable anymore if reviewers have to be paid. Maybe then we can finally move to university-hosted open-access publishing.

xvilka 7y ago

All research indeed will benefit from such an approach. Think of transitioning from the lack of version control systems to their appearance. But it is not enough for better scientific publishing. There was quite an interesting discussion[1] about creating something in between Overleaf[2], ArXiv[3], Git, and Wikipedia, moreover with the ability to do a peer-to-peer review, discussion, and social networking. See also the last[4] article in that series. There are a few implementations, albeit not covering all features, like Authorea[5] and MIT's PubPub[6] (it is the open source[7]). See also GitXiv[8]. See also the Publishing Reform[9] project. Moreover, there is quite an interesting initiative from DARPA, to create the scientific social network of a kind - Polyplexus[10]. [1] http://blog.jessriedel.com/2015/04/16/beyond-papers-gitwikxiv/ http://blog.jessriedel.com/2015/04/16/beyond-papers-gitwikxi... [2] https://www.overleaf.com/ https://www.overleaf.com/ [3] https://arxiv.org/ https://arxiv.org/ [4] http://blog.jessriedel.com/2015/05/20/gitwikxiv-follow-up-a-path-to-forkable-papers/ http://blog.jessriedel.com/2015/05/20/gitwikxiv-follow-up-a-... [5] https://authorea.com/ https://authorea.com/ [6] https://www.pubpub.org/ https://www.pubpub.org/ [7] https://github.com/pubpub https://github.com/pubpub [8] https://medium.com/@samim/gitxiv-collaborative-open-computer-science-e5fea734cd45 https://medium.com/@samim/gitxiv-collaborative-open-computer... [9] https://gitlab.com/publishing-reform/discussion https://gitlab.com/publishing-reform/discussion [10] https://polyplexus.com/ https://polyplexus.com/

Fomite 7y ago

It saddens me somewhat that in the push between open peer review and double-blind peer review, open peer review seems to be winning. I've had a couple bad experiences with it, and vastly prefer double-blind.

timkam 7y ago
In many communities, the problem with double-blind peer review is that it's pretty clear who the authors are, based on the exact line of research, writing style, et cetera. And in many cases, you can also guess who the reviewer is with some confidence. This means that double-blind is a nice feature for the "big guys" who dominate their communities to show how objectively strong their research is, although in fact oftentimes their friends review their work and know exactly who the authors are. Open peer reviews would at least make sure that there is some degree of accountability.
- Fomite 7y ago
  That's potentially true. However, I will say that as a reviewer for several double-blind journals, I've often thought "I totally know who is writing this..." I've been wrong every time.

tcpekin 7y ago

As mentioned in the article, Nat Comm has been doing this for a few years now. I have found it immensely interesting and helpful for my own research to have access to the peer review files (usually in the Supplementary Information [1]). It is a great way to show younger students the process and formatting behind peer review and also have examples of what constitutes a positive review process. As an author, it does feel like a positive as well to have the peer review file out, so that readers who are interested can see how the work developed, and what points or weaknesses the reviewers focused on. [1] randomly chosen from homepage - https://www.nature.com/articles/s41467-020-14530-7#Sec11 https://www.nature.com/articles/s41467-020-14530-7#Sec11

Vinnl 7y ago

Note that eLife will also be publishing peer review reports, even independent of "accepting" or "rejecting" a paper (and might even abandon that concept altogether): https://twitter.com/mbeisen/status/1155286615721254912 https://twitter.com/mbeisen/status/1155286615721254912 (Tweets by eLife's Editor-in-Chief.)

bluenose69 7y ago

Start by publishing the name of the editor alongside each paper. Some editors barely look at papers or even the reviews. They basically just look at whether the reviewers ticked the 'accept' or 'reject' box, and base the decision on that (after possibly getting another review to break a tie). This can go undetected for quite a while, unless the journal has a chief editor who keeps on top of things.

kaliali 7y ago

Why not have a site where people can openly comment and discuss individual research papers? This would make it more accessible to the public and increase the transparency in the research. You can have a mod that keeps the conversation on topic and in any case it's better than nothing.

joshvm 7y ago
This is somewhat the point of openreview, except the reviewers are still anonymous. https://openreview.net/ https://openreview.net/

isaacyes 7y ago

The British Medical Journal (BMJ) have been doing this for a while - publishing both the peer-review comments and author responses. I have published a paper in the BMJ and thought the process worked well and tempered the responses on both sides.

nestlequ1k 7y ago

They've also announced two days ago that they will have a new journal dedicated to aging. This is big news for those of us who want longer, healthier lives.