9 ms·

Scientific Peer Review Is Broken – Fighting to Fix It with Anonymity

176 points by gkuan 12y ago

tokenadult 12y ago
The submission here is an interesting article by the founders of PubPeer, which has already been in the news quite a bit for finding examples of shoddy science papers that have had to be withdrawn by journal editors. I learned about PubPeer on the group blog Retraction Watch (RT), and I just bopped over to Retraction Watch after reading the article kindly submitted here. RT reports in detail on the defamation suit against PubPeer that is mentioned in the parent article of this thread.[1] I hope the PubPeer experiment can continue and thrive and promote better scientific research practices. Some of the other comments here suggest that anonymity of reviewers is dangerous in itself. That's why some researchers promote an open review process. Jelte Wicherts and his co-authors put a set of general suggestions for more open data in science research in an article in Frontiers of Computational Neuroscience (an open-access journal).[2] "With the emergence of online publishing, opportunities to maximize transparency of scientific research have grown considerably. However, these possibilities are still only marginally used. We argue for the implementation of (1) peer-reviewed peer review, (2) transparent editorial hierarchies, and (3) online data publication. First, peer-reviewed peer review entails a community-wide review system in which reviews are published online and rated by peers. This ensures accountability of reviewers, thereby increasing academic quality of reviews. Second, reviewers who write many highly regarded reviews may move to higher editorial positions. Third, online publication of data ensures the possibility of independent verification of inferential claims in published papers. This counters statistical errors and overly positive reporting of statistical results. We illustrate the benefits of these strategies by discussing an example in which the classical publication system has gone awry, namely controversial IQ research. We argue that this case would have likely been avoided using more transparent publication practices. We argue that the proposed system leads to better reviews, meritocratic editorial hierarchies, and a higher degree of replicability of statistical analyses." Wicherts has published another article, "Publish (Your Data) or (Let the Data) Perish! Why Not Publish Your Data Too?"[3] on how important it is to make data available to other researchers. Wicherts does a lot of research on this issue to try to reduce the number of dubious publications in his main discipline, the psychology of human intelligence. When I see a new publication of primary research in that discipline, I don't take it seriously at all as a description of the facts of the world until I have read that independent researchers have examined the first author's data and found that they check out. Often the data are unavailable, or were misanalyzed in the first place. [1] http://retractionwatch.com/2014/12/10/pubpeer-files-motion-dismiss-sarkar-defamation-case/ http://retractionwatch.com/2014/12/10/pubpeer-files-motion-d... [2] Jelte M. Wicherts, Rogier A. Kievit, Marjan Bakker and Denny Borsboom. Letting the daylight in: reviewing the reviewers and other ways to maximize transparency in science. Front. Comput. Neurosci., 03 April 2012 doi: 10.3389/fncom.2012.00020 http://www.frontiersin.org/Computational_Neuroscience/10.3389/fncom.2012.00020/full http://www.frontiersin.org/Computational_Neuroscience/10.338... [3] Wicherts, J.M. & Bakker, M. (2012). Publish (your data) or (let the data) perish! Why not publish your data too? Intelligence,40, 73-76. http://wicherts.socsci.uva.nl/Wichertsbakker2012.pdf http://wicherts.socsci.uva.nl/Wichertsbakker2012.pdf
Fuzzwah 12y ago
The obvious problem here is with fields that are small and highly specialized, such that anyone knowledgeable enough to comment on a topic are known to be from a small circle of scientists.
lisper 12y ago
Anonymity can't fix scientific peer review, it can only replace type I errors with type II errors. Instead of suppressing criticism that ought not to be suppressed, anonymity can (and often does) fail to suppress criticism that really ought to be suppressed because it is in fact false and defamatory. And indeed, what this article is really about is a lawsuit that alleges that this kind of error has in fact taken place. Ironically, the very title of this article is a model of non-scientific thinking that ought to be subject to criticism, but attempts to inoculate itself against criticism by asserting that the subjects of the piece are protagonists "fighting to fix" a broken system. They're not. They're fighting to replace one broken system with a different broken system. There is no question that scientific peer review is broken and needs to be fixed. But anonymity is not the answer. And holding anonymity up as something that should itself be beyond criticism is certainly not the answer.
- seccess 12y ago
  I agree completely. In my experience as a PhD student (comp sci), all conferences make reviewers anonymous. As a result, there is very little accountability regarding the reviews you receive for your work. More than once, I have had papers rejected simply because a single reviewer barely read the paper and dismissed it. These kinds of reviews are very frustrating to receive, not only because they failed to understand the basic premises of your paper, but because these reviews contain no useful information on how to make your paper better for future submissions. Of course, the opposite can happen where mediocre works slips through, and the reviewers that allowed that should be held accountable too. Its painful to me that so much of the acceptance process for research papers (in my field at least) is based on luck. Moral of the story: reviewer anonymity is good but it comes at the expensive of accountability.
  - javert 12y ago
    I'm also a computer science PhD student. I don't disagree with anything you said, but I am even more cynical. > Of course, the opposite can happen where mediocre works slips through Slips through? Mediocre work is the norm, or at least close to being the norm. (And, as you say, good work is often rejected.) > Its painful to me that so much of the acceptance process for research papers (in my field at least) is based on luck. I don't think it's based on luck, it's based on politics. Academic science is a racket. There is some luck too, though.
HandleTheJandal 12y ago
Thank you for helping to promote reproducibility of published results by supporting anonymous peer review! Science Exchange (YC S11) is also making great progress in the facilitation of scientific reproducibility. We just completed independently validating select results from 50 cancer biology papers. https://news.ycombinator.com/item?id=8731274 https://news.ycombinator.com/item?id=8731274
Fede_V 12y ago
I think the increase in fraudulent papers getting published is a symptom, not the disease. The disease is that there are too many scientists doing research for a given research budget. Due to the ferocious competition for grant money, people are either sloppy and cut corners, or do whatever they need to do and outright cheat to publish in top tier journals. As a society, we need to make a decision about how much we want to fund scientific research - then, once we've made that decision, insure that we put in place a sustainable system in place - we cannot put in a put of money to finance 100 grants, but then build a pipeline that funnels an ever increasing amount of people into a pool that remains constant.
- dluan 12y ago
  Funding review is different than pre-publication or even post-publication review of results. One argument is that review of results is easier for a reviewee to defend in public, since the data and results are stated and there's nothing to hide. Reviewing funding proposals is more speculative once you get past basic methods or ethics review (e.g. depends on the risk appetite of the funders) and therefore might require anonymity. This is something we've been grappling at Experiment. It's possible a better design of the system is a double blind review for proposals but a public post-pub review for results.
- snowballsteve 12y ago
  One could argue that papers are the disease. It was a good medium, but with today's communication technology, something better needs to arise. The need to "publish to exist" in academia has gotten to the point that it promotes sub-standard work and hinders advancement of knowledge.
- jessriedel 12y ago
  Do you agree that that's a "command and control" approach to the market for research labor? In other words, it sounds like you want to have the government set the number of researchers in the field rather than simply pay for the amount of research it wants, in the same way some governments (the USSR is the classical example) in the past have decided on the number of manufacturers in a certain industry, not just how much they want to pay for a certain widget. All things being equal, such approaches are generally discouraged by economists. So presumably you think that the market for research labor is unusual, e.g. has some market failure. Could you describe what you think that is? Basically, my angle is that, once you've answered this question, it's almost always better to try to fix the market failure directly rather than go full command and control. For instance, if you think that prospective PhD students aren't well informed about the long odds against getting a tenured position, then it would be better to educate those students rather than instituting a cap on the number of PhDs that are funded.
vacri 12y ago
Fortunately, the First Amendment is on our side. It protects the right to anonymous speech. I don't understand how "The government cannot outlaw your speech" means "Your anonymity is protected by law from private parties". How does that interpretation come about?
- dalke 12y ago
  http://en.wikipedia.org/wiki/Dendrite_International,_Inc._v._Doe_No._3 http://en.wikipedia.org/wiki/Dendrite_International,_Inc._v.... . > "the court must balance the defendant's First Amendment right of anonymous free speech against the strength of the prima facie case presented and the necessity for the disclosure of the anonymous defendant's identity."
  - hibikir 12y ago
    I don't think that opinion says what you think it says. What is really being discussed there is if the government could compel Yahoo to unmask an anonymous user. It says nothing about actual anonymity guarantees among private parties. If Yahoo decided to hand over the account details, the court would not have intervened at all. So when you have an account with some service, any expectations on how the company will handle the data come from the TOS. In a similar fashion, a company is free to delete any content you submit to their website. The government can't curtail your speech, barring a few exceptions, but a company certainly can, as long as the government is not involved.
    - dalke 12y ago
      We know that the government could compel Yahoo to unmask an anonymous user - get a FISA order to inspect the business records and, poof, done. So I don't see how that's the issue. Here's the breakdown: "A prominent cancer scientist, unhappy with the attention his research papers have received on PubPeer, is suing some of our anonymous commenters for defamation." That's a third-party, who is not a government organization, suing PubPeer to de-anonyomize a commenter on PubPeer's site. "Dendrite International, Inc., a purveyor of computer software used in the pharmaceutical industry, brought a John Doe lawsuit against individuals who had anonymously posted criticisms of the company on a Yahoo message board. When Presiding Chancery Judge Kenneth MacKenzie rejected one of Dendrite's requests to compel Yahoo to reveal the identity of an anonymous defendant, Dendrite appealed." That's a third party, Dendrite, who is not a government organization, suing Yahoo to de-anonyomize a commenter on Yahoo's site. If Yahoo wanted to, they could turn that information over. The question was, could Dendrite compel Yahoo to do so if Yahoo didn't want to. And the answer is that there's a First Amendment issue which must be considered, so the answer is "sometimes." Yahoo didn't want to. And PubPeer doesn't want to. So I don't understand why you are bringing up first party actions (actions that Yahoo and PubPeer are free to do) when the issue is what third party actions may compel them to do.
ejz 12y ago
One of the main issues is that this is just a really tedious process. No one wants to go through algebra; that's why it's siphoned off to grad students. Do you really think a tenured professor is going to spend time checking the grunt work on /someone else's/ paper when they won't do it for their own? I'm intrigued by the possibility of using natural language processing and logical system tools like Wolfram Alpha. Wolfram recently posted on his blog about building machines that could store data about complex mathematical objects, and already you can build machines that confirm first order logical statements. Farming out low-level tasks to automated systems would be interesting. Imagine if the format of papers changed entirely, ie, you had to submit your proofs in certain formats, or at least certain parts in specific ways. I'm sure that many professors would be elated to see the number of papers they have to review go down drastically; although, I'm sure many will be disappointed to get a return letter that says, "I'm sorry, but the low level flaws were so serious that they were automatically rejected and are not fit for review."
- ylem 12y ago
  I'm a permanent staff member and I do actually check "algebra" and integrals when reviewing papers. Especially "technique" papers, because after the initial paper, people tend to make black boxes, so if the initial errors aren't caught then they can propagate for quite some time...I view it as community service.
Animats 12y ago
Another anonymous review system. What could possibly go wrong? If there's anything we've learned by now about "crowdsourced" review systems, it's that, without an elaborate way to evaluate reviewers, they fail. Badly. Facebook "likes", Google "+1", and Yelp reviews are heavily spammed. This just does not work.
jedbrown 12y ago
I take the opposite approach: I sign my reviews and have ever since I was a grad student. It compels me to do a better job reviewing and often leads to further discussion and sometimes collaboration with the authors. Some people don't like what I have to say, but by and large, they respect it. For many topics that I review for, the authors will have a pretty good idea that I wrote the review unless I am intentionally vague. I would rather write clearly and directly and stand behind it. It is a professional risk that I don't think anyone should be compelled to take, but I think signing reviews is generally good for science.
- derf_ 12y ago
  Yes, I think anonymity in reviews is largely a myth, even in "double blind" reviews. For many topics, there are only a few research groups actively pursuing them, and you can often tell from what other work they cite, what datasets they use, etc., which lab it is. So identifying at least some of the authors is relatively easy. You might not get the exact grad student right, but you probably know their advisor. Identifying reviewers is a little harder, but it's usually not difficult to make educated guesses unless they are, as you put it intentionally vague. Otherwise the other comments on this article about bad reviews from 'people from "a different camp" simply disliking a given approach' and the like would be impossible to justify.
dougmccune 12y ago
The actual complaint filed shows pretty nasty behavior. Here's the full complaint: https://retractionwatch.files.wordpress.com/2014/10/filed-complaint.pdf https://retractionwatch.files.wordpress.com/2014/10/filed-co... Someone very clearly was trying to get Dr. Sarkar fired. The PubPeer comments are only one small aspect. But the person then also (I assume anonymously) emailed his new employer to allege fraud, and then went so far as to print out these allegations and stamped them with official looking nonsense about being from the an NIH investigation (which didn't exist), and distributed them throughout Sarkar's department mail boxes. The complaint makes a decent case for why they think pretty much all the negativity directed at this researcher is likely from one angry person. They obviously can't prove that multiple anonymous comments are from the same person, but reading the content it certainly seems likely. What is assumed to be the same person then took things way past the line of what most of us would consider ethical. There may very well be a place for anonymous calling out of potential research misconduct. But making fraud allegations anonymously online, then printing out those comments, trying to fake them to look like an official government inquiry, and physically delivering them to the researcher's boss at his place of work isn't the way to do that. This case might be one of those cases where the anonymity should be protected at all costs out of principle, but it's a really shitty case to wave your "we're the good guys" flag for.
- tokenadult 12y ago
  A complaint is just a pleading before any trial of the facts has occurred. The statements about what may have happened (or may not have happened) are under the heading "FACTUAL ALLEGATIONS" in the document you kindly linked. So to describe the same issues you describe, I would write, "The actual complaint alleges some pretty nasty behavior." We have yet to see how many of those allegations will be believed by a trier of fact at trial. The plausible evidence of persistent research misconduct in this case seems to be a reasonable basis for someone not to get a new job with more research responsibilities, so there may not be any actual harm to the plaintiff here. The defense of truth is strong in a defamation case.
  - Natsu 12y ago
    Not sure why this was downmodded, but I tried to correct that. I do note that people have a bias to believe the first story they hear and it's hard to consciously correct for this. Merely knowing about the bias is not enough to correct it and I realize that I'm not immune to it myself. As far as I know, I don't know any of the people involved in this, but I've read enough legal complaints to know that they're not always as factual as they first appear, so I try to consciously restrain myself from simply believing it without hearing the rebuttal. Rushing to judgement may be quite common these days, but it's not the sort of behavior fair or intelligent people should engage in. It's weird that they lead in with the first few comments, which were tame enough that I discount them completely as being an appropriate subject for a lawsuit, though they seem to get progressively worse and some of the later allegations are much more serious. I do wonder why they lead with their weakest cards, though. That aside, it does seem that the site it exercising editorial control over the comments, which may work against it with respect to the CDA, even though it's trying to do the right thing based on its policy as quoted in the complaint. Given the policy, I wonder how the later comments ever went through, as it seemed like the site wanted to avoid drama like this entirely. Even if this was a lapse of some kind, it might work against them. As I have not heard the rebuttal, I will refrain from making any judgements one way or another about who is right here. It's only fair, after all. EDIT: Retraction Watch is covering this case. I believe them to be a reliable, respected and neutral 3rd party: http://retractionwatch.com/2014/12/10/pubpeer-files-motion-dismiss-sarkar-defamation-case/ http://retractionwatch.com/2014/12/10/pubpeer-files-motion-d...
ylem 12y ago
This is an interesting problem. On the one hand, there are merits to a referee being anonymous to the submitter. Part of this may be to avoid reprisal for younger referees, but even for more established referees, you may be freer to comment if you are anonymous to the author. BUT, you are not completely anonymous. Hopefully (though there have been some recent scandals related to this) an editor of the journal knows your work and has chosen to use you as a referee based on that. This can help to keep down some of the noise that another poster (lisper) mentioned. Also, the fact that the editor knows who you are may provide some constraint on how you may phrase a review as compared to if you were completely anonymous. I don't think this is something one should issue a lawsuit over--but I also don't think that their proposal of completely anonymous review is at all useful.
kazinator 12y ago
> A prominent cancer scientist, unhappy with the attention his research papers have received on PubPeer, is suing some of our anonymous commenters for defamation On the other hand, should anonymous commenters have the balance of power: in other words, say whatever they want with impunity, even if it actually is defamatory? (Not saying that is the case in this situation, but in general). The problem is that defamation is a legal concept, which can only be tried legally. So for instance, whereas a site can rigorously enforce rules which say that all comments are directed at the research material, and not at persons, and have a factual basis in that material, those measures cannot take away the right of someone, who feels they have been defamed, to take the matter to court (where they will almost certainly lose, which is neither here nor there). You can't just create a site and declare it above the law, so to speak. The only way to protect the identities of the anonymous is for the site to take responsibility for the statements it publishes: to assert that the anonymous statements are subject to rigorous standards of review, and when published, they in fact reflect the views of PubPeer, and PubPeer alone, and not of any anonymous persons (who do not publish any statements, but act only as sources of information). Then if someone feels they have been the target of defamation, the defendant shall be PubPeer.
- jessriedel 12y ago
  This situation is reminiscent of websites hosting copyrighted material posted by anonymous users. Does the relevant law (DMCA, I think) say anything about defamation? A reasonable compromise might be that websites are held harmless so long as they respond to official defamation complaints in some prescribed manner. (Of course, this might not work as well because defamation is significantly less clear-cut than copyright violations, which already has plenty of cases of over-reach due to disputes over fair use, etc.)
  - joshuacc 12y ago
    I think you may mean "prescribe" rather than "proscribe." :-)
    - jessriedel 12y ago
zmanian 12y ago
It seems plausible to imagine a system where PeerPub could retain their "published author to register This model relies on t a blind cryptographic signature.(https://en.wikipedia.org/wiki/Blind_signature https://en.wikipedia.org/wiki/Blind_signature) PeerPub generates a public/private key pair. 1. Alice wishes to register on PeerPub. She generates a Nonce N. 2. She blinds the nonce with a factor B. 3. She submits the blinded nonce and her identity to PeerPub. PeerPub checks her credentials and executes a blind signature of the number and returns it to Alice. 4.Alice now separately registers an account perhaps using a privacy protecting system like Tor. She uses the original N and the unblinded Signature. 5. PeerPub verifies N + signature and registers the account. PeerPub will have no way of linking N to the original credentials. PeerPub can record N and make sure it can only be used once to register an account.
- sweis 12y ago
  Not specific to PeerPub, but I helped work on a prototype of a federated login system based on OpenID that used blind signatures in a similar fashion: http://private-idp.appspot.com/ http://private-idp.appspot.com/ https://code.google.com/p/pseudoid/ https://code.google.com/p/pseudoid/ http://blind-signer.appspot.com/ http://blind-signer.appspot.com/ I haven't touched it for years and it looks like the login flow is broken now with the demo sites. Also, this uses unsafe server-side JS crypto for demo purposes only.
  - zmanian 12y ago
    I emailed the PeerPub people about something like this. Will see if they bite.
pizza_boy 12y ago
With Publons.com (https://publons.com https://publons.com) we have different philosophy: the more transparency we can bring to the review process, the better. At the same time we recognise that both blind and double-blind peer review play an important role in generating quality research. Our approach is to focus on turning review of all kinds (including both pre- and post-publication) into a measurable research output -- something you can add to your resume. We support both anonymous and signed review with the idea that it will lead to greater transparency in the long run and also motivate reviewers to contribute more. We have a significant number of both types of review now and are starting to look ways to measure if there are significant differences between blind and open review.
- efangs 12y ago
  Hi, I like the site and idea. I was wondering if you were thinking of taking it one step further, however, and making it an actual pre-print server? I would think that if you expect people to take the "credit for reviewing" model seriously, then likewise they should take seriously the notion of "credit for being reviewed" or "credit for being upvoted". Also, there would have to be a way to verify submitters/reviewers credentials (e.g. connection to a research institution). Obvious arXiv has a model for how this could be done.
weissadam 12y ago
My understanding is that most peer review systems in place at various journals and funding agencies today are already anonymous (except for when people are identified by their well known viewpoints.) If you ask me, the real problem can't be solved with communications technology, the real problem must be solved at the source: The funding agencies need to take the importance of reproduction of results seriously and require their grantees to do a certain amount of rote reproduction work in order to qualify for grants for novel research. Will it slow the pace of things down? Certainly. Will it increase the quality of the science? Certainly.
esbio 12y ago
As a person who did research until a few years ago, I must say that the problem is the exact opposite. When you send an article to a journal, the paper gets reviewed by a number of peers, which send their comments back to the Editor on the appropriateness of the claimed work. The problem with this mechanism is that reviewers have no liability, because their comment is anonymous to the author and won't be available to the readers, as it won't be published as part of the article. The result is that reviewers are not made accountable now or in the future for inaccuracies in their review, blatant attacks, or tactical requests for additional irrelevant investigation just out of spite or to stall you so that they can scoop your paper. Occasionally, the Editor can step in and disregard a particularly obnoxious reviewer, but it depends on the editor, the journal, and the political/scientific strength of the reviewer.
- seanmcdirmid 12y ago
  > because their comment is anonymous to the author and won't be available to the readers You can make comments on your papers available; there is nothing against that. I really think, however, peer review comments should actually be published with rejected and accepted papers (at the author's consent, not reviewers) so that conferences can be more transparent.
  - esbio 12y ago
    You can, but 1) there's no official channel 2) nobody will do it if they are not forced.
nemoniac 12y ago
What the scientific peer review needs is not anonymity, but accountability. Authors put their reputation on the line. Let reviewers put their reputation on the line too. Good reviews and good reviewers need to be appreciated, not anonymized. Let valued scientific reviewers gain reputation in a similar way to how contributors to HN, SE and other sites do. There could even be a viable business model in this. Publishers, be creative!
ignostic 12y ago
What we need here is a more nuanced approach. Anonymity can solve some problems in research, but it will make other worse. When people are anonymous, they ARE more likely to be truthful in their criticism. They have less incentive to hold back, and it's just human nature to tone down critical feedback when you're critiquing the work of someone with is either influential or an acquaintance. No one likes to make enemies. On the other hand, anonymity can pretty clearly bring out some of the worst in us. Some people feel little obligation to be fair or honest when their reputation isn't on the line, and so you see people trying to knock down rivals, people they don't like, or random strangers just for the "thrill of the troll." Imagine if every time you applied for a job your potential employer had access to anonymous feedback on your past work. Some of it might be fair and honest (whether positive or negative), but some of it might be lies from an anonymous coworker with a grudge. Maybe someone is trying to take you down a rung because you got the promotion over them. You could be penalized for any petty reason, and it would stick with you. Anonymous feedback communicated publicly is much the same. It holds the reviewer and the object of review on unequal footing. Anonymous feedback would be great for an author or even an editor, but it's just not fair to allow the pettiest of people to attack the works of others while wearing a mask. I'd like a system that helps researchers and others invested in the work to solicit anonymous feedback to make the work better. Public-facing commentary, on the other hand, should be tied to an identity.
pc2g4d 12y ago
Much of the discussion here seems to have become "anonymity vs. non-anonymity" --- i.e. either anonymity is good for research or it's bad for research. Why not just accept that in some venues there will be anonymity, and in some not, and let them each develop according to the merits of the respective approaches? No need to have all research anonymous or all research clearly attributed to a public identity.
kgarten 12y ago
I don't get their stance ... peer-review is already "anonymous". There are also a lot of issues with truly anonymous user forums (see 2chan and 4chan). For me it always seems as if the social aspects of communication disappear when one is truly anonymous, e.g. hate speech. The problem is not lack of anonymity but lack of incentives for reviews (I don't get anything from doing a thorough review of a paper and often it's hard to impossible to judge the contribution without dataset and code). It seems peerpub and similar systems will attract people who have the incentive to attack specific authors (as it happened in this case).
colechristensen 12y ago
Let's be sure to point out some terrible journalism, reading the following makes me completely disinterested in finishing the article and have considerably less respect for Wired. > Have you ever questioned the claims that scientists make? For example, last year’s discovery of the so-called “God particle,” Using the nickname for the Higgs in the context of questioning science claim is nothing but bait for the foolish and misinformed, and in this way anyone scientifically literate should seriously question any claims or opinions in this publication.
LiweiZ 12y ago
I don't think push some form of crowd intelligence would change much. I guess the best thing we could do is to create some conditions to let new nodes emerge and complete. I also believe better funding distribution mechanism/system will help a lot.
atsaloli 12y ago
Dr. Mark Burgess just blogged "Why I stopped caring about peer review, and learned to love the work". http://markburgess.org/blog_peer.html http://markburgess.org/blog_peer.html
analog31 12y ago
Something I've noticed is that most online articles about scientific misconduct and invalid results revolve around the medical sciences. Granted, medicine is probably the biggest piece of the scientific pie right now, and other branches of science may have their own problems, but I think it misleads the public to have a title like "scientific peer review is broken," for an article that focuses exclusively on one branch of science. Disclaimer: I'm a physicist. I'm sure that physics is not beyond critique, but unless somebody is willing to take enough of an interest in the inner workings of physics research to say something specific about it or credibly include it in a generalization, I'd rather say something like "medical peer review is broken."
oldsciemun333 12y ago
A thought suggestion: 1.) why this is important? THE GREAT CHINESE FAMINE - Mao's Hungry Ghosts book. not against mao or china. Belief in Lysenko - aka "we will grow oranges at the North Pole." will the great famine come again due to BAD SCIENCE to the USA? GMO crops / invasive pests / scientific 'POLITICS.' 2.) Of course will the science review in Japan, the plans are now to DOUBLE THE NUCLEAR POWER plants, after Fukushima 'incident.' 3.) So, as an average citizen of the USA, sometime reading of HN, I volunteer as 'anonymous submitter'.' 4.) I've noticed WORLD SCIENTISTS. that your code base is mostly broken. Even engineering codes use Matlab - What in the blazes??? and compiler tricks. Obviously, the scientists have NOT heard of Amazon, haskell or even of open source software. 12.) Basic focus folks like me. yes, how many pushups did you do today? even TRIPLE BLIND - see 'scientific method.' That means I as a VOLUNTEER submit the paper. GO AHEAD AND ATTACK 'the paper' of my repuation. I do NOT plan to be a professor at a majolr unversity. for the record, dropout grad school. and yes, I am still completing my studies when I am a bit older. Alas, I am not hopeful. 14.) first it was religion, Catholic Church and the inquisition 15.)then it was 'the great leader' like mao, etc or the suddenly the ROBBER BARRONS like Tom Editson (high school dropout) becoming sudden geniuses. 16..) now its, the game let's get the professor fired by going direcctly to the Board of Trustees of California State U or others. 17.) when in doubt use the 'federal agencies' to murky up the waters. 18.) Tobacco is good for you and healthy (when in moderation) some say. 19.)MY LAST SUGGESTION. the paper be chopped into parts. I don't want to see the strange science or arcane languages. I only want some of the sampled data to SHOOT HOLES THROUGH using python and haskell. the book is forthcoming - how to lie subtlely with Statistics and meta heuristics... THIS IS THE KEY WEAKNESS OF PEER review. Who is really a peer? and most are working at the same college and same area. want proof? See OpenSSH - and apple goto goto fail.
hyperion2010 12y ago
I usually think that peer review is broken, but that is because we almost always miss the point of what the actual purpose of peer review is for. Peer review is NOT for laymen. A scientific paper is part of a much longer process of discovery which includes massive amounts of debate and argument. Papers always reflect they very edge of a field where literally the only people who can judge whether the results are plausible are peers because they are the ones who are also trying to make sense of the same phenomenon. Over time some primary research papers are singled out because they really are excellent explanations of a phenomenon most of the time however you have to go look at a review paper. Furthermore peer review is absolutely NOT about reproducibility, it is a prerequisite that says that it MIGHT be worth trying to reproduce this work or work contingent on these results being valid. There are so many reasons why a result might be wrong that it often takes fields years to figure it out and the published literature is a record of that.
daemonk 12y ago
Anonymity is hard in practice when so much of science consist of small niche fields where it can be pretty obvious who the author of a paper is by just reading the content. A better solution might be to enforce non-anonymous peer-reviews that can be read by the public after the paper has been published.
- tnhh 12y ago
  Some venues are doing this, e.g. f1000. Here's a paper that I reviewed: http://f1000research.com/articles/3-38/v2 http://f1000research.com/articles/3-38/v2
sideshowb 12y ago
Not as deep an article as I'd hoped, but it points to a difficult question. As reviewers of scientific papers have unprecedented power over what research gets published, who reviews the reviewers? (A personal example - I have just spent a week constructing a counterargument to a reviewer who didn't read my paper properly. Imagine having an online discussion where your career success hangs on the response of an anonymous, disengaged flamer to a single post of yours: can you imagine how much effort you put into that post? Sadly this particular argument is pointless and serves nobody; it would be better use of my time to get on with actual research, but that's not how the system works).
p4bl0 12y ago
Peer review might be broken, but it's not anonymity that will fix it. For the simple reason that we already have double-blind reviews in many fields and the peer reviewing process is not any less broken there. There are people who are thinking of multiple solutions, one that I think is interesting is the proposition of Open Scholar, dubbed "Independant Peer-Review". I submitted it to HN so we can discuss it without flooding this thread, if it interests HNers: https://news.ycombinator.com/item?id=8734271 https://news.ycombinator.com/item?id=8734271. In there scheme, the papers are open access from the beginning, the peer reviewing process is open, and reviews are citable. I believe it would encourage better and deeper reviews. It would also be very nice for young scientists or students who want to apply for a PhD grant to be able to show that they are able to write a comprehensive review of a paper in their field, and that their review was good enough to significantly improve the paper or to be selected by the authors and/or publishers to be released alongside the paper, for instance.
return0 12y ago
Pubpeer is not a peer review site, it's site with comments. I think they are doing it wrong by promoting it as "anonymous peer review", because by definition if its anonymous it can't be "peers". It should (continue to) be a companion to published papers that every once in a while does the service of spotting an error.

176 points by gkuan 12y ago

The submission here is an interesting article by the founders of PubPeer, which has already been in the news quite a bit for finding examples of shoddy science papers that have had to be withdrawn by journal editors. I learned about PubPeer on the group blog Retraction Watch (RT), and I just bopped over to Retraction Watch after reading the article kindly submitted here. RT reports in detail on the defamation suit against PubPeer that is mentioned in the parent article of this thread.[1] I hope the PubPeer experiment can continue and thrive and promote better scientific research practices. Some of the other comments here suggest that anonymity of reviewers is dangerous in itself. That's why some researchers promote an open review process. Jelte Wicherts and his co-authors put a set of general suggestions for more open data in science research in an article in Frontiers of Computational Neuroscience (an open-access journal).[2] "With the emergence of online publishing, opportunities to maximize transparency of scientific research have grown considerably. However, these possibilities are still only marginally used. We argue for the implementation of (1) peer-reviewed peer review, (2) transparent editorial hierarchies, and (3) online data publication. First, peer-reviewed peer review entails a community-wide review system in which reviews are published online and rated by peers. This ensures accountability of reviewers, thereby increasing academic quality of reviews. Second, reviewers who write many highly regarded reviews may move to higher editorial positions. Third, online publication of data ensures the possibility of independent verification of inferential claims in published papers. This counters statistical errors and overly positive reporting of statistical results. We illustrate the benefits of these strategies by discussing an example in which the classical publication system has gone awry, namely controversial IQ research. We argue that this case would have likely been avoided using more transparent publication practices. We argue that the proposed system leads to better reviews, meritocratic editorial hierarchies, and a higher degree of replicability of statistical analyses." Wicherts has published another article, "Publish (Your Data) or (Let the Data) Perish! Why Not Publish Your Data Too?"[3] on how important it is to make data available to other researchers. Wicherts does a lot of research on this issue to try to reduce the number of dubious publications in his main discipline, the psychology of human intelligence. When I see a new publication of primary research in that discipline, I don't take it seriously at all as a description of the facts of the world until I have read that independent researchers have examined the first author's data and found that they check out. Often the data are unavailable, or were misanalyzed in the first place. [1] http://retractionwatch.com/2014/12/10/pubpeer-files-motion-dismiss-sarkar-defamation-case/ http://retractionwatch.com/2014/12/10/pubpeer-files-motion-d... [2] Jelte M. Wicherts, Rogier A. Kievit, Marjan Bakker and Denny Borsboom. Letting the daylight in: reviewing the reviewers and other ways to maximize transparency in science. Front. Comput. Neurosci., 03 April 2012 doi: 10.3389/fncom.2012.00020 http://www.frontiersin.org/Computational_Neuroscience/10.3389/fncom.2012.00020/full http://www.frontiersin.org/Computational_Neuroscience/10.338... [3] Wicherts, J.M. & Bakker, M. (2012). Publish (your data) or (let the data) perish! Why not publish your data too? Intelligence,40, 73-76. http://wicherts.socsci.uva.nl/Wichertsbakker2012.pdf http://wicherts.socsci.uva.nl/Wichertsbakker2012.pdf

Fuzzwah 12y ago

The obvious problem here is with fields that are small and highly specialized, such that anyone knowledgeable enough to comment on a topic are known to be from a small circle of scientists.

lisper 12y ago

Anonymity can't fix scientific peer review, it can only replace type I errors with type II errors. Instead of suppressing criticism that ought not to be suppressed, anonymity can (and often does) fail to suppress criticism that really ought to be suppressed because it is in fact false and defamatory. And indeed, what this article is really about is a lawsuit that alleges that this kind of error has in fact taken place. Ironically, the very title of this article is a model of non-scientific thinking that ought to be subject to criticism, but attempts to inoculate itself against criticism by asserting that the subjects of the piece are protagonists "fighting to fix" a broken system. They're not. They're fighting to replace one broken system with a different broken system. There is no question that scientific peer review is broken and needs to be fixed. But anonymity is not the answer. And holding anonymity up as something that should itself be beyond criticism is certainly not the answer.

seccess 12y ago
I agree completely. In my experience as a PhD student (comp sci), all conferences make reviewers anonymous. As a result, there is very little accountability regarding the reviews you receive for your work. More than once, I have had papers rejected simply because a single reviewer barely read the paper and dismissed it. These kinds of reviews are very frustrating to receive, not only because they failed to understand the basic premises of your paper, but because these reviews contain no useful information on how to make your paper better for future submissions. Of course, the opposite can happen where mediocre works slips through, and the reviewers that allowed that should be held accountable too. Its painful to me that so much of the acceptance process for research papers (in my field at least) is based on luck. Moral of the story: reviewer anonymity is good but it comes at the expensive of accountability.
- javert 12y ago
  I'm also a computer science PhD student. I don't disagree with anything you said, but I am even more cynical. > Of course, the opposite can happen where mediocre works slips through Slips through? Mediocre work is the norm, or at least close to being the norm. (And, as you say, good work is often rejected.) > Its painful to me that so much of the acceptance process for research papers (in my field at least) is based on luck. I don't think it's based on luck, it's based on politics. Academic science is a racket. There is some luck too, though.

HandleTheJandal 12y ago

Thank you for helping to promote reproducibility of published results by supporting anonymous peer review! Science Exchange (YC S11) is also making great progress in the facilitation of scientific reproducibility. We just completed independently validating select results from 50 cancer biology papers. https://news.ycombinator.com/item?id=8731274 https://news.ycombinator.com/item?id=8731274

Fede_V 12y ago

I think the increase in fraudulent papers getting published is a symptom, not the disease. The disease is that there are too many scientists doing research for a given research budget. Due to the ferocious competition for grant money, people are either sloppy and cut corners, or do whatever they need to do and outright cheat to publish in top tier journals. As a society, we need to make a decision about how much we want to fund scientific research - then, once we've made that decision, insure that we put in place a sustainable system in place - we cannot put in a put of money to finance 100 grants, but then build a pipeline that funnels an ever increasing amount of people into a pool that remains constant.

dluan 12y ago
Funding review is different than pre-publication or even post-publication review of results. One argument is that review of results is easier for a reviewee to defend in public, since the data and results are stated and there's nothing to hide. Reviewing funding proposals is more speculative once you get past basic methods or ethics review (e.g. depends on the risk appetite of the funders) and therefore might require anonymity. This is something we've been grappling at Experiment. It's possible a better design of the system is a double blind review for proposals but a public post-pub review for results.
snowballsteve 12y ago
One could argue that papers are the disease. It was a good medium, but with today's communication technology, something better needs to arise. The need to "publish to exist" in academia has gotten to the point that it promotes sub-standard work and hinders advancement of knowledge.
jessriedel 12y ago
Do you agree that that's a "command and control" approach to the market for research labor? In other words, it sounds like you want to have the government set the number of researchers in the field rather than simply pay for the amount of research it wants, in the same way some governments (the USSR is the classical example) in the past have decided on the number of manufacturers in a certain industry, not just how much they want to pay for a certain widget. All things being equal, such approaches are generally discouraged by economists. So presumably you think that the market for research labor is unusual, e.g. has some market failure. Could you describe what you think that is? Basically, my angle is that, once you've answered this question, it's almost always better to try to fix the market failure directly rather than go full command and control. For instance, if you think that prospective PhD students aren't well informed about the long odds against getting a tenured position, then it would be better to educate those students rather than instituting a cap on the number of PhDs that are funded.

vacri 12y ago

Fortunately, the First Amendment is on our side. It protects the right to anonymous speech. I don't understand how "The government cannot outlaw your speech" means "Your anonymity is protected by law from private parties". How does that interpretation come about?

dalke 12y ago
http://en.wikipedia.org/wiki/Dendrite_International,_Inc._v._Doe_No._3 http://en.wikipedia.org/wiki/Dendrite_International,_Inc._v.... . > "the court must balance the defendant's First Amendment right of anonymous free speech against the strength of the prima facie case presented and the necessity for the disclosure of the anonymous defendant's identity."
- hibikir 12y ago
  I don't think that opinion says what you think it says. What is really being discussed there is if the government could compel Yahoo to unmask an anonymous user. It says nothing about actual anonymity guarantees among private parties. If Yahoo decided to hand over the account details, the court would not have intervened at all. So when you have an account with some service, any expectations on how the company will handle the data come from the TOS. In a similar fashion, a company is free to delete any content you submit to their website. The government can't curtail your speech, barring a few exceptions, but a company certainly can, as long as the government is not involved.
  - dalke 12y ago
    We know that the government could compel Yahoo to unmask an anonymous user - get a FISA order to inspect the business records and, poof, done. So I don't see how that's the issue. Here's the breakdown: "A prominent cancer scientist, unhappy with the attention his research papers have received on PubPeer, is suing some of our anonymous commenters for defamation." That's a third-party, who is not a government organization, suing PubPeer to de-anonyomize a commenter on PubPeer's site. "Dendrite International, Inc., a purveyor of computer software used in the pharmaceutical industry, brought a John Doe lawsuit against individuals who had anonymously posted criticisms of the company on a Yahoo message board. When Presiding Chancery Judge Kenneth MacKenzie rejected one of Dendrite's requests to compel Yahoo to reveal the identity of an anonymous defendant, Dendrite appealed." That's a third party, Dendrite, who is not a government organization, suing Yahoo to de-anonyomize a commenter on Yahoo's site. If Yahoo wanted to, they could turn that information over. The question was, could Dendrite compel Yahoo to do so if Yahoo didn't want to. And the answer is that there's a First Amendment issue which must be considered, so the answer is "sometimes." Yahoo didn't want to. And PubPeer doesn't want to. So I don't understand why you are bringing up first party actions (actions that Yahoo and PubPeer are free to do) when the issue is what third party actions may compel them to do.

ejz 12y ago

One of the main issues is that this is just a really tedious process. No one wants to go through algebra; that's why it's siphoned off to grad students. Do you really think a tenured professor is going to spend time checking the grunt work on /someone else's/ paper when they won't do it for their own? I'm intrigued by the possibility of using natural language processing and logical system tools like Wolfram Alpha. Wolfram recently posted on his blog about building machines that could store data about complex mathematical objects, and already you can build machines that confirm first order logical statements. Farming out low-level tasks to automated systems would be interesting. Imagine if the format of papers changed entirely, ie, you had to submit your proofs in certain formats, or at least certain parts in specific ways. I'm sure that many professors would be elated to see the number of papers they have to review go down drastically; although, I'm sure many will be disappointed to get a return letter that says, "I'm sorry, but the low level flaws were so serious that they were automatically rejected and are not fit for review."

ylem 12y ago
I'm a permanent staff member and I do actually check "algebra" and integrals when reviewing papers. Especially "technique" papers, because after the initial paper, people tend to make black boxes, so if the initial errors aren't caught then they can propagate for quite some time...I view it as community service.

Animats 12y ago

Another anonymous review system. What could possibly go wrong? If there's anything we've learned by now about "crowdsourced" review systems, it's that, without an elaborate way to evaluate reviewers, they fail. Badly. Facebook "likes", Google "+1", and Yelp reviews are heavily spammed. This just does not work.

jedbrown 12y ago

I take the opposite approach: I sign my reviews and have ever since I was a grad student. It compels me to do a better job reviewing and often leads to further discussion and sometimes collaboration with the authors. Some people don't like what I have to say, but by and large, they respect it. For many topics that I review for, the authors will have a pretty good idea that I wrote the review unless I am intentionally vague. I would rather write clearly and directly and stand behind it. It is a professional risk that I don't think anyone should be compelled to take, but I think signing reviews is generally good for science.

derf_ 12y ago
Yes, I think anonymity in reviews is largely a myth, even in "double blind" reviews. For many topics, there are only a few research groups actively pursuing them, and you can often tell from what other work they cite, what datasets they use, etc., which lab it is. So identifying at least some of the authors is relatively easy. You might not get the exact grad student right, but you probably know their advisor. Identifying reviewers is a little harder, but it's usually not difficult to make educated guesses unless they are, as you put it intentionally vague. Otherwise the other comments on this article about bad reviews from 'people from "a different camp" simply disliking a given approach' and the like would be impossible to justify.

dougmccune 12y ago

The actual complaint filed shows pretty nasty behavior. Here's the full complaint: https://retractionwatch.files.wordpress.com/2014/10/filed-complaint.pdf https://retractionwatch.files.wordpress.com/2014/10/filed-co... Someone very clearly was trying to get Dr. Sarkar fired. The PubPeer comments are only one small aspect. But the person then also (I assume anonymously) emailed his new employer to allege fraud, and then went so far as to print out these allegations and stamped them with official looking nonsense about being from the an NIH investigation (which didn't exist), and distributed them throughout Sarkar's department mail boxes. The complaint makes a decent case for why they think pretty much all the negativity directed at this researcher is likely from one angry person. They obviously can't prove that multiple anonymous comments are from the same person, but reading the content it certainly seems likely. What is assumed to be the same person then took things way past the line of what most of us would consider ethical. There may very well be a place for anonymous calling out of potential research misconduct. But making fraud allegations anonymously online, then printing out those comments, trying to fake them to look like an official government inquiry, and physically delivering them to the researcher's boss at his place of work isn't the way to do that. This case might be one of those cases where the anonymity should be protected at all costs out of principle, but it's a really shitty case to wave your "we're the good guys" flag for.

tokenadult 12y ago
A complaint is just a pleading before any trial of the facts has occurred. The statements about what may have happened (or may not have happened) are under the heading "FACTUAL ALLEGATIONS" in the document you kindly linked. So to describe the same issues you describe, I would write, "The actual complaint alleges some pretty nasty behavior." We have yet to see how many of those allegations will be believed by a trier of fact at trial. The plausible evidence of persistent research misconduct in this case seems to be a reasonable basis for someone not to get a new job with more research responsibilities, so there may not be any actual harm to the plaintiff here. The defense of truth is strong in a defamation case.
- Natsu 12y ago
  Not sure why this was downmodded, but I tried to correct that. I do note that people have a bias to believe the first story they hear and it's hard to consciously correct for this. Merely knowing about the bias is not enough to correct it and I realize that I'm not immune to it myself. As far as I know, I don't know any of the people involved in this, but I've read enough legal complaints to know that they're not always as factual as they first appear, so I try to consciously restrain myself from simply believing it without hearing the rebuttal. Rushing to judgement may be quite common these days, but it's not the sort of behavior fair or intelligent people should engage in. It's weird that they lead in with the first few comments, which were tame enough that I discount them completely as being an appropriate subject for a lawsuit, though they seem to get progressively worse and some of the later allegations are much more serious. I do wonder why they lead with their weakest cards, though. That aside, it does seem that the site it exercising editorial control over the comments, which may work against it with respect to the CDA, even though it's trying to do the right thing based on its policy as quoted in the complaint. Given the policy, I wonder how the later comments ever went through, as it seemed like the site wanted to avoid drama like this entirely. Even if this was a lapse of some kind, it might work against them. As I have not heard the rebuttal, I will refrain from making any judgements one way or another about who is right here. It's only fair, after all. EDIT: Retraction Watch is covering this case. I believe them to be a reliable, respected and neutral 3rd party: http://retractionwatch.com/2014/12/10/pubpeer-files-motion-dismiss-sarkar-defamation-case/ http://retractionwatch.com/2014/12/10/pubpeer-files-motion-d...

ylem 12y ago

This is an interesting problem. On the one hand, there are merits to a referee being anonymous to the submitter. Part of this may be to avoid reprisal for younger referees, but even for more established referees, you may be freer to comment if you are anonymous to the author. BUT, you are not completely anonymous. Hopefully (though there have been some recent scandals related to this) an editor of the journal knows your work and has chosen to use you as a referee based on that. This can help to keep down some of the noise that another poster (lisper) mentioned. Also, the fact that the editor knows who you are may provide some constraint on how you may phrase a review as compared to if you were completely anonymous. I don't think this is something one should issue a lawsuit over--but I also don't think that their proposal of completely anonymous review is at all useful.

kazinator 12y ago

> A prominent cancer scientist, unhappy with the attention his research papers have received on PubPeer, is suing some of our anonymous commenters for defamation On the other hand, should anonymous commenters have the balance of power: in other words, say whatever they want with impunity, even if it actually is defamatory? (Not saying that is the case in this situation, but in general). The problem is that defamation is a legal concept, which can only be tried legally. So for instance, whereas a site can rigorously enforce rules which say that all comments are directed at the research material, and not at persons, and have a factual basis in that material, those measures cannot take away the right of someone, who feels they have been defamed, to take the matter to court (where they will almost certainly lose, which is neither here nor there). You can't just create a site and declare it above the law, so to speak. The only way to protect the identities of the anonymous is for the site to take responsibility for the statements it publishes: to assert that the anonymous statements are subject to rigorous standards of review, and when published, they in fact reflect the views of PubPeer, and PubPeer alone, and not of any anonymous persons (who do not publish any statements, but act only as sources of information). Then if someone feels they have been the target of defamation, the defendant shall be PubPeer.

jessriedel 12y ago
This situation is reminiscent of websites hosting copyrighted material posted by anonymous users. Does the relevant law (DMCA, I think) say anything about defamation? A reasonable compromise might be that websites are held harmless so long as they respond to official defamation complaints in some prescribed manner. (Of course, this might not work as well because defamation is significantly less clear-cut than copyright violations, which already has plenty of cases of over-reach due to disputes over fair use, etc.)
- joshuacc 12y ago
  I think you may mean "prescribe" rather than "proscribe." :-)
  - jessriedel 12y ago

zmanian 12y ago

It seems plausible to imagine a system where PeerPub could retain their "published author to register This model relies on t a blind cryptographic signature.(https://en.wikipedia.org/wiki/Blind_signature https://en.wikipedia.org/wiki/Blind_signature) PeerPub generates a public/private key pair. 1. Alice wishes to register on PeerPub. She generates a Nonce N. 2. She blinds the nonce with a factor B. 3. She submits the blinded nonce and her identity to PeerPub. PeerPub checks her credentials and executes a blind signature of the number and returns it to Alice. 4.Alice now separately registers an account perhaps using a privacy protecting system like Tor. She uses the original N and the unblinded Signature. 5. PeerPub verifies N + signature and registers the account. PeerPub will have no way of linking N to the original credentials. PeerPub can record N and make sure it can only be used once to register an account.

sweis 12y ago
Not specific to PeerPub, but I helped work on a prototype of a federated login system based on OpenID that used blind signatures in a similar fashion: http://private-idp.appspot.com/ http://private-idp.appspot.com/ https://code.google.com/p/pseudoid/ https://code.google.com/p/pseudoid/ http://blind-signer.appspot.com/ http://blind-signer.appspot.com/ I haven't touched it for years and it looks like the login flow is broken now with the demo sites. Also, this uses unsafe server-side JS crypto for demo purposes only.
- zmanian 12y ago
  I emailed the PeerPub people about something like this. Will see if they bite.

pizza_boy 12y ago

With Publons.com (https://publons.com https://publons.com) we have different philosophy: the more transparency we can bring to the review process, the better. At the same time we recognise that both blind and double-blind peer review play an important role in generating quality research. Our approach is to focus on turning review of all kinds (including both pre- and post-publication) into a measurable research output -- something you can add to your resume. We support both anonymous and signed review with the idea that it will lead to greater transparency in the long run and also motivate reviewers to contribute more. We have a significant number of both types of review now and are starting to look ways to measure if there are significant differences between blind and open review.

efangs 12y ago
Hi, I like the site and idea. I was wondering if you were thinking of taking it one step further, however, and making it an actual pre-print server? I would think that if you expect people to take the "credit for reviewing" model seriously, then likewise they should take seriously the notion of "credit for being reviewed" or "credit for being upvoted". Also, there would have to be a way to verify submitters/reviewers credentials (e.g. connection to a research institution). Obvious arXiv has a model for how this could be done.

weissadam 12y ago

My understanding is that most peer review systems in place at various journals and funding agencies today are already anonymous (except for when people are identified by their well known viewpoints.) If you ask me, the real problem can't be solved with communications technology, the real problem must be solved at the source: The funding agencies need to take the importance of reproduction of results seriously and require their grantees to do a certain amount of rote reproduction work in order to qualify for grants for novel research. Will it slow the pace of things down? Certainly. Will it increase the quality of the science? Certainly.

esbio 12y ago

As a person who did research until a few years ago, I must say that the problem is the exact opposite. When you send an article to a journal, the paper gets reviewed by a number of peers, which send their comments back to the Editor on the appropriateness of the claimed work. The problem with this mechanism is that reviewers have no liability, because their comment is anonymous to the author and won't be available to the readers, as it won't be published as part of the article. The result is that reviewers are not made accountable now or in the future for inaccuracies in their review, blatant attacks, or tactical requests for additional irrelevant investigation just out of spite or to stall you so that they can scoop your paper. Occasionally, the Editor can step in and disregard a particularly obnoxious reviewer, but it depends on the editor, the journal, and the political/scientific strength of the reviewer.

seanmcdirmid 12y ago
> because their comment is anonymous to the author and won't be available to the readers You can make comments on your papers available; there is nothing against that. I really think, however, peer review comments should actually be published with rejected and accepted papers (at the author's consent, not reviewers) so that conferences can be more transparent.
- esbio 12y ago
  You can, but 1) there's no official channel 2) nobody will do it if they are not forced.

nemoniac 12y ago

What the scientific peer review needs is not anonymity, but accountability. Authors put their reputation on the line. Let reviewers put their reputation on the line too. Good reviews and good reviewers need to be appreciated, not anonymized. Let valued scientific reviewers gain reputation in a similar way to how contributors to HN, SE and other sites do. There could even be a viable business model in this. Publishers, be creative!

ignostic 12y ago

What we need here is a more nuanced approach. Anonymity can solve some problems in research, but it will make other worse. When people are anonymous, they ARE more likely to be truthful in their criticism. They have less incentive to hold back, and it's just human nature to tone down critical feedback when you're critiquing the work of someone with is either influential or an acquaintance. No one likes to make enemies. On the other hand, anonymity can pretty clearly bring out some of the worst in us. Some people feel little obligation to be fair or honest when their reputation isn't on the line, and so you see people trying to knock down rivals, people they don't like, or random strangers just for the "thrill of the troll." Imagine if every time you applied for a job your potential employer had access to anonymous feedback on your past work. Some of it might be fair and honest (whether positive or negative), but some of it might be lies from an anonymous coworker with a grudge. Maybe someone is trying to take you down a rung because you got the promotion over them. You could be penalized for any petty reason, and it would stick with you. Anonymous feedback communicated publicly is much the same. It holds the reviewer and the object of review on unequal footing. Anonymous feedback would be great for an author or even an editor, but it's just not fair to allow the pettiest of people to attack the works of others while wearing a mask. I'd like a system that helps researchers and others invested in the work to solicit anonymous feedback to make the work better. Public-facing commentary, on the other hand, should be tied to an identity.

pc2g4d 12y ago

Much of the discussion here seems to have become "anonymity vs. non-anonymity" --- i.e. either anonymity is good for research or it's bad for research. Why not just accept that in some venues there will be anonymity, and in some not, and let them each develop according to the merits of the respective approaches? No need to have all research anonymous or all research clearly attributed to a public identity.

kgarten 12y ago

I don't get their stance ... peer-review is already "anonymous". There are also a lot of issues with truly anonymous user forums (see 2chan and 4chan). For me it always seems as if the social aspects of communication disappear when one is truly anonymous, e.g. hate speech. The problem is not lack of anonymity but lack of incentives for reviews (I don't get anything from doing a thorough review of a paper and often it's hard to impossible to judge the contribution without dataset and code). It seems peerpub and similar systems will attract people who have the incentive to attack specific authors (as it happened in this case).

colechristensen 12y ago

Let's be sure to point out some terrible journalism, reading the following makes me completely disinterested in finishing the article and have considerably less respect for Wired. > Have you ever questioned the claims that scientists make? For example, last year’s discovery of the so-called “God particle,” Using the nickname for the Higgs in the context of questioning science claim is nothing but bait for the foolish and misinformed, and in this way anyone scientifically literate should seriously question any claims or opinions in this publication.

LiweiZ 12y ago

I don't think push some form of crowd intelligence would change much. I guess the best thing we could do is to create some conditions to let new nodes emerge and complete. I also believe better funding distribution mechanism/system will help a lot.

atsaloli 12y ago

Dr. Mark Burgess just blogged "Why I stopped caring about peer review, and learned to love the work". http://markburgess.org/blog_peer.html http://markburgess.org/blog_peer.html

analog31 12y ago

Something I've noticed is that most online articles about scientific misconduct and invalid results revolve around the medical sciences. Granted, medicine is probably the biggest piece of the scientific pie right now, and other branches of science may have their own problems, but I think it misleads the public to have a title like "scientific peer review is broken," for an article that focuses exclusively on one branch of science. Disclaimer: I'm a physicist. I'm sure that physics is not beyond critique, but unless somebody is willing to take enough of an interest in the inner workings of physics research to say something specific about it or credibly include it in a generalization, I'd rather say something like "medical peer review is broken."

oldsciemun333 12y ago

A thought suggestion: 1.) why this is important? THE GREAT CHINESE FAMINE - Mao's Hungry Ghosts book. not against mao or china. Belief in Lysenko - aka "we will grow oranges at the North Pole." will the great famine come again due to BAD SCIENCE to the USA? GMO crops / invasive pests / scientific 'POLITICS.' 2.) Of course will the science review in Japan, the plans are now to DOUBLE THE NUCLEAR POWER plants, after Fukushima 'incident.' 3.) So, as an average citizen of the USA, sometime reading of HN, I volunteer as 'anonymous submitter'.' 4.) I've noticed WORLD SCIENTISTS. that your code base is mostly broken. Even engineering codes use Matlab - What in the blazes??? and compiler tricks. Obviously, the scientists have NOT heard of Amazon, haskell or even of open source software. 12.) Basic focus folks like me. yes, how many pushups did you do today? even TRIPLE BLIND - see 'scientific method.' That means I as a VOLUNTEER submit the paper. GO AHEAD AND ATTACK 'the paper' of my repuation. I do NOT plan to be a professor at a majolr unversity. for the record, dropout grad school. and yes, I am still completing my studies when I am a bit older. Alas, I am not hopeful. 14.) first it was religion, Catholic Church and the inquisition 15.)then it was 'the great leader' like mao, etc or the suddenly the ROBBER BARRONS like Tom Editson (high school dropout) becoming sudden geniuses. 16..) now its, the game let's get the professor fired by going direcctly to the Board of Trustees of California State U or others. 17.) when in doubt use the 'federal agencies' to murky up the waters. 18.) Tobacco is good for you and healthy (when in moderation) some say. 19.)MY LAST SUGGESTION. the paper be chopped into parts. I don't want to see the strange science or arcane languages. I only want some of the sampled data to SHOOT HOLES THROUGH using python and haskell. the book is forthcoming - how to lie subtlely with Statistics and meta heuristics... THIS IS THE KEY WEAKNESS OF PEER review. Who is really a peer? and most are working at the same college and same area. want proof? See OpenSSH - and apple goto goto fail.

hyperion2010 12y ago

I usually think that peer review is broken, but that is because we almost always miss the point of what the actual purpose of peer review is for. Peer review is NOT for laymen. A scientific paper is part of a much longer process of discovery which includes massive amounts of debate and argument. Papers always reflect they very edge of a field where literally the only people who can judge whether the results are plausible are peers because they are the ones who are also trying to make sense of the same phenomenon. Over time some primary research papers are singled out because they really are excellent explanations of a phenomenon most of the time however you have to go look at a review paper. Furthermore peer review is absolutely NOT about reproducibility, it is a prerequisite that says that it MIGHT be worth trying to reproduce this work or work contingent on these results being valid. There are so many reasons why a result might be wrong that it often takes fields years to figure it out and the published literature is a record of that.

daemonk 12y ago

Anonymity is hard in practice when so much of science consist of small niche fields where it can be pretty obvious who the author of a paper is by just reading the content. A better solution might be to enforce non-anonymous peer-reviews that can be read by the public after the paper has been published.

tnhh 12y ago
Some venues are doing this, e.g. f1000. Here's a paper that I reviewed: http://f1000research.com/articles/3-38/v2 http://f1000research.com/articles/3-38/v2

sideshowb 12y ago

Not as deep an article as I'd hoped, but it points to a difficult question. As reviewers of scientific papers have unprecedented power over what research gets published, who reviews the reviewers? (A personal example - I have just spent a week constructing a counterargument to a reviewer who didn't read my paper properly. Imagine having an online discussion where your career success hangs on the response of an anonymous, disengaged flamer to a single post of yours: can you imagine how much effort you put into that post? Sadly this particular argument is pointless and serves nobody; it would be better use of my time to get on with actual research, but that's not how the system works).

p4bl0 12y ago

Peer review might be broken, but it's not anonymity that will fix it. For the simple reason that we already have double-blind reviews in many fields and the peer reviewing process is not any less broken there. There are people who are thinking of multiple solutions, one that I think is interesting is the proposition of Open Scholar, dubbed "Independant Peer-Review". I submitted it to HN so we can discuss it without flooding this thread, if it interests HNers: https://news.ycombinator.com/item?id=8734271 https://news.ycombinator.com/item?id=8734271. In there scheme, the papers are open access from the beginning, the peer reviewing process is open, and reviews are citable. I believe it would encourage better and deeper reviews. It would also be very nice for young scientists or students who want to apply for a PhD grant to be able to show that they are able to write a comprehensive review of a paper in their field, and that their review was good enough to significantly improve the paper or to be selected by the authors and/or publishers to be released alongside the paper, for instance.

return0 12y ago

Pubpeer is not a peer review site, it's site with comments. I think they are doing it wrong by promoting it as "anonymous peer review", because by definition if its anonymous it can't be "peers". It should (continue to) be a companion to published papers that every once in a while does the service of spotting an error.