12 ms·
The fate of published peer-reviewed articles, submitted again (1982)
- waitwhatwhoa 13y agoCertainly not a new problem (this paper is from 1982, so old that my University subscription doesn't cover it). This might be a better link to start such a discussion: http://jrs.sagepub.com/content/99/4/178.long http://jrs.sagepub.com/content/99/4/178.long
- deleted 13y ago[deleted]
- richardlblair 13y agoI find it kind of funny that an article criticizing the current review practices is not free and openly available.
- gcr 13y agoOf course it is. You just have to know where to look: https://www.google.com/search?q=Richard+Smith+peer+review+filetype%3Apdf https://www.google.com/search?q=Richard+Smith+peer+review+fi... Here's the .pdf: http://www2.psych.ubc.ca/~schaller/349and449Readings/Smith2006.pdf http://www2.psych.ubc.ca/~schaller/349and449Readings/Smith20... In my field, it's common (and even encouraged) for authors to post their work to their own website. When authors give up their copyright to the IEEE, they explicitly retain the right to redistribute it on their own personal web sites.. From the text of the CVPR IEEE copyright agreement: Author Online Use 6. Personal Servers. Authors and/or their employers shall have the right to post the accepted version of IEEE-copyrighted articles on their own personal servers or the servers of their institutions or employers without permission from IEEE, provided that the posted version includes a prominently displayed IEEE copyright notice and, when published, a full citation to the original IEEE publication, including a link to the article abstract in IEEE Xplore. Authors shall not post the final, published versions of their papers. (the last sentence refers to the "camera-ready" version, which ideally only contains eg. typo fixes relative to the accepted version)
- notahacker 13y agoIt's a different article though, which only briefly summarises the original post. Worth a read though, for the author's recollection of his (unsuccessful) attempts to reject a Karl Popper paper he thought was substandard.
- leephillips 13y agoOpen publication and review practices are nearly orthogonal issues.
- piokuc 13y agoAt least one aspect of the peer review process should be better now than in 1982, when the paper was published: resubmission detection. Detecting plagiarism seems easy these days, just google for "detect plagiarism". But are reviewers actually using these tools? It would be interesting to repeat the resubmission experiment and see the results.
- edtechdev 13y agoHard to detect plagiarism when the original article is behind a paywall of another publisher
- piokuc 13y agoTrue, to some extend, but see the comment by gcr: https://news.ycombinator.com/item?id=5781899 https://news.ycombinator.com/item?id=5781899
- gcr 13y agoMany publishers allow Google to crawl the full text of the protected resources. This is why Google Scholar works, for example. Copy+pasting the abstract into Google Scholar is surprisingly effective at finding plagiarism.
- gcr 13y agoAbsolutely. Every reviewer that I've met does plagiarism detection on their own. My advisor has even found instances where the author copy+pasted articles from the likes of the Wikipedia article on the subject or 'about.com'; such papers get desk-rejected without review, and in some cases, the author is forbidden from submitting to IEEE conferences for the rest of their career. That's how important plagiarism is. Just copy+pasting the abstract into Google is often enough to find it. At least for computer science conferences, the conference organizers run the papers through additional plagiarism screening tools before publishing, but it's mostly the reviewers' job.
- leephillips 13y ago"Every reviewer that I've met does plagiarism detection on their own." I don't think it's a bad idea, but it never occurred to me to do this. If you're familiar with the field in which you are reviewing papers, you'll know if something is not new, in which case it is rejected, quite aside from the possibility of the author having lifted some actual text.
- edtechdev 13y agoHere's hoping open, post-publication review catches on: http://futureofscipub.wordpress.com/open-post-publication-peer-review/ http://futureofscipub.wordpress.com/open-post-publication-pe... http://www.frontiersin.org/Computational_Neuroscience/10.3389/fncom.2012.00063/ful http://www.frontiersin.org/Computational_Neuroscience/10.338... http://retractionwatch.wordpress.com/2013/03/22/brian-deers-modest-proposal-for-post-publication-peer-review/ http://retractionwatch.wordpress.com/2013/03/22/brian-deers-...
- gcr 13y agoIn academia, articles already have a form of post-publication review: citation count. If you don't do good work, you won't get any citations, though the converse isn't always true. Professors usually ask their grad students to read the most important and influential papers in the field. When those grad students then turn around and start composing papers of their own, they'll refer to the ones they read, creating a kind of circular feedback cycle. Thus, if you want to find a field's best work, it often suffices to look for highly cited papers first. It's like Reddit's karma voting system, just a lot slower. :)
- pessimizer 13y ago>If you don't do good work, you won't get any citations A lot of terrible, and even retracted papers have a high citation count, though.
- gcr 13y agoSure, just like how a lot of terrible and uninteresting articles on Reddit have a high karma count. By and large, both systems do a fairly good job at sorting out the interesting from the uninteresting, though there are always mistakes
- notahacker 13y agoTrue, but citations are qualitative as well as quantitative, so at least researchers should know that Controversial Paper X mostly gets citations as an example of "unusual findings, possibly due to methodological shortcomings" if they are deciding where it's worth summarising in the literature review section of their paper. And arguably (especially in social sciences) papers which provoke interesting rebuttals them are deservedly more influential than papers which are esoteric, trivial or dull.
- bjornsing 13y agoInteresting article, albeit with serious methodological flaws. EDIT: Oops, I didn't see Cornell there, I take it back.
- helmut_hed 13y agoYour edit is itself very interesting :) Did the Cornell affiliation of one of the authors cause you to change your opinion in a positive or a negative way, and why?
- bjornsing 13y agoOnly joking. :) The joke being something along the lines of "look - I can peer review papers too - that wasn't so hard". Buy the way, I'd love to see this research taken one step further, in a humoristic direction: a full statistical analysis comparing the predictive power of academic affiliation in determining acceptance for publication, to that of the contents of the paper itself. :P
- T-A 13y agoTwelve years later, Caltech provost David Goodstein wrote http://www.its.caltech.edu/~dg/crunch_art.html http://www.its.caltech.edu/~dg/crunch_art.html Skip down to "honesty and ethical behavior" for his remarks on peer review.
- MrQuincle 13y agoThere's just one solution. Make the reviews public. All academia I encounter are against it. So, it's probably the right way to go. :-)
- sampo 13y ago> Make the reviews public At least The European Geosciences Union publishes 15 academic journals that do exactly that. (They're also Open Access, and CC-licensed.) http://www.egu.eu/publications/open-access-journals/ http://www.egu.eu/publications/open-access-journals/ Some on them, like Atmospheric Chemistry and Physics, are also pretty highly regarded.
- leephillips 13y agoThe other popularly suggested solution is sort-of the opposite: use double-blind reviewing, where the reviewer doesn't know who the authors are, at least formally.
- kenko 13y agoThis is the standard in most disciplines. I wish the authors of this article had done the same test with journals that do blind reviewing. I'd bet they still got a lot of rejections of previously accepted articles for bizarre reasons, since in general acceptance depends a lot on the particular reviewers you draw.
- leephillips 13y ago"This is the standard in most disciplines." Any citation for this? I'm curious, because in physics, although this is often suggested, it has not been generally adopted.
- kenko 13y agoReally? I guess I should've said "this is standard in most disciplines ... that I know about". I'm really surprised that it isn't done in physics!
- deleted 13y ago[deleted]
- beloch 13y agoTwo of my biggest gripes with the peer-review process would be relatively easy to address: 1. Time 2. One-way communication 1. You sometimes have to waste a lot of time to get articles published. The way the current process works, it is very easy for referees who are either competitors, incompetent, or simply assholes to effectively filibuster the review process. Even when things go smoothly the process is often agonizingly slow. 2. The peer-review process operates under the obsolete limitation of snail-mail correspondence. There is no two-way communication between the referees. Referees often return either unprofessional or obviously wrong (to everyone but the editor!) comments. If the referees could talk to each other this would be sorted out quickly. It doesn't help that editors are typically very reluctant to reconsider rejection verdicts even when they are based on clear cases of referee error. How would I address this? Use the #$%@!ing internet people! Journals should abandon this one-way snail-mail process and build a private message board for each paper being reviewed. Referees are invited to post their comments anonymously on the forum. Other referees and, ideally, the authors could then reply to those comments. After a reasonable time period has elapsed, perhaps a couple of weeks, the editor should take stock of what valid concerns remain and reach a verdict.
- leephillips 13y agoSnail mail? I haven't seen this as part of the reviewing process in decades. What field and country are you talking about?
- beloch 13y agoThey use email, but I meant that the process hasn't really adapted to take advantage newer technology than snail-mail.
- kenko 13y agoHow so? The one time I was a referee I got the paper by downloading a PDF and sent in my comments by entering them into a form. That technology is newer than snail mail.
- tokenadult 13y agoThis is an early paper by Stephen Ceci, and illustrates the approach Ceci takes of challenging the assumptions that other psychologists assume to be true. This is why I love reading books and articles by Ceci. His Web-based CV http://far.human.cornell.edu/FAR/uploads/webcv/sjc9_webcv.pdf http://far.human.cornell.edu/FAR/uploads/webcv/sjc9_webcv.pd... lists his extensive publications (you hope that most of those got better peer review after his 1982 paper than psychology papers used to). Google Scholar http://scholar.google.com/scholar?hl=en&q=Stephen+Ceci&btnG=&as_sdt=1%2C24&as_sdtp= http://scholar.google.com/scholar?hl=en&q=Stephen+Ceci&#... helpfully lists some papers by Ceci for which there are free full-text versions that you or I can download for our learning and reading pleasure. From Jelte Wicherts (another psychologist skeptical about peer review in psychology) writing in Frontiers of Computational Neuroscience (an open-access journal) comes a much more recent set of general suggestions Jelte M. Wicherts, Rogier A. Kievit, Marjan Bakker and Denny Borsboom. Letting the daylight in: reviewing the reviewers and other ways to maximize transparency in science. Front. Comput. Neurosci., 03 April 2012 doi: 10.3389/fncom.2012.00020 http://www.frontiersin.org/Computational_Neuroscience/10.3389/fncom.2012.00020/full http://www.frontiersin.org/Computational_Neuroscience/10.338... on how to make the peer-review process in scientific publishing more reliable. Wicherts does a lot of research on this issue to try to reduce the number of dubious publications in his main discipline, the psychology of human intelligence. "With the emergence of online publishing, opportunities to maximize transparency of scientific research have grown considerably. However, these possibilities are still only marginally used. We argue for the implementation of (1) peer-reviewed peer review, (2) transparent editorial hierarchies, and (3) online data publication. First, peer-reviewed peer review entails a community-wide review system in which reviews are published online and rated by peers. This ensures accountability of reviewers, thereby increasing academic quality of reviews. Second, reviewers who write many highly regarded reviews may move to higher editorial positions. Third, online publication of data ensures the possibility of independent verification of inferential claims in published papers. This counters statistical errors and overly positive reporting of statistical results. We illustrate the benefits of these strategies by discussing an example in which the classical publication system has gone awry, namely controversial IQ research. We argue that this case would have likely been avoided using more transparent publication practices. We argue that the proposed system leads to better reviews, meritocratic editorial hierarchies, and a higher degree of replicability of statistical analyses."