8 ms·

The Future of Peer Review

36 points by RichardPrice 15y ago

billswift 15y ago
For an example of lousy peer review, I posted this yesterday on LessWrong: A new science journal recently published a seriously crackpot paper, this has the abstract a link for the pdf. I first heard about it from Derek Lowe, who has also written two follow-up posts. The first has a couple of links discussing how news of the paper spread, while the second includes a link to the journal making excuses for why they published it. <blockquote> Moreover, members of the Editorial Board have objected to these papers; some have resigned, and others have questioned the scientific validity of the contributions. In response I want to first state some basic facts regarding all publications in this journal. All papers are peer-reviewed, although it is often difficult to obtain expert reviewers for some of the interdisciplinary topics covered by this journal. I feel obliged to stress that although we will strive to guarantee the scientific standard of the papers published in this journal, all the responsibility for the ideas contained in the published articles rests entirely on their authors. </blockquote> I included the links to all of Derek Lowe's posts because they have other interesting links, including in the comments. The permalink is here http://lesswrong.com/lw/9p9/open_thread_february_114_2012/5trq http://lesswrong.com/lw/9p9/open_thread_february_114_2012/5t... , so you can check out the links I included.
- RichardPrice 15y ago
  Fascinating example of peer review not working out. It reminds me of the Sokal hoax, except that this one doesn't look like a hoax.
apu 15y ago
I'm all for new publishing models in academia, but this post doesn't address the fundamental difference between the web in general and academic papers: expert curation. The web's "curation" works (to whatever degree it does) because there are thousands, if not millions of people who can rate the quality of most articles/essays/websites. But in many scientific fields, there are often only a very small number of people who would be qualified to review a particular paper (usually well under a hundred, sometimes as few as a handful). And make no mistake, this is a numbers game -- there's the well known 100/10/1 rule for content websites (for every 100 viewers, there are 10 who rate content, and only 1 who creates content). The exact numbers vary quite a bit from website to website, but the point is that it's orders of magnitude difference. That's all fine if your initial pool of people is huge, but if you start with 20 qualified people...you see the issue. Another related point not addressed is that peer review is fairly demanding work. You're not just saying "accept" or "reject", but you're obliged to write a page or two suggesting the strong and weak points of the paper, things to improve, etc. You need some mechanism for enforcing this (even if it's via social norms), otherwise review quality deteriorates quite severely, which in turn affects paper quality. In lieu of this article, I'd recommend seeing lectures by (or essays by) Michael Nielsen, who's thought about this problem in much more detail and (I believe) has a much better understanding of the way things work. http://michaelnielsen.org/blog/essays/ http://michaelnielsen.org/blog/essays/
- RichardPrice 15y ago
  Thanks for your detailed thoughts. Peer review is already evolving. Increasingly academics discover their papers via search engines (Google, Google Scholar), and social platforms (blogs, email, arXiv, Academia.edu, Twitter). The usage of these channels as discovery channels for research shows that the metrics that drive those channels are doing a good job at expert curation - i.e. surfacing good content. On Google Scholar, it's citation counts that drive the ranking, and therefore the discovery process. This is the "crowd review" process I talked about in the post. With social platforms, it's recommendations from friends and colleagues that drive the discovery process. This is the "social review" process I talked about in the post. In the pre-web days, people used to walk down to their libraries to read the latest edition from a particular journal. In those days, the journals did drive a lot of the discovery process. Nowadays, pretty much all discovery of research papers happens on the web, and most of the discovery is via channels such as search, and recommendations from colleagues over social & communication platforms. You are right to think about the incentive systems that drive different kinds of peer review. In the case of the social review process, you want to share good recommendations with your friends, not bad ones (this is why Twitter and Facebook work as discovery channels). In the case of the crowd review process, you want to cite good works that your work genuinely built on, not bad work.
  - flom 15y ago
    My understanding is that most publications are not heavily cited, and address very specific problems. I'm curious as to how you plan on creating incentives for qualified people to review the papers in their field which focus on problems outside of their present research interests. To give a personal example: I used to work in a research group that applied techniques of a field called geometric control theory to the problem of modeling flocks of birds. There are not that many geometric control theorists in the world, and their interests range from robotics, to circuits, to modelings schools of fish. Right now, there are journals of control theory that force their reviewers to review all submissions, regardless of their personal interests. How do you plan on emulating that? In other words, there need to be incentives for geometric control theorists who work on modeling schools of fish to spend a few days of their time grinding through the proofs of some grad student's paper in modeling bird flocks, which he needs for his thesis.
jfmercer 15y ago
Am I the only one that finds it ironic that TechCrunch is writing about peer review?
wybo 15y ago
Also see the now finished LiquidPub project by the university of Trento, Springer, and others, for more ideas and views on peer review in the digital age: http://liquidpub.org/ http://liquidpub.org/ This paper about journals: http://wiki.liquidpub.org/mediawiki/upload/9/9b/Liquid-journal-proposal_v0.13.pdf http://wiki.liquidpub.org/mediawiki/upload/9/9b/Liquid-journ... (Liquid Journals) And this paper about peer review: http://ubiquity.acm.org/article.cfm?id=1226695 http://ubiquity.acm.org/article.cfm?id=1226695 (Publish and Perish) Though in a way the differences between peoples descriptions and views are details mostly, as what is clear is that peer review will change, and that it will move away from publisher-control. Imho the biggest difficulty will be the social aspect; breaking the feedback-loop between journal-publications and perceived academic credit. Because until individual scholars can escape from the social trap of journal-publishing & handing over their copyrights, without harming their careers, they will be unlikely to do so. (http://en.wikipedia.org/wiki/Social_trap http://en.wikipedia.org/wiki/Social_trap, and a great article on social traps: http://www.agls.uidaho.edu/agec467/agec467/Other%20readings/Platt.PDF http://www.agls.uidaho.edu/agec467/agec467/Other%20readings/...)
- RichardPrice 15y ago
  Wybo - thanks for these links. I totally agree that breaking the feedback loop you mentioned, that binds academics to the current journal system, is key. I think the central things that needs to happen are: building an alternative discovery system (already happening with the web), and building an alternative credit system (happening with citation counts, and other metrics, but is lagging behind the growth in alternative discovery systems, and needs to be accelerated).
  - femto 15y ago
    I wonder if a distributed algorithms, as used in web based currencies (eg. bitcoin) or incentive based systems (eg. bit torrent), can be adapted to the peer review system? The currency being traded would be academic credit.
vladoh 15y ago
I think it is not possible to achieve the quality of a peer review by "crowd review" or "social review". While most of the people are able to rate a funny picture or an interesting article on Internet, the review of an academic paper requires much more expertise in the field and knowledge of the related work. Therefore I don't believe that a truly objective rating can be created. Also, making a peer review is time consuming because you have to really understand the paper in detail and think about if the contributions are big enough and well explained, if the experiments are done properly and if the comparisons are fare and so on. This is not an easy thing to achieve by just throwing a lot of people on it (the users of the Internet). Furthermore, when a paper is published in a well recognized journal or conference one has a guarantee of the quality of the paper. For example when I see something published at CVPR (Conference on Computer Vision and Pattern Recognition) I know it is certainly worth reading. I don't want to imagine how much more difficult it will be to find the valuable papers among all the bad stuff that is being submitted for review and rejected. I don't want to say that the peer review process is perfect, but if I have to choose between "peer review" and "crowd review" I choose the former.
- RichardPrice 15y ago
  Academics have always based reading decisions on recommendations from colleagues. This trend has accelerated since email and the web were invented, as your colleagues can communicate with you through more channels, and more recommendations go back and forth. Social review always has been a big part of research discovery, and it's become more important with the advent of the web. Search engines play a huge role in driving research discovery, and their ranking systems are fundamentally based on crowd review - calculations using links, as in the case of Google and Bing, or citations, as in the case of Google Scholar. One might wonder in theory whether a search engine could do a good job of ranking academic papers, or whether recommendations from one's colleagues is a viable way of discovering papers. But when you look, in practice, into how academics operate day to day, those channels are the ones that academics increasingly use to discover research.
andreadallera 15y ago
The article is missing the point. The purpose of peer review is not to assure that a given publication is of good quality - it's to maintain the status quo. A system in which any undergraduate/PhD can publish his or her research without passing from a committee, or his own professor, is exactly the opposite of what people in academia are looking for.
tylerneylon 15y ago
Summary: Current peer review is not 100%; we need to think in terms of authors' motivations (tenure); try a hacker news model, possibly layered with badges-of-approval from editorial board. The quality of peer review is only as good as the editorial board and reviewers make it -- often this is a lower standard than we expect. This article lists several extreme examples of poor practices by a (previously-)prominent publisher: http://blog.mathunion.org/journals/?no_cache=1&tx_t3blog_pi1%5BblogList%5D%5BshowUid%5D=30&tx_t3blog_pi1%5BblogList%5D%5Byear%5D=2012&tx_t3blog_pi1%5BblogList%5D%5Bmonth%5D=02&tx_t3blog_pi1%5BblogList%5D%5Bday%5D=05&cHash=a2d6424f899302a7ea3b75b9bb591802 http://blog.mathunion.org/journals/?no_cache=1&tx_t3blog... The key to shifting academic quality control away from the present archaic model is through author's motivation. The replacement needs to be a system that improves and filters incoming papers, and at the same time earns at least as much prestige for the authors who participate. One idea is something like a combination hacker news and stackoverflow with academic articles in place of news articles / technical questions as the object of discussion. Papers can be changed in response to comments. High scores for either a paper or an author on such a site could replace the prestige granted by acceptance into a top-tier journal. Points are awarded for reviews, and researchers are respected for their scores (closer to tenure), so there is more positive motivation to review than in today's system (which is: it's rude not to review when asked). I think that alone could work. Some academics might balk at a lack of editorial boards. Fine, let's allow some editors in the mix. An editorial board is an aggregator. They choose what content fits their filter, and may open a discussion with an author on improvements, possibly bringing in reviewers. In the end, readers will see a prestigious badge of the board's approval on high-quality articles, and there could be a page or rss feed to browse the articles meeting this standard. This can all exist on top of the hacker news / stackoverflow system, which itself could be built on top of an arXiv-like system. Several of these ideas are motivated by thoughts from the blogs of Tim Gowers and Michael Nielsen.
- RichardPrice 15y ago
  Hi Tyler - nice comment. Also, wonderful work on www.thecostofknowledge.com! I agree that discovery and credit/prestige are the two key things to get right. I'm sure that the kinds of systems you're talking about, voting systems, will be part of the research discovery/credit process in the future. It works for web content, and also for very esoteric content on the web (MathOverflow etc), so it should work for research papers too.
knowtheory 15y ago
It's also worth noting that academia.edu is a for-profit enterprise. The domain academia.edu was purchased prior to the strictures put in place to ensure that only qualified educational institutions could register .edu domains, and has been grandfathered ever since. Academia.edu itself launched in 2008 apparently. The author of this piece very much has a horse in this game, and as a result I very much view his writing with a skeptical eye. There are serious and substantial problems with social models of publishing, especially when organizations/institutions start tying promotions and/or remuneration to metrics like impact factor (which itself is a proprietary formula). We should keep in mind to reward good science because it's good science, not just because it's popular.
- RichardPrice 15y ago
  I think you are getting the horse and the cart in the wrong order (to continue the horse analogies :) ). I started Academia.edu, after finishing my PhD at Oxford, because I believe that science, is too closed, and too slow, and that I think that we can build a faster, cheaper, and better system for disseminating research. You're wondering whether I'm asserting the things in the post because I started the company, but really it's the other way round. I started the company because I believe the philosophy expressed in the post. One has to be the change one wants to see in the world.
  - knowtheory 15y ago
    I'm sad to see that someone downvoted factual information provided about you and the post you wrote. And I'm not asserting anything about your motivations for why you started a for-profit company. I agree with you that the publishing models we have currently are broken. At the same time, I'm not entirely happy with some of the changes that have taken place in the past 5-10 years. When you look to the changes that digital distribution have wrought in industries like the news media over a similar time span, the changes are not all healthy ones either from the perspective of publishers or consumers. The fact that HuffPo and Gawker have risen to the top of the pile as the sorts of money making enterprises is not such a great example to set. Moreover, I'm unconvinced that adding more eyeballs to papers is going to improve quality of review. If your assertion is that we can make scientific review faster and cheaper without a subsequent loss of quality (let's just say "more scalable"), then I'd love to hear your response to a claim i like to make. I'd assert that publishing scientific papers is more similar to the trials and travails of online media than it is to publishing source code. The thing that makes open source software work is that software behavior (broadly speaking) is verifiable. I can run your code and we can agree on the results, or you can tell me i'm an idiot for not configuring the software appropriately. I'd assert that FOSS scalability derives from this ability. More eyeballs helps because everyone is working from/over the same artifact. The same can't be said for most academic papers, because most academes don't even publish their raw data, let alone possess have the facilities or the will to publish verifiable algorithmic processes for arriving at their data. I'd love to see a project which encouraged better replicability and verification of results/test data. But you haven't suggested any of those things. P.S. absolutely, please do be the change you're looking for. But I'm not convinced that you're the change i'm looking for.
merraksh 15y ago
The drawbacks of the Two Person review process are that it is [...] expensive: $8 billion a year is spent on subscriptions to journals, which is money that could be spent on more research. None of that money goes to the two reviewers. The reviewers could do the same work for open access journals, and they would have (IMHO) no motivation not to do it for free.

36 points by RichardPrice 15y ago

For an example of lousy peer review, I posted this yesterday on LessWrong: A new science journal recently published a seriously crackpot paper, this has the abstract a link for the pdf. I first heard about it from Derek Lowe, who has also written two follow-up posts. The first has a couple of links discussing how news of the paper spread, while the second includes a link to the journal making excuses for why they published it. <blockquote> Moreover, members of the Editorial Board have objected to these papers; some have resigned, and others have questioned the scientific validity of the contributions. In response I want to first state some basic facts regarding all publications in this journal. All papers are peer-reviewed, although it is often difficult to obtain expert reviewers for some of the interdisciplinary topics covered by this journal. I feel obliged to stress that although we will strive to guarantee the scientific standard of the papers published in this journal, all the responsibility for the ideas contained in the published articles rests entirely on their authors. </blockquote> I included the links to all of Derek Lowe's posts because they have other interesting links, including in the comments. The permalink is here http://lesswrong.com/lw/9p9/open_thread_february_114_2012/5trq http://lesswrong.com/lw/9p9/open_thread_february_114_2012/5t... , so you can check out the links I included.

RichardPrice 15y ago
Fascinating example of peer review not working out. It reminds me of the Sokal hoax, except that this one doesn't look like a hoax.

apu 15y ago

I'm all for new publishing models in academia, but this post doesn't address the fundamental difference between the web in general and academic papers: expert curation. The web's "curation" works (to whatever degree it does) because there are thousands, if not millions of people who can rate the quality of most articles/essays/websites. But in many scientific fields, there are often only a very small number of people who would be qualified to review a particular paper (usually well under a hundred, sometimes as few as a handful). And make no mistake, this is a numbers game -- there's the well known 100/10/1 rule for content websites (for every 100 viewers, there are 10 who rate content, and only 1 who creates content). The exact numbers vary quite a bit from website to website, but the point is that it's orders of magnitude difference. That's all fine if your initial pool of people is huge, but if you start with 20 qualified people...you see the issue. Another related point not addressed is that peer review is fairly demanding work. You're not just saying "accept" or "reject", but you're obliged to write a page or two suggesting the strong and weak points of the paper, things to improve, etc. You need some mechanism for enforcing this (even if it's via social norms), otherwise review quality deteriorates quite severely, which in turn affects paper quality. In lieu of this article, I'd recommend seeing lectures by (or essays by) Michael Nielsen, who's thought about this problem in much more detail and (I believe) has a much better understanding of the way things work. http://michaelnielsen.org/blog/essays/ http://michaelnielsen.org/blog/essays/

RichardPrice 15y ago
Thanks for your detailed thoughts. Peer review is already evolving. Increasingly academics discover their papers via search engines (Google, Google Scholar), and social platforms (blogs, email, arXiv, Academia.edu, Twitter). The usage of these channels as discovery channels for research shows that the metrics that drive those channels are doing a good job at expert curation - i.e. surfacing good content. On Google Scholar, it's citation counts that drive the ranking, and therefore the discovery process. This is the "crowd review" process I talked about in the post. With social platforms, it's recommendations from friends and colleagues that drive the discovery process. This is the "social review" process I talked about in the post. In the pre-web days, people used to walk down to their libraries to read the latest edition from a particular journal. In those days, the journals did drive a lot of the discovery process. Nowadays, pretty much all discovery of research papers happens on the web, and most of the discovery is via channels such as search, and recommendations from colleagues over social & communication platforms. You are right to think about the incentive systems that drive different kinds of peer review. In the case of the social review process, you want to share good recommendations with your friends, not bad ones (this is why Twitter and Facebook work as discovery channels). In the case of the crowd review process, you want to cite good works that your work genuinely built on, not bad work.
- flom 15y ago
  My understanding is that most publications are not heavily cited, and address very specific problems. I'm curious as to how you plan on creating incentives for qualified people to review the papers in their field which focus on problems outside of their present research interests. To give a personal example: I used to work in a research group that applied techniques of a field called geometric control theory to the problem of modeling flocks of birds. There are not that many geometric control theorists in the world, and their interests range from robotics, to circuits, to modelings schools of fish. Right now, there are journals of control theory that force their reviewers to review all submissions, regardless of their personal interests. How do you plan on emulating that? In other words, there need to be incentives for geometric control theorists who work on modeling schools of fish to spend a few days of their time grinding through the proofs of some grad student's paper in modeling bird flocks, which he needs for his thesis.

jfmercer 15y ago

Am I the only one that finds it ironic that TechCrunch is writing about peer review?

wybo 15y ago

Also see the now finished LiquidPub project by the university of Trento, Springer, and others, for more ideas and views on peer review in the digital age: http://liquidpub.org/ http://liquidpub.org/ This paper about journals: http://wiki.liquidpub.org/mediawiki/upload/9/9b/Liquid-journal-proposal_v0.13.pdf http://wiki.liquidpub.org/mediawiki/upload/9/9b/Liquid-journ... (Liquid Journals) And this paper about peer review: http://ubiquity.acm.org/article.cfm?id=1226695 http://ubiquity.acm.org/article.cfm?id=1226695 (Publish and Perish) Though in a way the differences between peoples descriptions and views are details mostly, as what is clear is that peer review will change, and that it will move away from publisher-control. Imho the biggest difficulty will be the social aspect; breaking the feedback-loop between journal-publications and perceived academic credit. Because until individual scholars can escape from the social trap of journal-publishing & handing over their copyrights, without harming their careers, they will be unlikely to do so. (http://en.wikipedia.org/wiki/Social_trap http://en.wikipedia.org/wiki/Social_trap, and a great article on social traps: http://www.agls.uidaho.edu/agec467/agec467/Other%20readings/Platt.PDF http://www.agls.uidaho.edu/agec467/agec467/Other%20readings/...)

RichardPrice 15y ago
Wybo - thanks for these links. I totally agree that breaking the feedback loop you mentioned, that binds academics to the current journal system, is key. I think the central things that needs to happen are: building an alternative discovery system (already happening with the web), and building an alternative credit system (happening with citation counts, and other metrics, but is lagging behind the growth in alternative discovery systems, and needs to be accelerated).
- femto 15y ago
  I wonder if a distributed algorithms, as used in web based currencies (eg. bitcoin) or incentive based systems (eg. bit torrent), can be adapted to the peer review system? The currency being traded would be academic credit.

vladoh 15y ago

I think it is not possible to achieve the quality of a peer review by "crowd review" or "social review". While most of the people are able to rate a funny picture or an interesting article on Internet, the review of an academic paper requires much more expertise in the field and knowledge of the related work. Therefore I don't believe that a truly objective rating can be created. Also, making a peer review is time consuming because you have to really understand the paper in detail and think about if the contributions are big enough and well explained, if the experiments are done properly and if the comparisons are fare and so on. This is not an easy thing to achieve by just throwing a lot of people on it (the users of the Internet). Furthermore, when a paper is published in a well recognized journal or conference one has a guarantee of the quality of the paper. For example when I see something published at CVPR (Conference on Computer Vision and Pattern Recognition) I know it is certainly worth reading. I don't want to imagine how much more difficult it will be to find the valuable papers among all the bad stuff that is being submitted for review and rejected. I don't want to say that the peer review process is perfect, but if I have to choose between "peer review" and "crowd review" I choose the former.

RichardPrice 15y ago
Academics have always based reading decisions on recommendations from colleagues. This trend has accelerated since email and the web were invented, as your colleagues can communicate with you through more channels, and more recommendations go back and forth. Social review always has been a big part of research discovery, and it's become more important with the advent of the web. Search engines play a huge role in driving research discovery, and their ranking systems are fundamentally based on crowd review - calculations using links, as in the case of Google and Bing, or citations, as in the case of Google Scholar. One might wonder in theory whether a search engine could do a good job of ranking academic papers, or whether recommendations from one's colleagues is a viable way of discovering papers. But when you look, in practice, into how academics operate day to day, those channels are the ones that academics increasingly use to discover research.

andreadallera 15y ago

The article is missing the point. The purpose of peer review is not to assure that a given publication is of good quality - it's to maintain the status quo. A system in which any undergraduate/PhD can publish his or her research without passing from a committee, or his own professor, is exactly the opposite of what people in academia are looking for.

tylerneylon 15y ago

Summary: Current peer review is not 100%; we need to think in terms of authors' motivations (tenure); try a hacker news model, possibly layered with badges-of-approval from editorial board. The quality of peer review is only as good as the editorial board and reviewers make it -- often this is a lower standard than we expect. This article lists several extreme examples of poor practices by a (previously-)prominent publisher: http://blog.mathunion.org/journals/?no_cache=1&tx_t3blog_pi1%5BblogList%5D%5BshowUid%5D=30&tx_t3blog_pi1%5BblogList%5D%5Byear%5D=2012&tx_t3blog_pi1%5BblogList%5D%5Bmonth%5D=02&tx_t3blog_pi1%5BblogList%5D%5Bday%5D=05&cHash=a2d6424f899302a7ea3b75b9bb591802 http://blog.mathunion.org/journals/?no_cache=1&tx_t3blog... The key to shifting academic quality control away from the present archaic model is through author's motivation. The replacement needs to be a system that improves and filters incoming papers, and at the same time earns at least as much prestige for the authors who participate. One idea is something like a combination hacker news and stackoverflow with academic articles in place of news articles / technical questions as the object of discussion. Papers can be changed in response to comments. High scores for either a paper or an author on such a site could replace the prestige granted by acceptance into a top-tier journal. Points are awarded for reviews, and researchers are respected for their scores (closer to tenure), so there is more positive motivation to review than in today's system (which is: it's rude not to review when asked). I think that alone could work. Some academics might balk at a lack of editorial boards. Fine, let's allow some editors in the mix. An editorial board is an aggregator. They choose what content fits their filter, and may open a discussion with an author on improvements, possibly bringing in reviewers. In the end, readers will see a prestigious badge of the board's approval on high-quality articles, and there could be a page or rss feed to browse the articles meeting this standard. This can all exist on top of the hacker news / stackoverflow system, which itself could be built on top of an arXiv-like system. Several of these ideas are motivated by thoughts from the blogs of Tim Gowers and Michael Nielsen.

RichardPrice 15y ago
Hi Tyler - nice comment. Also, wonderful work on www.thecostofknowledge.com! I agree that discovery and credit/prestige are the two key things to get right. I'm sure that the kinds of systems you're talking about, voting systems, will be part of the research discovery/credit process in the future. It works for web content, and also for very esoteric content on the web (MathOverflow etc), so it should work for research papers too.

knowtheory 15y ago

It's also worth noting that academia.edu is a for-profit enterprise. The domain academia.edu was purchased prior to the strictures put in place to ensure that only qualified educational institutions could register .edu domains, and has been grandfathered ever since. Academia.edu itself launched in 2008 apparently. The author of this piece very much has a horse in this game, and as a result I very much view his writing with a skeptical eye. There are serious and substantial problems with social models of publishing, especially when organizations/institutions start tying promotions and/or remuneration to metrics like impact factor (which itself is a proprietary formula). We should keep in mind to reward good science because it's good science, not just because it's popular.

RichardPrice 15y ago
I think you are getting the horse and the cart in the wrong order (to continue the horse analogies :) ). I started Academia.edu, after finishing my PhD at Oxford, because I believe that science, is too closed, and too slow, and that I think that we can build a faster, cheaper, and better system for disseminating research. You're wondering whether I'm asserting the things in the post because I started the company, but really it's the other way round. I started the company because I believe the philosophy expressed in the post. One has to be the change one wants to see in the world.
- knowtheory 15y ago
  I'm sad to see that someone downvoted factual information provided about you and the post you wrote. And I'm not asserting anything about your motivations for why you started a for-profit company. I agree with you that the publishing models we have currently are broken. At the same time, I'm not entirely happy with some of the changes that have taken place in the past 5-10 years. When you look to the changes that digital distribution have wrought in industries like the news media over a similar time span, the changes are not all healthy ones either from the perspective of publishers or consumers. The fact that HuffPo and Gawker have risen to the top of the pile as the sorts of money making enterprises is not such a great example to set. Moreover, I'm unconvinced that adding more eyeballs to papers is going to improve quality of review. If your assertion is that we can make scientific review faster and cheaper without a subsequent loss of quality (let's just say "more scalable"), then I'd love to hear your response to a claim i like to make. I'd assert that publishing scientific papers is more similar to the trials and travails of online media than it is to publishing source code. The thing that makes open source software work is that software behavior (broadly speaking) is verifiable. I can run your code and we can agree on the results, or you can tell me i'm an idiot for not configuring the software appropriately. I'd assert that FOSS scalability derives from this ability. More eyeballs helps because everyone is working from/over the same artifact. The same can't be said for most academic papers, because most academes don't even publish their raw data, let alone possess have the facilities or the will to publish verifiable algorithmic processes for arriving at their data. I'd love to see a project which encouraged better replicability and verification of results/test data. But you haven't suggested any of those things. P.S. absolutely, please do be the change you're looking for. But I'm not convinced that you're the change i'm looking for.

merraksh 15y ago

The drawbacks of the Two Person review process are that it is [...] expensive: $8 billion a year is spent on subscriptions to journals, which is money that could be spent on more research. None of that money goes to the two reviewers. The reviewers could do the same work for open access journals, and they would have (IMHO) no motivation not to do it for free.