8 ms·

Two comments: 1. Scientists publish in slow peer reviewed journals, because scientific publication has to be peer reviewed. The reviewers are a very small sele

Two comments: 1. Scientists publish in slow peer reviewed journals, because scientific publication has to be peer reviewed. The reviewers are a very small selection of other scientists, who are expert in the field of the publication. As long as the reviewers don't signal white smoke, a publication is no science by definition. This way of work is important to keep quality excellent. We certainly don't need more quickly published junk that stirs the world but is found to be inaccurate later. Can we accelerate the process of peer reviewing? I doubt that. As I have written, the reviewers are other scientists who have to find the time to read and comment new articles. The more time they spend with this, the less time they spend doing research. So the small amount of experts usually is responsible for being slow. You may be quicker by internet publishing, if (IF!) you work on a highly dynamic field where other scientists are hungry for your results. As the author writes, this is the exception. Most scientific results will be useful only a long time after the publication, if they will be useful at all. In this way, science is much faster than application already. 2. The author writes, we could make progress faster. Please: why? It seems to be a sign of the time that we want to accelerate everything. It is most likely that this will only produce more noise. We need to slow down our lives and our thinking to stay accurate and produce real value. I am glad that science results, at least in my field, are still reliably scrutinized by many. This way I know that reading and spending the time to understand the stuff is worth my time.

beambot 15y ago
To add to that, 3. Scientists (particularly academics) should focus on a smaller number of solid, flushed-out papers rather than seeking out the minimum publishable unit. The latter seems to be a byproduct of the tenure process (IMO) and just adds noise.
- eru 15y ago
  How about you focus on publishing diffs to an overview (or so) instead of self-contained articles?
  - kd0amg 15y ago
    What do you do with the details too small to belong in an overview?
    - eru 15y ago
      Overview was probably the wrong word.
JoeAltmaier 15y ago
The article made the point that this is an "accident of paper". Peer review comes from the expense of curating paper documents. Of course it is arguable that peer review weeds out noise too. But you have to get your work in front of a peer, and they have to understand your contribution. Its also arguable that peer review leads to stagnation and missed opportunities/discoveries. In fact, surely it does.
- rflrob 15y ago
  >But you have to get your work in front of a peer, and they have to understand your contribution. A very small (but growing) minority of journals, like PLoS One or , make no effort to understand or estimate the significance of the contribution, only whether or not the work is scientifically sound.
dantheman 15y ago
Speed is important for a number of factors, mistakes cost less if they can be quickly corrected. So being risky and publishing about your failures becomes useful (no one publishes failures...). Speed also reduces duplicate work. Peer review is slow because publishing and retracting used to take a long time, now it is instant. And as for the quality of the reviewer, perhaps it would make sense for the journal to employ professional scientists that only review -- they'll be up to date on all the current research (that's their job), can highlight gaps, and can help cross pollinate other disciplines by reviewing 2 or 3 different but related topics. The idea that a scientist should do everything seems to be more and more inefficient. We need to break down tasks like we do in the commercial world so that experts can move faster.
- cube13 15y ago
  >Speed is important for a number of factors, mistakes cost less if they can be quickly corrected. So being risky and publishing about your failures becomes useful (no one publishes failures...). Speed also reduces duplicate work. You're missing the point of scientific publication. The entire point of publication is so others can reproduce your work. Speed doesn't work if you need a billion data points taken over 20 years to prove a long term issue. >The idea that a scientist should do everything seems to be more and more inefficient. We need to break down tasks like we do in the commercial world so that experts can move faster. The problem here is that for a lot of cutting edge theoretical research, the experts are the only ones qualified to really vet the paper. And finding experts that: 1. Are capable of understanding everything in the paper and 2. Are willing to put aside their personal research to review the paper is a VERY small set of people. For example, the P=NP proofs produced over the last few years were only presented to a small group of about 20 very qualified mathematicians to vet. I, with an undergrad CS degree, could barely understand the summary of the proofs. As far as I know, those proofs still have not been completely refuted or accepted. I don't think that this is inefficient, it's simply the fact that there aren't enough people with enough time to really work with extraordinarily complex concepts and ideas.
  - pnathan 15y ago
    > 1. Are capable of understanding everything in the paper Agreed. In my academic field (Debuggers), to understand and know enough to give a real review of the subject requires reading what I would estimate to be somewhere around a thousand papers. It also requires keeping up with the major industrial producers of the product. I think the field has something like two to three thousand papers right now, I haven't checked in a while. For my work, I am just hitting the seminal papers and trying to avoid spending time reading work that led nowhere. I've read maybe two-three hundred articles (haven't tracked), and my thesis cites over one hundred. So no, ordinary people won't do that. Ordinary computer programmers won't do that. Most people are not capable (or prepared) to understand everything in a given paper and comment on what has gone before and what has been tried before. Academic knowledge maintenance is the total philosophical opposite of tl;dr and Twitter.
angersock 15y ago
I'm sorry, but I must disagree with the spirit of your points. Scientists publish in slow peer reviewed journals, because scientific publication has to be peer reviewed. The reviewers are a very small selection of other scientists, who are expert in the field of the publication. So, when you throw work out into the open, anyone with an interest in your topic (your functional peers) will be able to do their own analysis of your work (a review)--some sort of peer-review, if you will. All without needing to go through a stodgy paper review process. This way of work is important to keep quality excellent. We certainly don't need more quickly published junk that stirs the world but is found to be inaccurate later. So, if there is one thing we've seen in the programming world, secluded cabals of experts don't always result in excellent quality (I submit to you, OpenGL, C++, CORBA, for a few examples). New patches are submitted to open-source projects all the time; some are accepted, some are rebuked publicly, and some merely catalyze interesting discussion. "Stirring the world" is hardly a bad thing--in the case of Fleischmann-Pons, the debunking happened quickly and the world was free to move on. In a similar avenue, one of the reasons that the eCat story is frustrating is the lack of transparency (both from the inventors and the media refusing to even air solid rebuttals)--were the information more open, we could sort it into bunk/science and move on. Can we accelerate the process of peer reviewing? I doubt that. As I have written, the reviewers are other scientists who have to find the time to read and comment new articles. The more time they spend with this, the less time they spend doing research. So the small amount of experts usually is responsible for being slow. So, you make the (valid) claim that the process can't go faster, predicating on the scarcity of experts. You seem to have the answer at hand: get more experts. This can be accomplished by either getting more researchers reviewing, or loosening the restrictions on who is considered a valid source. Indeed, places like StackExchange have shown us how to pick out decently good judges of these things--we don't have as great a need for the older method of establishing experts, anymore. 2. The author writes, we could make progress faster. Please: why? It seems to be a sign of the time that we want to accelerate everything. It is most likely that this will only produce more noise. We need to slow down our lives and our thinking to stay accurate and produce real value. In order to help motivate the need for faster progress, please note the following: Earth has over 7 billion people now. (http://www.unfpa.org/swp/ http://www.unfpa.org/swp/) The bottom 50% of people in the world (rightly or wrongly) own around 1% of the wealth. (http://escholarship.org/uc/item/3jv048hx#page-4 http://escholarship.org/uc/item/3jv048hx#page-4) We (at least those of us in the first world) don't have a lot of time left, unless we can find ways of solving the Big Problems. Progress in tech is likely the only palatable solution to this predicament. We can't really afford to slow down now. I am glad that science results, at least in my field, are still reliably scrutinized by many. This way I know that reading and spending the time to understand the stuff is worth my time. I'm greatly concerned about this sort of attitude. Might I suggest that an occasional dalliance in "time-wasting" articles might prove refreshing, and further that it could help ferret out ideas that would otherwise be overlooked? edit: Modified for civility. Sorry folks.
bmuon 15y ago
I agree that time isn't a matter of concern for most scientists. The article seems to be a badly researched rehash of other Open Science articles. Open Science is not about faster delivery, it's about access to information. The problem is not that publications are slow, it's that without paying for the magazine other scientists, the media and non scientists can't access those papers and learn, divulge, criticize or contribute.
_delirium 15y ago
On #2, it varies by field, but I agree somewhat, in that I generally think there's already a significant problem of people being too quick-on-the-draw without first familiarizing themselves with the literature. There is a lot of stuff published that rehashes (without citation) existing published results or minor variations on them, sometimes from decades earlier, that the authors and reviewers just didn't happen to be familiar with. If you accelerate everything and distribute the peer-review to people with less deep knowledge of the literature, I suspect this will only increase, adding more noise to the literature as we spend the 2010s rediscovering the 1970s, accompanied by triumphant press releases and blog posts about Breakthrough Discoveries. I notice this particularly in computer science, where it seems nobody is capable of locating or reading journal articles published before around 1995, a handful of canonical "classic" papers excepted.
- pnathan 15y ago
  Agreed. Vast amounts of very solid literature have 'fallen' off the public face of the planet in CS, except in the archives of publishing houses and libraries. (Of course, I'm confident part of the problem is that expensive paywalls discourage exhaustive trawling by non-academics).
  - deleted 15y ago
    [deleted]
- quanticle 15y ago
  How much of that is because research from the 1970s is effectively locked away behind ACM and IEEE paywalls? How much research gets rehashed because the researcher can't find the result he or she is looking for via the atrocious search tools provided by the likes of Elsevier? I think that more openness in scientific publication will in fact mitigate the problem of people being quick-on-the-draw, as contradictory findings and previous publications of similar results will be easier to find, even by the non-scientific audience.
  - kd0amg 15y ago
    >> I notice this particularly in computer science, where it seems nobody is capable of locating or reading journal articles published before around 1995, a handful of canonical "classic" papers excepted. How much of that is because research from the 1970s is effectively locked away behind ACM and IEEE paywalls? If you're serious about keeping up with research, a couple hundred dollars every year to get through a paywall doesn't sound so bad. The real problem with the stuff from the 1970s and earlier is that it's not readily available even to paying members. When I went looking for an article from 1970, the ACM Digital Library had an entry for it but not the document itself.
ellyagg 15y ago
I'm sorry, your claim seems to be: The process is working perfectly right now. This is absurd. The scientific method is a lot more open to interpretation than you suggest, and indeed it is interpreted with wide variety in different disciplines. The methods of science will be disrupted and improved. Bright lay people will absolutely be able to poke holes in research once the methods and data are completely open, as has been indisputably proven with the advent and prominence of open source software. Research has shown systemic flaws in numerous disciplines. These sorts of flaws fester because of the closed nature of the system. Open research will have its day. It's only a matter of time.
- pnathan 15y ago
  > Bright lay people will absolutely be able to poke holes in research By the time they can astutely manage the actual field with understanding, they will no longer be lay people.
zeratul 15y ago
The article makes at lest one important point. There are only two incentives for a scientist: publications and grant money. Getting published fast reduces quality of research. Also, the focus on publications hinders other research outcomes like publicly available data or software. Industry or start-ups may not make full use of publications that attack the same problem over and over again but they surely can improve with a new data or newly implemented algorithms. [EDIT] Publicly available data and software make replicability more realistic. Currently lack of details in publications make it almost impossible. If only there was some way to influence NIH and NSF grant requirements ...
- lutorm 15y ago
  There are only two incentives for a scientist: publications and grant money. I disagree. There are three incentives: first, getting a permanent job, second getting grant money, and third, doing good work. The two former are unfortunately sometimes in opposition to the third.
- jeltz 15y ago
  NIH are among the proponents of Open access so they should be one of the easier organisations to convince. I believe all articles resulting from NIH sponsored research must now be published in Open access papers.
gwern 15y ago
Why do we need the process of peer review? Peer review is not robust against even low levels of collusion (http://arxiv.org/abs/1008.4324v1 http://arxiv.org/abs/1008.4324v1). Scientists who win the Nobel Prize find their other work suddenly being heavily cited (http://www.nature.com/news/2011/110506/full/news.2011.270.html http://www.nature.com/news/2011/110506/full/news.2011.270.ht...), suggesting either that the community either badly failed in recognizing the work's true value or that they are now sucking up & attempting to look better by the halo effect. (A mathematician once told me that often, to boost a paper's acceptance chance, they would add citations to papers by the journal's editors - a practice that will surprise none familiar with Goodhart's law and the use of citations in tenure & grants.) Physicist Michael Nielsen points out (http://michaelnielsen.org/blog/three-myths-about-scientific-peer-review/ http://michaelnielsen.org/blog/three-myths-about-scientific-...) that peer review is historically rare (just one of Einstein's 300 papers was peer reviewed! the famous _Nature_ did not institute peer review until 1967), has been poorly studied (http://jama.ama-assn.org/cgi/content/abstract/287/21/2784 http://jama.ama-assn.org/cgi/content/abstract/287/21/2784) & not shown to be effective, is nationally biased (http://jama.ama-assn.org/cgi/content/full/295/14/1675 http://jama.ama-assn.org/cgi/content/full/295/14/1675), erroneously rejects many historic discoveries (one study lists "34 Nobel Laureates whose awarded work was rejected by peer review" (http://www.canonicalscience.org/publications/canonicalsciencereports/20082.html http://www.canonicalscience.org/publications/canonicalscienc...); Horribin 1990 (http://jama.ama-assn.org/content/263/10/1438.abstract http://jama.ama-assn.org/content/263/10/1438.abstract) lists others like the discovery of quarks), and catches only a small fraction (http://jama.ama-assn.org/cgi/content/abstract/280/3/237 http://jama.ama-assn.org/cgi/content/abstract/280/3/237) of errors. And fraud, like the one we just saw in psychology? Forget about it (http://www.plosone.org/article/info%3Adoi%2F10.1371%2Fjournal.pone.0005738 http://www.plosone.org/article/info%3Adoi%2F10.1371%2Fjourna...); > "A pooled weighted average of 1.97% (N = 7, 95%CI: 0.86–4.45) of scientists admitted to have fabricated, falsified or modified data or results at least once –a serious form of misconduct by any standard– and up to 33.7% admitted other questionable research practices. In surveys asking about the behaviour of colleagues, admission rates were 14.12% (N = 12, 95% CI: 9.91–19.72) for falsification, and up to 72% for other questionable research practices....When these factors were controlled for, misconduct was reported more frequently by medical/pharmacological researchers than others." No, peer review is not the secret sauce of science. Replication is more like it.

beambot 15y ago

To add to that, 3. Scientists (particularly academics) should focus on a smaller number of solid, flushed-out papers rather than seeking out the minimum publishable unit. The latter seems to be a byproduct of the tenure process (IMO) and just adds noise.

eru 15y ago
How about you focus on publishing diffs to an overview (or so) instead of self-contained articles?
- kd0amg 15y ago
  What do you do with the details too small to belong in an overview?
  - eru 15y ago
    Overview was probably the wrong word.

JoeAltmaier 15y ago

The article made the point that this is an "accident of paper". Peer review comes from the expense of curating paper documents. Of course it is arguable that peer review weeds out noise too. But you have to get your work in front of a peer, and they have to understand your contribution. Its also arguable that peer review leads to stagnation and missed opportunities/discoveries. In fact, surely it does.

rflrob 15y ago
>But you have to get your work in front of a peer, and they have to understand your contribution. A very small (but growing) minority of journals, like PLoS One or , make no effort to understand or estimate the significance of the contribution, only whether or not the work is scientifically sound.

dantheman 15y ago

Speed is important for a number of factors, mistakes cost less if they can be quickly corrected. So being risky and publishing about your failures becomes useful (no one publishes failures...). Speed also reduces duplicate work. Peer review is slow because publishing and retracting used to take a long time, now it is instant. And as for the quality of the reviewer, perhaps it would make sense for the journal to employ professional scientists that only review -- they'll be up to date on all the current research (that's their job), can highlight gaps, and can help cross pollinate other disciplines by reviewing 2 or 3 different but related topics. The idea that a scientist should do everything seems to be more and more inefficient. We need to break down tasks like we do in the commercial world so that experts can move faster.

cube13 15y ago
>Speed is important for a number of factors, mistakes cost less if they can be quickly corrected. So being risky and publishing about your failures becomes useful (no one publishes failures...). Speed also reduces duplicate work. You're missing the point of scientific publication. The entire point of publication is so others can reproduce your work. Speed doesn't work if you need a billion data points taken over 20 years to prove a long term issue. >The idea that a scientist should do everything seems to be more and more inefficient. We need to break down tasks like we do in the commercial world so that experts can move faster. The problem here is that for a lot of cutting edge theoretical research, the experts are the only ones qualified to really vet the paper. And finding experts that: 1. Are capable of understanding everything in the paper and 2. Are willing to put aside their personal research to review the paper is a VERY small set of people. For example, the P=NP proofs produced over the last few years were only presented to a small group of about 20 very qualified mathematicians to vet. I, with an undergrad CS degree, could barely understand the summary of the proofs. As far as I know, those proofs still have not been completely refuted or accepted. I don't think that this is inefficient, it's simply the fact that there aren't enough people with enough time to really work with extraordinarily complex concepts and ideas.
- pnathan 15y ago
  > 1. Are capable of understanding everything in the paper Agreed. In my academic field (Debuggers), to understand and know enough to give a real review of the subject requires reading what I would estimate to be somewhere around a thousand papers. It also requires keeping up with the major industrial producers of the product. I think the field has something like two to three thousand papers right now, I haven't checked in a while. For my work, I am just hitting the seminal papers and trying to avoid spending time reading work that led nowhere. I've read maybe two-three hundred articles (haven't tracked), and my thesis cites over one hundred. So no, ordinary people won't do that. Ordinary computer programmers won't do that. Most people are not capable (or prepared) to understand everything in a given paper and comment on what has gone before and what has been tried before. Academic knowledge maintenance is the total philosophical opposite of tl;dr and Twitter.

angersock 15y ago

I'm sorry, but I must disagree with the spirit of your points. Scientists publish in slow peer reviewed journals, because scientific publication has to be peer reviewed. The reviewers are a very small selection of other scientists, who are expert in the field of the publication. So, when you throw work out into the open, anyone with an interest in your topic (your functional peers) will be able to do their own analysis of your work (a review)--some sort of peer-review, if you will. All without needing to go through a stodgy paper review process. This way of work is important to keep quality excellent. We certainly don't need more quickly published junk that stirs the world but is found to be inaccurate later. So, if there is one thing we've seen in the programming world, secluded cabals of experts don't always result in excellent quality (I submit to you, OpenGL, C++, CORBA, for a few examples). New patches are submitted to open-source projects all the time; some are accepted, some are rebuked publicly, and some merely catalyze interesting discussion. "Stirring the world" is hardly a bad thing--in the case of Fleischmann-Pons, the debunking happened quickly and the world was free to move on. In a similar avenue, one of the reasons that the eCat story is frustrating is the lack of transparency (both from the inventors and the media refusing to even air solid rebuttals)--were the information more open, we could sort it into bunk/science and move on. Can we accelerate the process of peer reviewing? I doubt that. As I have written, the reviewers are other scientists who have to find the time to read and comment new articles. The more time they spend with this, the less time they spend doing research. So the small amount of experts usually is responsible for being slow. So, you make the (valid) claim that the process can't go faster, predicating on the scarcity of experts. You seem to have the answer at hand: get more experts. This can be accomplished by either getting more researchers reviewing, or loosening the restrictions on who is considered a valid source. Indeed, places like StackExchange have shown us how to pick out decently good judges of these things--we don't have as great a need for the older method of establishing experts, anymore. 2. The author writes, we could make progress faster. Please: why? It seems to be a sign of the time that we want to accelerate everything. It is most likely that this will only produce more noise. We need to slow down our lives and our thinking to stay accurate and produce real value. In order to help motivate the need for faster progress, please note the following: Earth has over 7 billion people now. (http://www.unfpa.org/swp/ http://www.unfpa.org/swp/) The bottom 50% of people in the world (rightly or wrongly) own around 1% of the wealth. (http://escholarship.org/uc/item/3jv048hx#page-4 http://escholarship.org/uc/item/3jv048hx#page-4) We (at least those of us in the first world) don't have a lot of time left, unless we can find ways of solving the Big Problems. Progress in tech is likely the only palatable solution to this predicament. We can't really afford to slow down now. I am glad that science results, at least in my field, are still reliably scrutinized by many. This way I know that reading and spending the time to understand the stuff is worth my time. I'm greatly concerned about this sort of attitude. Might I suggest that an occasional dalliance in "time-wasting" articles might prove refreshing, and further that it could help ferret out ideas that would otherwise be overlooked? edit: Modified for civility. Sorry folks.

bmuon 15y ago

I agree that time isn't a matter of concern for most scientists. The article seems to be a badly researched rehash of other Open Science articles. Open Science is not about faster delivery, it's about access to information. The problem is not that publications are slow, it's that without paying for the magazine other scientists, the media and non scientists can't access those papers and learn, divulge, criticize or contribute.

_delirium 15y ago

On #2, it varies by field, but I agree somewhat, in that I generally think there's already a significant problem of people being too quick-on-the-draw without first familiarizing themselves with the literature. There is a lot of stuff published that rehashes (without citation) existing published results or minor variations on them, sometimes from decades earlier, that the authors and reviewers just didn't happen to be familiar with. If you accelerate everything and distribute the peer-review to people with less deep knowledge of the literature, I suspect this will only increase, adding more noise to the literature as we spend the 2010s rediscovering the 1970s, accompanied by triumphant press releases and blog posts about Breakthrough Discoveries. I notice this particularly in computer science, where it seems nobody is capable of locating or reading journal articles published before around 1995, a handful of canonical "classic" papers excepted.

pnathan 15y ago
Agreed. Vast amounts of very solid literature have 'fallen' off the public face of the planet in CS, except in the archives of publishing houses and libraries. (Of course, I'm confident part of the problem is that expensive paywalls discourage exhaustive trawling by non-academics).
- deleted 15y ago
  [deleted]
quanticle 15y ago
How much of that is because research from the 1970s is effectively locked away behind ACM and IEEE paywalls? How much research gets rehashed because the researcher can't find the result he or she is looking for via the atrocious search tools provided by the likes of Elsevier? I think that more openness in scientific publication will in fact mitigate the problem of people being quick-on-the-draw, as contradictory findings and previous publications of similar results will be easier to find, even by the non-scientific audience.
- kd0amg 15y ago
  >> I notice this particularly in computer science, where it seems nobody is capable of locating or reading journal articles published before around 1995, a handful of canonical "classic" papers excepted. How much of that is because research from the 1970s is effectively locked away behind ACM and IEEE paywalls? If you're serious about keeping up with research, a couple hundred dollars every year to get through a paywall doesn't sound so bad. The real problem with the stuff from the 1970s and earlier is that it's not readily available even to paying members. When I went looking for an article from 1970, the ACM Digital Library had an entry for it but not the document itself.

ellyagg 15y ago

I'm sorry, your claim seems to be: The process is working perfectly right now. This is absurd. The scientific method is a lot more open to interpretation than you suggest, and indeed it is interpreted with wide variety in different disciplines. The methods of science will be disrupted and improved. Bright lay people will absolutely be able to poke holes in research once the methods and data are completely open, as has been indisputably proven with the advent and prominence of open source software. Research has shown systemic flaws in numerous disciplines. These sorts of flaws fester because of the closed nature of the system. Open research will have its day. It's only a matter of time.

pnathan 15y ago
> Bright lay people will absolutely be able to poke holes in research By the time they can astutely manage the actual field with understanding, they will no longer be lay people.

zeratul 15y ago

The article makes at lest one important point. There are only two incentives for a scientist: publications and grant money. Getting published fast reduces quality of research. Also, the focus on publications hinders other research outcomes like publicly available data or software. Industry or start-ups may not make full use of publications that attack the same problem over and over again but they surely can improve with a new data or newly implemented algorithms. [EDIT] Publicly available data and software make replicability more realistic. Currently lack of details in publications make it almost impossible. If only there was some way to influence NIH and NSF grant requirements ...

lutorm 15y ago
There are only two incentives for a scientist: publications and grant money. I disagree. There are three incentives: first, getting a permanent job, second getting grant money, and third, doing good work. The two former are unfortunately sometimes in opposition to the third.
jeltz 15y ago
NIH are among the proponents of Open access so they should be one of the easier organisations to convince. I believe all articles resulting from NIH sponsored research must now be published in Open access papers.

gwern 15y ago

Why do we need the process of peer review? Peer review is not robust against even low levels of collusion (http://arxiv.org/abs/1008.4324v1 http://arxiv.org/abs/1008.4324v1). Scientists who win the Nobel Prize find their other work suddenly being heavily cited (http://www.nature.com/news/2011/110506/full/news.2011.270.html http://www.nature.com/news/2011/110506/full/news.2011.270.ht...), suggesting either that the community either badly failed in recognizing the work's true value or that they are now sucking up & attempting to look better by the halo effect. (A mathematician once told me that often, to boost a paper's acceptance chance, they would add citations to papers by the journal's editors - a practice that will surprise none familiar with Goodhart's law and the use of citations in tenure & grants.) Physicist Michael Nielsen points out (http://michaelnielsen.org/blog/three-myths-about-scientific-peer-review/ http://michaelnielsen.org/blog/three-myths-about-scientific-...) that peer review is historically rare (just one of Einstein's 300 papers was peer reviewed! the famous _Nature_ did not institute peer review until 1967), has been poorly studied (http://jama.ama-assn.org/cgi/content/abstract/287/21/2784 http://jama.ama-assn.org/cgi/content/abstract/287/21/2784) & not shown to be effective, is nationally biased (http://jama.ama-assn.org/cgi/content/full/295/14/1675 http://jama.ama-assn.org/cgi/content/full/295/14/1675), erroneously rejects many historic discoveries (one study lists "34 Nobel Laureates whose awarded work was rejected by peer review" (http://www.canonicalscience.org/publications/canonicalsciencereports/20082.html http://www.canonicalscience.org/publications/canonicalscienc...); Horribin 1990 (http://jama.ama-assn.org/content/263/10/1438.abstract http://jama.ama-assn.org/content/263/10/1438.abstract) lists others like the discovery of quarks), and catches only a small fraction (http://jama.ama-assn.org/cgi/content/abstract/280/3/237 http://jama.ama-assn.org/cgi/content/abstract/280/3/237) of errors. And fraud, like the one we just saw in psychology? Forget about it (http://www.plosone.org/article/info%3Adoi%2F10.1371%2Fjournal.pone.0005738 http://www.plosone.org/article/info%3Adoi%2F10.1371%2Fjourna...); > "A pooled weighted average of 1.97% (N = 7, 95%CI: 0.86–4.45) of scientists admitted to have fabricated, falsified or modified data or results at least once –a serious form of misconduct by any standard– and up to 33.7% admitted other questionable research practices. In surveys asking about the behaviour of colleagues, admission rates were 14.12% (N = 12, 95% CI: 9.91–19.72) for falsification, and up to 72% for other questionable research practices....When these factors were controlled for, misconduct was reported more frequently by medical/pharmacological researchers than others." No, peer review is not the secret sauce of science. Replication is more like it.