8 ms·
“Peer review” is younger than we think
- kobeya 9y agoThe journals published by the Royal Society, where modern enlightenment science was born, was informally peer reviewed by the mid 1700's, and peer review was systematized in the 1800's. The explosive growth of the industrial revolution, the reordering of society in the enlightenment, and everything we call modern and progressive is associated with a sequence of events that is also tightly correlated with the birth of peer review. Most philosophers of science don't consider this a coincidence. Science gave birth to exponential progress, and science itself is dependent on peer review. Maybe the integration of peer review into practical industries, medical studies, and government regulation came later. But those were cases of taking something that worked astonishingly well and apply it to other domains. It's hardly fair to say that peer review "started" in the 1960's or 70's.
- GalacticDomin8r 9y agoThe blog indicates this quite clearly and is speaking of only the formal, modern and monetized usage of it. Science itself is basically useless without some type of peer review in whatever form.
- kobeya 9y agoWhat I'm taking issue with is this: > But even if peer review is ancient, "peer review" itself is quite new. Google charts about the history of usage of a particular grouping of terms is not definitive evidence about the history of the underlying activity named by that phrase. Peer review is indeed ancient and fundamental to science and scientific progress. It just may not have been talked about as broadly or referred to under different names before then. It's hard to draw more than linguistic trivia out of that. It's especially bad to make policy suggestions based on that.
- cwyers 9y agoBut that's the author's point. This current thing we call peer review isn't what's fundamental to science, the broader concept is. The modern for-profit journal system is not the only way to do science, and just because it cloaks itself in the name "peer review" doesn't make it the only sort of peer review that facilitates good science.
- trapperkeeper74 9y agoThere are three activities: - critical feedback on papers - hypermonetization (and deleterious siloing) of works often produced with government grants, leading to primarily academics only having ready access to papers, whom don’t see the subscription costs as a burden to them. - disseminating only papers meeting a threshold to maintain quality of the journal Maybe scientists and academics should seek to create and use equivalent quality peer / committee adversalism but publish in open forum that operates based on donations. Academia seems currently over-commercialised: industry getting cheap labor, undergrads going into debt and journals, law libraries and textbook companies making a killing by locking up knowledge behind a paywall. Research papers and case law should be available to the most people as possible for little or no cost.
- dnautics 9y agoWhy the emphasis on money? The structural organization of science as a hierarchical enterpise with gatekeepers to advancement will result in these problems regardless of funding or commercialization.
- AstralStorm 9y agoTying funding to it makes them much more acute though. Essentially forcing you to play by these rules in order to live. Without being forced like this, many deadbeat fake journals would die and places like arxiv would flourish.
- marcosdumay 9y ago
- pnathan 9y agoJust for the sake of solidity, would you care to have a citation there?
- raverbashing 9y agoThe key word is: informally Formal peer review seems to encourage collective backscratching and discourage new ideas (even though some places do anonymous PR, it's not hard to guess where it's coming from if you're a specialist on the area and familiar with the most prominent labs)
- montecarl 9y agoI can confirm that it is very easy to determine who your anonymous reviewers are in specialist areas. In one topic that I used to publish in, I could easily identify one or two of the three reviewers. The main tells are when the reviewer brings up their tangentially related niche interests (e.g. how does this compare to method X or have you considered the effects of <relatively obscure phenomenon>).
- kobeya 9y agoAlso in some cases (e.g. many conferences) you can know who the list of reviewers are. Just look up the program committee. Only a subset reviews your paper submission, but with the field of possible reviewers constrained to 10-12 people you personally know, it becomes very hard to hide.
- ninguem2 9y agoThe term "peer review" has gained usage relatively recently but the term "refereeing" has been in usage much longer. The practice itself, as noted in kobeya's comment, is 100's of years old.
- kang 9y agoPeer-review is deeply flawed and in its current form merely an advertisement form. We should focus on formatting science in a way that accounts for merit based on reproducibility, falsifiability and paying for real-life usage for recipe invented. Problem is not all science work is technological application.
- devdoomari 9y agomaybe it's time to augment peer-review with some other stuff instead of completely getting rid of it?
- trapperkeeper74 9y agoThere are many, many BS papers because of the mass-production death-march researchers are told to produce, low p-value and sample size standards and ”Is Most Published Reseach Wrong?” (Veritasium, 2016) https://youtu.be/42QuXLucH3Q https://youtu.be/42QuXLucH3Q Which is about an essay from 2005: http://journals.plos.org/plosmedicine/article?id=10.1371%2Fjournal.pmed.0020124 http://journals.plos.org/plosmedicine/article?id=10.1371%2Fj... This doesn’t mean adversarial and critical feedback are bad, it means dis/incentives, politics and standards need adjustments.
- jonathanstrange 9y agoThe problem is not primarily peer-review, the problem is the ridiculous publication pressure we're under. I'm a postdoc in the humanities and some of my colleagues publish 10+ bullshit papers in crappy journals per year - their career is virtually guaranteed, even though everybody knows and agrees that it's impossible to write that much and maintain any level of quality. It doesn't matter how bad or insignificant your work is, if you manage to push out 5+ Scopus-indexed articles per year, nobody will be able to fire you or overlook you in tenure track applications, and I see more and more people gaming the system in that way. It's leads to a horrible display of mediocricity, of course, and anyone with a genuine interest in substantial scientific research has a hard time competing. Yes, some decision makers still take quality into account, but overall there is a clearcut trend towards chickenshit and bean-counting, at least in Portugal where I currently live.
- raverbashing 9y agoHas the "modern way" of doing science been evaluated and compared against older ways of doing things? While discoveries do go down in numbers it seems most of past advancements wouldn't have survived modern standards (while a lot of one-off studies do go through today)
- egocodedinsol 9y agoThis is an important point, but the comparison is difficult. Past advancements occurred in a drastically different time period - few professional scientists, for instance. I don’t defend current peer review as optimal, but I can’t see an easy way to carry out a fair comparison.
- dredmorbius 9y agoOver what time period? Modern science starts roghly with the Renaissance and really takes off in the 18th and 19th centuries. Empiricism owes a great deal to the Bacons, Francis and Roger (no relation), of the 17th and 13th centuries
- egocodedinsol 9y agoI don’t understand the second part of the title. I don’t see anything in the essay to suggest it can or should go away. The first part is probably solid: peer review is new. But older scholarship was performed in a very different context - few professional scientists, etc. But the second part of the title, while provocative, makes no sense to me.
- murphyslab 9y agoOne of the key factors missing in this discussion is that there are two overlapping forms of peer review: One which is merely technical, checking for correctness and completeness. The other is a much more subjective form, in which a reviewer is asked to comment on the "merit" or potential "impact" of a given study. The former is often necessary, as honestly there's a lot of crap out there, but more often it comes down to minor details that the expertise of another scientist's specialty can best address. The downside is that this can inspire a certain pedantry and pressure to undertake what is essentially unnecessary formatting. The latter is essentially the real problem of science in its current state: prestige. There is no reason to not just publish everything in a mega-journal that doesn't consider "merit". It would mean more replication studies and less publication bias against "negative" results (i.e. this drug didn't work as hoped). High-prestige journals also tend to charge correspondingly higher subscription fees, while adding nothing special to the scholarly ecosystem, beyond delaying (often by months) the eventual publication elsewhere. However careers are made on publishing in high-prestige journals. A recent tweet makes one wonder what the value of high-prestige journals in the first place, if they don't even bother to correct or improve the scholarly record. A PNAS editor's rejection of a paper with greater statistical power and improved methodology, but which fails to replicate a highly cited PNAS paper was on the basis that "in these days of computer literature searches, I see no reason a journal needs to feel compelled to publish follow-up papers on previously published work. The follow-up paper would be found by those interested where ever it is published" https://twitter.com/mendel_random/status/917115714544324609 https://twitter.com/mendel_random/status/917115714544324609
- eccstartup 9y agoA piece of work should be peer reviewed if the authors write it for money.
- anon1253 9y agoAnother interesting tidbit is that Evidence Based Medicine, what we've come to known as randomized controlled trials with systematic reviews, is /much/ younger than people think. The movement properly started in the 1960s, but only came to fruition after Archie Cochrane (1970s). Still Evidence Based Practice is a contested thing, and very few systematic reviews eventually end up informing policy.
- alphydan 9y agoBlind experiments in medicine were carried out a lot earlier. See for example the revolutionary work of Claude Bernard in 1865 [0] [0] https://en.wikipedia.org/wiki/Claude_Bernard https://en.wikipedia.org/wiki/Claude_Bernard
- anon1253 9y agoSure, but I was specifically pointing to EBM https://en.wikipedia.org/wiki/Evidence-based_medicine https://en.wikipedia.org/wiki/Evidence-based_medicine
- TempleOS 9y agoI have divine intellect as God's avatar filled with HolySpirit. You are brainwashed retardniggers with attitude. God says... mackinaws gentled pitiably lemming's Gilligan's histamines unripest Nebraskan's poppycock unleashing Utopias changed cowboy's haiku hunchbacked victory seersucker's herdsmen fulfils polite ligature's pompom's drawstring privier dismiss rigmaroles pinhole Morse carom sideline raster envelope's
- arca_vorago 9y agoThe next revolution needs to be in reproducing what has been "peer reviewed". During my time at a genetics company I learned bad science abounds even in the " top" journals. One particular example is people "collabing" just to get their name on more papers, which they turn around and use as a marketing tool. Any time I hear the phrase "I've been published $number of times" these days I tend to think they are worse scientist's not better.
- jwilk 9y agoTypography of this article was not peer-reviewed. They somehow managed to use two different fonts within a single word. T<span style="font-family: LMRoman10-Regular-Identity-H;">he earliest usage oO
- galeforcewinds 9y agoWhile the concept of peer review of intellectual property work products may be relatively new, there is long history of craft guilds applying peer quality controls on physical work products in part to protect the reputation of that class of products within the market.
- Bromskloss 9y agoI feel that this title is putting words in my mouth.