11 ms·

The rise and fall of peer review

199 points by Vinnl 4y ago

truth777 4y ago
I see a few major problems in the pursuit of modern scientific inquiry. - Bureaucratization. The modern university model is basically the DMV. It functions to sustain the lifestyle of the administrators in charge and to prevent them from losing power, money, status. And no one enjoys it, not the employees, not the customers, etc. and yet who can change it for the better? It can and does only get worse, more bureaucratic, more soul sucking. Ask the post-docs. - Feminization of academia and science. This is related to the bureaucratization. Process, safety, paperwork, meetings and community consensus are paramount. All becomes politics. Everyone must agree. Only small questions can be answered. You need permission for everything. Anything truly novel is considered a threat to the scientific community. Modern day "scientism" and other beliefs are the replacement to christianity in the west, and women are more religious (look it up), and academe is the modern day church. All dissent or inquiry is squashed. - Denial of Great Man Theory. Great men invented modernity, and of course our modern industrial state requires many layers of managers due to the huge complexity, so sizes of labs, colleges, corporations, assembly lines, supply chains have exploded, and no one person can manage it all in their mind. The false belief is thus: modern industrial society requires the managerial class to function, therefore the managerial class invented it, and it requires the managerial class to progress. Therefore great man theory, or the idea of the innovative genius, is supposedly debunked. And yet you see nearly all major advances come from tech bros like Elon Musk et al who are trailblazers, ignore complaints of the bureacracy, etc. - Could the university system have produced facebook, spacex, tesla, microsoft, apple? You can go on and on. Please don't interpret this as an attack on women or femininity. Society requires everyone for it to function properly, and everyone has a positive contribution to make, but we have to be able to iterate and change when we realize certain modes of endeavor simply do.not.work.
- fastaguy88 4y ago
  Interesting political philosophy, but little that speaks to the pursuit of science. Universities may be bureaucratic, but those bureaucracies have almost nothing to do with scientific agendas. Likewise, grant review panels work hard to find innovative, not consensus, proposals. Again, the main thing the management does is demand more external funding, it does not set scientific direction. And, of course, Facebook, Microsoft, etc is not science- at best it is engineering.
  - logicalmonster 4y ago
    > Likewise, grant review panels work hard to find innovative, not consensus, proposals. I don't want to give specific examples of research topics to avoid poisoning the well, but could you argue with a straight face that an academic that wanted to do research into some topics that could be likely to yield certain types of politically incorrect conclusions wouldn't face extreme difficulty getting funding or extreme risk to their career? I think it's easy to see how much scientific research on certain topics could be stuck within a narrow range of opinion because people are more concerned with what gets funding or doesn't get them shunned.
    - fastaguy88 4y ago
      I do not know anything about grant panels outside my field of biology. But I am certain that the overwhelming majority of grant money is spent on scientific questions that have virtually no obvious political dimension. I’m sure there are grant applications that have a substantial political component, but I would be surprised if they accounted for even 5% of research funds. Social and political science receives a very small fraction of research funds, and while one might argue that allocations of health research budgets are politically shaped, viruses and oncogenes have no politics.
- p0pcult 4y ago
  >"Feminization"...[bad stuff]...[dubious, gender essentialization] >Denial of Great Men...[bad stuff] >Please don't interpret this as an attack on women or femininity
- poulpy123 4y ago
  > - Bureaucratization Bureaucratization is the consequence of wanting to control that taxes are "properly" used. > - Feminization of academia and science. That's abolutely moronic viewpoint if you ever worked in science > - Denial of Great Man Theory All scientific works, even by genius, lie on the works of others, predecessors or colleagues. > And yet you see nearly all major advances come from tech bros like Elon Musk et al who are trailblazers, ignore complaints of the bureacracy lol what scientific progress Elon Musk did except giving material for the sociology of internet trolls ? > - Could the university system have produced facebook, spacex, tesla, microsoft, apple? You're so close to understand that university and industry are not the same and have not the same goals
  - p0pcult 4y ago
    Apparently, the sexism is rampant in here.
- iosono88 4y ago
p0pcult 4y ago
Compelling arguments!
osigurdson 4y ago
I'm not sure about the pros and cons of peer review but if anyone is claiming that that are doing science, then they must publish raw data so that others can review it. This was the first thing that jumped out at me after reading "Accelerate" - so many claims that everything based on "data". I don't necessarily disagree with the conclusions in the book but I think really just a formalization of Google's development practices. The book should have been called "How we do things at Google".
- phyrex 4y ago
  Well in the book that actually has that content (and very nearly that title: https://abseil.io/resources/swe-book https://abseil.io/resources/swe-book) they explain that they set themselves up to follow the best practices found by the company that eventually published Accelerate, even before google bought them
  - osigurdson 4y ago
    >> they explain that they set themselves up to follow the best practices found by the company that eventually published Accelerate Do you mean as far back as the early days < 2000 or did this happen well after they were established (> 2015). If the latter, how much actually changed?
  - deleted 4y ago
    [deleted]
nixpulvis 4y ago
> Weak-link thinking makes scientific censorship seem reasonable, but all censorship does is make old ideas harder to defeat. Reduce the social, political, professional, and academic cost of "being wrong", learn to have a little more empathy and suddenly negative reviews will hold more meaning once again. People will learn to learn instead of learn to defend. (Of course, defense is a skill worth learning as well. Please don't take what I'm saying as this-or-that.) You simply cannot do away with peer review! What you can eliminate is the control strictures that exist today which punish failure unfairly. This is true at all levels, from the highest institution right on down to that conversation you had with a colleague over coffee yesterday.
- bratwurst3000 4y ago
  Thanks you nailed it
- Vinnl 4y ago
  That's the key here: if peer review is feedback rather than a barrier to sharing your findings and progressing in your career, then negative (though of course, in this situation review would no longer be a binary negative/positive) feedback could be considered input again. But the current setup of peer review, like this article is arguing against, assigns a high cost to "being wrong" (in the eyes of about two reviewers).
  - nixpulvis 4y ago
    Right, but where is the threshold for wrongness which is so widely accepted as truth that it becomes harmful enough to be actively removed? If I began preaching of the correctness of 1+1=3 surely someone would deny my publication, no? Or, the critics would outweigh the proponents. How can this be accomplished without division or empathy?
    - ProjectArcturis 4y ago
      You'd still have peer review, but in a very different sense. Instead of a formal process by 3 reviewers, you'd have an informal process by everyone who read the paper. People would read your dumb 1+1=3 paper, realize that it was wrong, ignore it, and pay less attention to you in the future.
qudat 4y ago
Peer review incentivizes the worst features of humanity: it affords peers in your field the ability to reject your paper because they want to take the idea for themselves and then publish it.
- photochemsyn 4y ago
  In the corporate university, the usual game is to get the patent approved first, and hold off on the publication until that happens. At least that's the case in applied technical science - the social science world seems more like a club where the involved players just rubber-stamp each other's non-reproducible work in order to up their publication count and pack their CV. The actual work has no value, so why would anyone bother stealing it?
  - sideshowb 4y ago
    If that were true we'd see more patent applications in, say, engineering, than journal papers in the same field. (Or at least around the same order of magnitude). Do the numbers check out?
    - ketzu 4y ago
      My experience was: Most papers are not useful for patents. Corporate funding might require a pre-publication review for patent worthyness, and if it is there, it is applied for before publication. Most researchers I knew had little interest in patents, so non-corporate funded papers got published without patent considerations. When it came up, it was a before-you-publish-think-about-it topic.
- rubslopes 4y ago
  I'm not sure that's a real problem. Papers go around a lot before being submitted to a journal. People would notice that on the spot.
  - qudat 4y ago
    Shrug. I used to work in a lab where they talked about it happening. Purely anecdotal
fastaguy88 4y ago
Wow. Pretty clear this paper was not peer reviewed. Many of the arguments would not get past a high school English teacher. ‘A whole lot of money for nothin’ perhaps, but that money is not spent on peer review. Peer review doesn’t improve research productivity …. This reminds me of the argument that we should not spend more money on education, because education budgets have gone up, but test scores have not. The argument implies (but does not state, because of the absurdity), that spending less money would raise scores. And, remarkably (/s), peer review does not guarantee that all scientific results are reproducible and correct. As the author points out, no one has done the control. We don’t know how much worse the scientific literature would be without peer review, and we don’t no how many important insights are lost because of poor reviews. But as someone who has done a lot of peer review (and has had papers rejected with uninformed reviews), it is difficult to state how much worse things could be. (But perhaps that is an overstatement, as very few papers do not find their way into some journal). The scientific literature today is overwhelming. Peer review helps make it less so.
- Vinnl 4y ago
  > it is difficult to state how much worse things could be Because there's no proof. The only thing we know for certain, is that a lot of money is being spent on it, and that some of the mechanisms that are supposed to work can't work (i.e. in the case where people look at and trust preprints anyway). There will definitely be anecdotes of peer review having caught things (though it'll be harder to show that that prevented harm). But likewise, there are also many, many anecdotes that it has been actively harmful. No proven upside, limited anecdotal upside, and proven downside. If peer review as a publication gatekeeper hadn't already been a thing, we wouldn't introduce it — that would be considered unscientific. Sounds like something that's ripe for reconsideration.
  - fastaguy88 4y ago
    I don’t understand how we know that a whole lot of money is spent on it. A lot of volunteer (unpaid) time is spent on it, but not much money. And I’m also unaware of the “proven” downside. But you might ask yourself, why do so many scientists waste so much of their time doing peer review. Perhaps they are ignorant, or easily duped, or perhaps they believe that peer review is a valuable use of their time.
photochemsyn 4y ago
Peer review is not the reason academic research has turned into a garbage fire - it's because of the corporatization of the university and the transition from the open research model to the patent-seeking profit-generating research model. The latter has its place, certainly, but before the 1980s that place was industry-funded private research institutions, the most famous and well-known being Bell Labs. These institutions were funded by groups of corporations who in return got first dibs on their patentable work, and the researchers and engineers in turn got good salaries and didn't have any teaching responsibilities. After Bayh-Dole passed in the 1980s, universities were allowed to grant exclusive licenses to their patentable research to private entities, and this resulted in the rise of the corporate univesity model - people running around hiding their data until they got the patent, people fudging results to make their novel drug look good so their startup would be bought by some pharma corporation, administrations being overrun by pharmaceutical executives who only cared about universities generating a steady stream of patents, and forget about quality-of-teaching. It's been a complete disaster, and not only that, the private research world has largely been defunded as corporations realized that, with the aid of politicians, they could just outsource basic R&D to universities, using grad students as cheap underpaid lab labor on projects where they're little more than glorified lab techs, and not really doing any original research. In addition, the whole system is increasingly authoritarian and censorship is rife - any academic who calls for the elimination of exclusive licensing and the adoption of open-source drug discovery will soon find themselves without a job. It's similar in many ways to the Lysenko era in Soviet science, just run by corporate authoritarians instead of Stalinists. Avoid if possible.
- pdonis 4y ago
  I would add that the funding source for most research now is governments, which means that research is now optimized for getting government grants instead of for actually advancing science.
cs702 4y ago
Whether the "peer review" experiment has succeeded or failed is debatable... but anecdotally, everyone I know in academia feels the current publish-or-perish peer-reviewed system is BROKEN: It incentivizes everyone to pursue low-risk, incremental, mostly inconsequential, easy-to-explain research with high certainty of short-term success, instead of high-risk, exploratory, consequential, hard-to-explain research with high uncertainty of long-term payoff. This is true: > From antiquity to modernity, scientists wrote letters and circulated monographs, and the main barriers stopping them from communicating their findings were the cost of paper, postage, or a printing press, or on rare occasions, the cost of a visit from the Catholic Church. Scientific journals appeared in the 1600s, but they operated more like magazines or newsletters, and their processes of picking articles ranged from “we print whatever we get” to “the editor asks his friend what he thinks” to “the whole society votes.” Sometimes journals couldn’t get enough papers to publish, so editors had to go around begging their friends to submit manuscripts, or fill the space themselves. Scientific publishing remained a hodgepodge for centuries. This is true as well: > That all changed after World War II. Governments poured funding into research, and they convened “peer reviewers” to ensure they weren’t wasting their money on foolish proposals. That funding turned into a deluge of papers, and journals that previously struggled to fill their pages now struggled to pick which articles to print. Reviewing papers before publication, which was “quite rare” until the 1960s, became much more common. Then it became universal. Finally, this is also true: > Now pretty much every journal uses outside experts to vet papers, and papers that don’t please reviewers get rejected. You can still write to your friends about your findings, but hiring committees and grant agencies act as if the only science that exists is the stuff published in peer-reviewed journals. This is the grand experiment we’ve been running for six decades.
- inglor_cz 4y ago
  "everyone I know in academia feels the current publish-or-perish peer-reviewed system is BROKEN: It incentivizes everyone to pursue low-risk, incremental, mostly inconsequential, easy-to-explain research with high certainty of short-term success, instead of high-risk, exploratory, consequential, hard-to-explain research with high uncertainty of long-term payoff." That is what I keep hearing from my academic acquaintances as well - When a measure becomes a target, it ceases to be a good measure. And this is what happened with publish or perish. Let's not forget that Katalin Karikó, a giant in the mRNA field, was demoted by UPenn in 1997 because they considered her work unproductive. But mRNA was precisely the topic that required a multi-decade commitment. Fortunately she was stubborn like hell. God know how many other people give up after being degraded like that.
poulpy123 4y ago
What a load of horseshit: * There are less great discoveries because there are less great and easy things to discover * Time has filtered the scientific studies of the previous centuries * You don't look at the efficiency of a filter (peer reviewing) by the amount of shit that pass through, but by the rating between this amount and the amount of shit that doesn't pass through * A lot of researchers don't want to release their raw data not because they invented their data but because they want to keep the monopoly for them * Peer reviewing has failures and could be improved, but it is the best and quickest way to judge a scientific work (with credentials let's be honest)
- casey2 4y ago
  I agree, that was a load of horseshit, especially bullet 1.
  - Lichtso 4y ago
    I strongly disagree with the first point: As a society the more we discover, the more we increase the surface of the discoverable and the more we can combine things. It only gets harder for the individual because it takes longer and longer to learn everything necessary to get to the frontier of the unknown. It is not a problem of running out of viable research topics.
  - janalsncm 4y ago
    I agree with the easy part. The low hanging fruit has mostly been discovered. If you want to invent something new you’re probably going to need a team of SMEs working on it.
notlukesky 4y ago
Humans can game every system. Then you can tweak the system to make it better for a period of time till it is gamed again. Rinse, wash and repeat. Same flaws exist with all ratings systems and many are just mathematical garbage.
- toomim 4y ago
  Not every system. Some systems can detect gaming, and fight back, like a body's immune system. https://docs.google.com/presentation/d/1SRWbyA2uCnda1JwAZ3JiDKFt1-tWxyjK_FgqUSkBPjc/edit#slide=id.g1508f4412b2_0_354 https://docs.google.com/presentation/d/1SRWbyA2uCnda1JwAZ3Ji... https://peeryview.org/about https://peeryview.org/about
lo_zamoyski 4y ago
Peer review in academia may very well be like the suppression of prostitution in Aquinas' thought: yeah, prostitution is really bad, but if you try to suppress it, things will become far worse. The prudent statesman, while rightly repulsed by prostitution as such, humbly recognizes the limits of his power to constrain human vice through legal means. Of course, that doesn't mean that there aren't ways to encourage a populace toward self-reflection and self-discipline that don't involve legal suppression.
xracy 4y ago
> All we can say from these big trends is that we have no idea whether peer review helped, it might have hurt, it cost a ton, and the current state of the scientific literature is pretty abysmal. In this biz, we call this a total flop. "We can't say anything about this, so it's clearly a total failure." lol Seems like the author wants this both ways. Also, I don't know why they think replication would be more likely if we didn't have peer reviewers at all. Feels like a larger jump from "your study won't be reviewed" to "your study must be replicated" than from "your study is being reviewed" to "your study is being replicated".
larkinnaire 4y ago
There are some good points here, but the sweeping conclusion (presented with utter certainty) does not follow from them. It seems like the author's beef is with journals, rather than peer-review. If we did away with "peer review" today, journals would still have to operate the same way -- they'd still have many more submissions than they have room for, so a team of people (ideally, peers) would need to, uh, review those submissions according to some criteria. We can discuss whether the criteria should be adjusted, but I don't see how journals survive without gatekeepers. So, fine, he wants to do away with journals. Without a description of an alternative system, it sounds like the best researchers would just...upload their stuff to Arxiv and hope that someone reads it? Again, I'm not saying there is no alternative, but because he spends all his time arguing against "peer review", he spends no time discussing alternatives to journals that would solve more problems than they create. He addresses the question "can we fix peer review instead of replacing it" by discussing ways that fixes have failed in the real world. So what makes him think that a replacement would be easier? The "burn it all down and rebuild it according to my preferences" approach also doesn't have a great track record! And the certainty with which he states his conclusions gives me a sense that this is not someone who's super open to feedback.
- deleted 4y ago
  [deleted]
- larkinnaire 4y ago
  [response to a deleted comment] > The issue is too systemic at this point Ah, but there are also systemic issues that prevent Congress from doing the things you'd like them to do. For instance, Congress is vulnerable to money. Journals have money they can use to lobby. Are there other players with money and incentive to lobby for the "burn it all down" side? Which mountain of systemic issues is easier to climb? Sounds like an empirical question. Solving the problem by either approach is going to be really really complicated and full of compromises you won't like, and anyone who talks like their solution is super simple and obvious should be read with skepticism.
- renonce 4y ago
  He did make an example by publishing his paper on his own blog. So maybe every researcher can put a blog and upload papers there and expect them to appear in someone else's browser who followed the researcher because either he shared something interesting or someone else shared his paper? I've always wondered why academia can't work like social media. Not Twitter or Facebook, which is algorithm-controlled, blogs would work where everyone maintains their own sovereignty.
mgamache 4y ago
Most published medical research is wrong... https://journals.plos.org/plosmedicine/article?id=10.1371/journal.pmed.0020124 https://journals.plos.org/plosmedicine/article?id=10.1371/jo...
- Retric 4y ago
  Wrong is hardly the ideal metric here. Being less wrong than random can beat being correct and utterly trivial.
  - mgamache 4y ago
    The standard is a p value of .9 or .95. Not a flip of the coin.
    - Retric 4y ago
      That’s hardly the only way to be wrong. Suppose you made a mistake in the synthesis process and the dosage described doesn’t match the dosage revived by patients. Now the paper is incorrect but possibly still quite useful.
user3939382 4y ago
Here's a great talk on this subject from the world of medical science: "Financial Conflicts of Interests and the End of Evidence-Based Medicine" https://www.youtube.com/watch?v=z6IO2DZjOkY https://www.youtube.com/watch?v=z6IO2DZjOkY
Hmann01 4y ago
passwordoops 4y ago
Had one paper with 5 reviewers: 4 thought it was pretty good or awesome, the 5th not... With about 7 pages' worth of criticism that were mostly invalid. I resubmit, with modifications. That same week a paper of virtually the same study (not plagiarized, just coincidentally the same study) is published by a heavyweight 6 hours up the road from us. So reviewer 5 says, even if the modifications are good enough, this has already been done by a bigger, better researcher and is therefore not original. Paper gets rejected outright. So my supervisor does some digging, and guess who reviewer #5 just so happened to be... Frack academia
- broberts01 4y ago
  Academia is fundamentally broken. All the incentives are messed up and I have a hard time trusting anything that has been "researched" at all. Really disappointing.
- titzer 4y ago
  > guess who reviewer #5 just so happened to be... Write to the PC chair and the steering committee of the conference. Those kinds of actions do have consequences. At a minimum you should be able to request a conflict-of-interest review of the situation.
  - passwordoops 4y ago
    1- This was 2008 and the guy's retired by now. 2- Outside of engineering (or at least in the Chemistry/Bio field where I came from), conference submissions are abstracts not papers. There may sometimes be special issues where speakers are invited to submit a paper after the fact. But "paper" here means drafting the full manuscript and submitting it to the publisher when you're ready to be published in a future issue. So no steering committee, just an Editor-in-Chief who conducts research in the field. In fact referencing conference abstracts was always frowned upon because they never contained enough information to be properly scrutinized. Speaking of, this whole "reputable papers only happen with conferences" phenomenon was unheard of to me until I joined an engineering company
    - 4y ago
karaterobot 4y ago
> (When one editor started asking authors to add their raw data after they submitted a paper to his journal, half of them declined and retracted their submissions. This suggests, in the editor’s words, “a possibility that the raw data did not exist from the beginning.”) I'm not a researcher, but I've interviewed a lot of researchers in the context of data sharing. I'd go out on a limb and attribute this reluctance not to lack of any dataset whatsoever in most cases, but to a feeling of ownership over your "IP", and an instinct to protect it. In a niche scientific domain, it's not totally paranoid to worry that you could be giving your data to the competition. I heard this again and again, though mostly it was everyone else who did it, not the interview subject... There's also just the work involved in getting the data in shape to send to people, which can be a pain in the butt.
AlbertCory 4y ago
Excellent article. The "experts" very rarely take enough time to actually understand the paper and question its assumptions, and often aren't even qualified to do so. I had one article on the Xerox Star, way back in the day, that passed peer review for TOOIS with just some reasonable suggestions to cite related work. However, this one: https://papers.ssrn.com/sol3/results.cfm https://papers.ssrn.com/sol3/results.cfm (which was cited in an amicus brief to SCOTUS for the CLS Bank case) was rejected by CACM, by an "expert" review at Microsoft Research who admitted he knew nothing about patent law. Microsoft is a big believer in software patents, so he's hardly unbiased. His comments said at one point, because he talked to a patent lawyer "the patent law has changed!" while clearly not knowing that the laws on obviousness had not changed at all.
h-sg 4y ago
I did my time in the peer review trenches trying to give constructive feedback and reproduce results, only for editors to stop asking me to review or saying "don't waste this much time on this"
ynab4 4y ago
"God isn't real! Religion is a farce! Your imaginary sky daddy wont save you." Worships peer reviewed science journals and believes that AI and technology will save us Really makes you think.
bjornsing 4y ago
Scrap peer review and slash research funding. Saves taxpayer money, and saves science.
urthor 4y ago
The system before peer review wasn't particularly good either. Peer review is worst system possible, except for all the others that have been tried. I'm aware that many subfields have good solutions. But, from the outside as a non-researcher, nobody has conceived of a truly effective "whole of science" narrative so far.
YeGoblynQueenne 4y ago
>> (Only one of Einstein’s papers was ever peer-reviewed, by the way, and he was so surprised and upset that he published his paper in a different journal instead.) Yes, well, how shall I put it? Not everyone who attempts to publish a paper is Albert fucking Einstein.
rebeccapaul30 4y ago
I'm so happy to be able to share this testimony with all of you, I invested 6BT, in an online investment bitcoin platform. Scammers stole all of my belongings! I have abandoned everything! I was concerned about breaking my promises to my family this Christmas because I had expenses to cover. I talked to a handful of my friends, but all of their responses were unfavourable. When I went online, I came across an article promoting Spyware Cyber, A hacker group that specialized in cryptocurrency recovery. I made the decision to get in touch with them, and in three days, spyware(@)cybergal(.)com was able to recover all of my stolen bitcoins. It was amazing to have my lost Bitcoins back. Anyone who has likewise lost money in Bitcoin should read this post. and get in touch with them to retrieve back your stolen funds.

199 points by Vinnl 4y ago

I see a few major problems in the pursuit of modern scientific inquiry. - Bureaucratization. The modern university model is basically the DMV. It functions to sustain the lifestyle of the administrators in charge and to prevent them from losing power, money, status. And no one enjoys it, not the employees, not the customers, etc. and yet who can change it for the better? It can and does only get worse, more bureaucratic, more soul sucking. Ask the post-docs. - Feminization of academia and science. This is related to the bureaucratization. Process, safety, paperwork, meetings and community consensus are paramount. All becomes politics. Everyone must agree. Only small questions can be answered. You need permission for everything. Anything truly novel is considered a threat to the scientific community. Modern day "scientism" and other beliefs are the replacement to christianity in the west, and women are more religious (look it up), and academe is the modern day church. All dissent or inquiry is squashed. - Denial of Great Man Theory. Great men invented modernity, and of course our modern industrial state requires many layers of managers due to the huge complexity, so sizes of labs, colleges, corporations, assembly lines, supply chains have exploded, and no one person can manage it all in their mind. The false belief is thus: modern industrial society requires the managerial class to function, therefore the managerial class invented it, and it requires the managerial class to progress. Therefore great man theory, or the idea of the innovative genius, is supposedly debunked. And yet you see nearly all major advances come from tech bros like Elon Musk et al who are trailblazers, ignore complaints of the bureacracy, etc. - Could the university system have produced facebook, spacex, tesla, microsoft, apple? You can go on and on. Please don't interpret this as an attack on women or femininity. Society requires everyone for it to function properly, and everyone has a positive contribution to make, but we have to be able to iterate and change when we realize certain modes of endeavor simply do.not.work.

fastaguy88 4y ago
Interesting political philosophy, but little that speaks to the pursuit of science. Universities may be bureaucratic, but those bureaucracies have almost nothing to do with scientific agendas. Likewise, grant review panels work hard to find innovative, not consensus, proposals. Again, the main thing the management does is demand more external funding, it does not set scientific direction. And, of course, Facebook, Microsoft, etc is not science- at best it is engineering.
- logicalmonster 4y ago
  > Likewise, grant review panels work hard to find innovative, not consensus, proposals. I don't want to give specific examples of research topics to avoid poisoning the well, but could you argue with a straight face that an academic that wanted to do research into some topics that could be likely to yield certain types of politically incorrect conclusions wouldn't face extreme difficulty getting funding or extreme risk to their career? I think it's easy to see how much scientific research on certain topics could be stuck within a narrow range of opinion because people are more concerned with what gets funding or doesn't get them shunned.
  - fastaguy88 4y ago
    I do not know anything about grant panels outside my field of biology. But I am certain that the overwhelming majority of grant money is spent on scientific questions that have virtually no obvious political dimension. I’m sure there are grant applications that have a substantial political component, but I would be surprised if they accounted for even 5% of research funds. Social and political science receives a very small fraction of research funds, and while one might argue that allocations of health research budgets are politically shaped, viruses and oncogenes have no politics.
p0pcult 4y ago
>"Feminization"...[bad stuff]...[dubious, gender essentialization] >Denial of Great Men...[bad stuff] >Please don't interpret this as an attack on women or femininity
poulpy123 4y ago
> - Bureaucratization Bureaucratization is the consequence of wanting to control that taxes are "properly" used. > - Feminization of academia and science. That's abolutely moronic viewpoint if you ever worked in science > - Denial of Great Man Theory All scientific works, even by genius, lie on the works of others, predecessors or colleagues. > And yet you see nearly all major advances come from tech bros like Elon Musk et al who are trailblazers, ignore complaints of the bureacracy lol what scientific progress Elon Musk did except giving material for the sociology of internet trolls ? > - Could the university system have produced facebook, spacex, tesla, microsoft, apple? You're so close to understand that university and industry are not the same and have not the same goals
- p0pcult 4y ago
  Apparently, the sexism is rampant in here.
iosono88 4y ago

p0pcult 4y ago

Compelling arguments!

osigurdson 4y ago

I'm not sure about the pros and cons of peer review but if anyone is claiming that that are doing science, then they must publish raw data so that others can review it. This was the first thing that jumped out at me after reading "Accelerate" - so many claims that everything based on "data". I don't necessarily disagree with the conclusions in the book but I think really just a formalization of Google's development practices. The book should have been called "How we do things at Google".

phyrex 4y ago
Well in the book that actually has that content (and very nearly that title: https://abseil.io/resources/swe-book https://abseil.io/resources/swe-book) they explain that they set themselves up to follow the best practices found by the company that eventually published Accelerate, even before google bought them
- osigurdson 4y ago
  >> they explain that they set themselves up to follow the best practices found by the company that eventually published Accelerate Do you mean as far back as the early days < 2000 or did this happen well after they were established (> 2015). If the latter, how much actually changed?
- deleted 4y ago
  [deleted]

nixpulvis 4y ago

> Weak-link thinking makes scientific censorship seem reasonable, but all censorship does is make old ideas harder to defeat. Reduce the social, political, professional, and academic cost of "being wrong", learn to have a little more empathy and suddenly negative reviews will hold more meaning once again. People will learn to learn instead of learn to defend. (Of course, defense is a skill worth learning as well. Please don't take what I'm saying as this-or-that.) You simply cannot do away with peer review! What you can eliminate is the control strictures that exist today which punish failure unfairly. This is true at all levels, from the highest institution right on down to that conversation you had with a colleague over coffee yesterday.

bratwurst3000 4y ago
Thanks you nailed it
Vinnl 4y ago
That's the key here: if peer review is feedback rather than a barrier to sharing your findings and progressing in your career, then negative (though of course, in this situation review would no longer be a binary negative/positive) feedback could be considered input again. But the current setup of peer review, like this article is arguing against, assigns a high cost to "being wrong" (in the eyes of about two reviewers).
- nixpulvis 4y ago
  Right, but where is the threshold for wrongness which is so widely accepted as truth that it becomes harmful enough to be actively removed? If I began preaching of the correctness of 1+1=3 surely someone would deny my publication, no? Or, the critics would outweigh the proponents. How can this be accomplished without division or empathy?
  - ProjectArcturis 4y ago
    You'd still have peer review, but in a very different sense. Instead of a formal process by 3 reviewers, you'd have an informal process by everyone who read the paper. People would read your dumb 1+1=3 paper, realize that it was wrong, ignore it, and pay less attention to you in the future.

qudat 4y ago

Peer review incentivizes the worst features of humanity: it affords peers in your field the ability to reject your paper because they want to take the idea for themselves and then publish it.

photochemsyn 4y ago
In the corporate university, the usual game is to get the patent approved first, and hold off on the publication until that happens. At least that's the case in applied technical science - the social science world seems more like a club where the involved players just rubber-stamp each other's non-reproducible work in order to up their publication count and pack their CV. The actual work has no value, so why would anyone bother stealing it?
- sideshowb 4y ago
  If that were true we'd see more patent applications in, say, engineering, than journal papers in the same field. (Or at least around the same order of magnitude). Do the numbers check out?
  - ketzu 4y ago
    My experience was: Most papers are not useful for patents. Corporate funding might require a pre-publication review for patent worthyness, and if it is there, it is applied for before publication. Most researchers I knew had little interest in patents, so non-corporate funded papers got published without patent considerations. When it came up, it was a before-you-publish-think-about-it topic.
rubslopes 4y ago
I'm not sure that's a real problem. Papers go around a lot before being submitted to a journal. People would notice that on the spot.
- qudat 4y ago
  Shrug. I used to work in a lab where they talked about it happening. Purely anecdotal

fastaguy88 4y ago

Wow. Pretty clear this paper was not peer reviewed. Many of the arguments would not get past a high school English teacher. ‘A whole lot of money for nothin’ perhaps, but that money is not spent on peer review. Peer review doesn’t improve research productivity …. This reminds me of the argument that we should not spend more money on education, because education budgets have gone up, but test scores have not. The argument implies (but does not state, because of the absurdity), that spending less money would raise scores. And, remarkably (/s), peer review does not guarantee that all scientific results are reproducible and correct. As the author points out, no one has done the control. We don’t know how much worse the scientific literature would be without peer review, and we don’t no how many important insights are lost because of poor reviews. But as someone who has done a lot of peer review (and has had papers rejected with uninformed reviews), it is difficult to state how much worse things could be. (But perhaps that is an overstatement, as very few papers do not find their way into some journal). The scientific literature today is overwhelming. Peer review helps make it less so.

Vinnl 4y ago
> it is difficult to state how much worse things could be Because there's no proof. The only thing we know for certain, is that a lot of money is being spent on it, and that some of the mechanisms that are supposed to work can't work (i.e. in the case where people look at and trust preprints anyway). There will definitely be anecdotes of peer review having caught things (though it'll be harder to show that that prevented harm). But likewise, there are also many, many anecdotes that it has been actively harmful. No proven upside, limited anecdotal upside, and proven downside. If peer review as a publication gatekeeper hadn't already been a thing, we wouldn't introduce it — that would be considered unscientific. Sounds like something that's ripe for reconsideration.
- fastaguy88 4y ago
  I don’t understand how we know that a whole lot of money is spent on it. A lot of volunteer (unpaid) time is spent on it, but not much money. And I’m also unaware of the “proven” downside. But you might ask yourself, why do so many scientists waste so much of their time doing peer review. Perhaps they are ignorant, or easily duped, or perhaps they believe that peer review is a valuable use of their time.

photochemsyn 4y ago

Peer review is not the reason academic research has turned into a garbage fire - it's because of the corporatization of the university and the transition from the open research model to the patent-seeking profit-generating research model. The latter has its place, certainly, but before the 1980s that place was industry-funded private research institutions, the most famous and well-known being Bell Labs. These institutions were funded by groups of corporations who in return got first dibs on their patentable work, and the researchers and engineers in turn got good salaries and didn't have any teaching responsibilities. After Bayh-Dole passed in the 1980s, universities were allowed to grant exclusive licenses to their patentable research to private entities, and this resulted in the rise of the corporate univesity model - people running around hiding their data until they got the patent, people fudging results to make their novel drug look good so their startup would be bought by some pharma corporation, administrations being overrun by pharmaceutical executives who only cared about universities generating a steady stream of patents, and forget about quality-of-teaching. It's been a complete disaster, and not only that, the private research world has largely been defunded as corporations realized that, with the aid of politicians, they could just outsource basic R&D to universities, using grad students as cheap underpaid lab labor on projects where they're little more than glorified lab techs, and not really doing any original research. In addition, the whole system is increasingly authoritarian and censorship is rife - any academic who calls for the elimination of exclusive licensing and the adoption of open-source drug discovery will soon find themselves without a job. It's similar in many ways to the Lysenko era in Soviet science, just run by corporate authoritarians instead of Stalinists. Avoid if possible.

pdonis 4y ago
I would add that the funding source for most research now is governments, which means that research is now optimized for getting government grants instead of for actually advancing science.

cs702 4y ago

Whether the "peer review" experiment has succeeded or failed is debatable... but anecdotally, everyone I know in academia feels the current publish-or-perish peer-reviewed system is BROKEN: It incentivizes everyone to pursue low-risk, incremental, mostly inconsequential, easy-to-explain research with high certainty of short-term success, instead of high-risk, exploratory, consequential, hard-to-explain research with high uncertainty of long-term payoff. This is true: > From antiquity to modernity, scientists wrote letters and circulated monographs, and the main barriers stopping them from communicating their findings were the cost of paper, postage, or a printing press, or on rare occasions, the cost of a visit from the Catholic Church. Scientific journals appeared in the 1600s, but they operated more like magazines or newsletters, and their processes of picking articles ranged from “we print whatever we get” to “the editor asks his friend what he thinks” to “the whole society votes.” Sometimes journals couldn’t get enough papers to publish, so editors had to go around begging their friends to submit manuscripts, or fill the space themselves. Scientific publishing remained a hodgepodge for centuries. This is true as well: > That all changed after World War II. Governments poured funding into research, and they convened “peer reviewers” to ensure they weren’t wasting their money on foolish proposals. That funding turned into a deluge of papers, and journals that previously struggled to fill their pages now struggled to pick which articles to print. Reviewing papers before publication, which was “quite rare” until the 1960s, became much more common. Then it became universal. Finally, this is also true: > Now pretty much every journal uses outside experts to vet papers, and papers that don’t please reviewers get rejected. You can still write to your friends about your findings, but hiring committees and grant agencies act as if the only science that exists is the stuff published in peer-reviewed journals. This is the grand experiment we’ve been running for six decades.

inglor_cz 4y ago
"everyone I know in academia feels the current publish-or-perish peer-reviewed system is BROKEN: It incentivizes everyone to pursue low-risk, incremental, mostly inconsequential, easy-to-explain research with high certainty of short-term success, instead of high-risk, exploratory, consequential, hard-to-explain research with high uncertainty of long-term payoff." That is what I keep hearing from my academic acquaintances as well - When a measure becomes a target, it ceases to be a good measure. And this is what happened with publish or perish. Let's not forget that Katalin Karikó, a giant in the mRNA field, was demoted by UPenn in 1997 because they considered her work unproductive. But mRNA was precisely the topic that required a multi-decade commitment. Fortunately she was stubborn like hell. God know how many other people give up after being degraded like that.

poulpy123 4y ago

What a load of horseshit: * There are less great discoveries because there are less great and easy things to discover * Time has filtered the scientific studies of the previous centuries * You don't look at the efficiency of a filter (peer reviewing) by the amount of shit that pass through, but by the rating between this amount and the amount of shit that doesn't pass through * A lot of researchers don't want to release their raw data not because they invented their data but because they want to keep the monopoly for them * Peer reviewing has failures and could be improved, but it is the best and quickest way to judge a scientific work (with credentials let's be honest)

casey2 4y ago
I agree, that was a load of horseshit, especially bullet 1.
- Lichtso 4y ago
  I strongly disagree with the first point: As a society the more we discover, the more we increase the surface of the discoverable and the more we can combine things. It only gets harder for the individual because it takes longer and longer to learn everything necessary to get to the frontier of the unknown. It is not a problem of running out of viable research topics.
- janalsncm 4y ago
  I agree with the easy part. The low hanging fruit has mostly been discovered. If you want to invent something new you’re probably going to need a team of SMEs working on it.

notlukesky 4y ago

Humans can game every system. Then you can tweak the system to make it better for a period of time till it is gamed again. Rinse, wash and repeat. Same flaws exist with all ratings systems and many are just mathematical garbage.

toomim 4y ago
Not every system. Some systems can detect gaming, and fight back, like a body's immune system. https://docs.google.com/presentation/d/1SRWbyA2uCnda1JwAZ3JiDKFt1-tWxyjK_FgqUSkBPjc/edit#slide=id.g1508f4412b2_0_354 https://docs.google.com/presentation/d/1SRWbyA2uCnda1JwAZ3Ji... https://peeryview.org/about https://peeryview.org/about

lo_zamoyski 4y ago

Peer review in academia may very well be like the suppression of prostitution in Aquinas' thought: yeah, prostitution is really bad, but if you try to suppress it, things will become far worse. The prudent statesman, while rightly repulsed by prostitution as such, humbly recognizes the limits of his power to constrain human vice through legal means. Of course, that doesn't mean that there aren't ways to encourage a populace toward self-reflection and self-discipline that don't involve legal suppression.

xracy 4y ago

> All we can say from these big trends is that we have no idea whether peer review helped, it might have hurt, it cost a ton, and the current state of the scientific literature is pretty abysmal. In this biz, we call this a total flop. "We can't say anything about this, so it's clearly a total failure." lol Seems like the author wants this both ways. Also, I don't know why they think replication would be more likely if we didn't have peer reviewers at all. Feels like a larger jump from "your study won't be reviewed" to "your study must be replicated" than from "your study is being reviewed" to "your study is being replicated".

larkinnaire 4y ago

There are some good points here, but the sweeping conclusion (presented with utter certainty) does not follow from them. It seems like the author's beef is with journals, rather than peer-review. If we did away with "peer review" today, journals would still have to operate the same way -- they'd still have many more submissions than they have room for, so a team of people (ideally, peers) would need to, uh, review those submissions according to some criteria. We can discuss whether the criteria should be adjusted, but I don't see how journals survive without gatekeepers. So, fine, he wants to do away with journals. Without a description of an alternative system, it sounds like the best researchers would just...upload their stuff to Arxiv and hope that someone reads it? Again, I'm not saying there is no alternative, but because he spends all his time arguing against "peer review", he spends no time discussing alternatives to journals that would solve more problems than they create. He addresses the question "can we fix peer review instead of replacing it" by discussing ways that fixes have failed in the real world. So what makes him think that a replacement would be easier? The "burn it all down and rebuild it according to my preferences" approach also doesn't have a great track record! And the certainty with which he states his conclusions gives me a sense that this is not someone who's super open to feedback.

deleted 4y ago
[deleted]
larkinnaire 4y ago
[response to a deleted comment] > The issue is too systemic at this point Ah, but there are also systemic issues that prevent Congress from doing the things you'd like them to do. For instance, Congress is vulnerable to money. Journals have money they can use to lobby. Are there other players with money and incentive to lobby for the "burn it all down" side? Which mountain of systemic issues is easier to climb? Sounds like an empirical question. Solving the problem by either approach is going to be really really complicated and full of compromises you won't like, and anyone who talks like their solution is super simple and obvious should be read with skepticism.
renonce 4y ago
He did make an example by publishing his paper on his own blog. So maybe every researcher can put a blog and upload papers there and expect them to appear in someone else's browser who followed the researcher because either he shared something interesting or someone else shared his paper? I've always wondered why academia can't work like social media. Not Twitter or Facebook, which is algorithm-controlled, blogs would work where everyone maintains their own sovereignty.

mgamache 4y ago

Most published medical research is wrong... https://journals.plos.org/plosmedicine/article?id=10.1371/journal.pmed.0020124 https://journals.plos.org/plosmedicine/article?id=10.1371/jo...

Retric 4y ago
Wrong is hardly the ideal metric here. Being less wrong than random can beat being correct and utterly trivial.
- mgamache 4y ago
  The standard is a p value of .9 or .95. Not a flip of the coin.
  - Retric 4y ago
    That’s hardly the only way to be wrong. Suppose you made a mistake in the synthesis process and the dosage described doesn’t match the dosage revived by patients. Now the paper is incorrect but possibly still quite useful.

user3939382 4y ago

Here's a great talk on this subject from the world of medical science: "Financial Conflicts of Interests and the End of Evidence-Based Medicine" https://www.youtube.com/watch?v=z6IO2DZjOkY https://www.youtube.com/watch?v=z6IO2DZjOkY

Hmann01 4y ago

passwordoops 4y ago

Had one paper with 5 reviewers: 4 thought it was pretty good or awesome, the 5th not... With about 7 pages' worth of criticism that were mostly invalid. I resubmit, with modifications. That same week a paper of virtually the same study (not plagiarized, just coincidentally the same study) is published by a heavyweight 6 hours up the road from us. So reviewer 5 says, even if the modifications are good enough, this has already been done by a bigger, better researcher and is therefore not original. Paper gets rejected outright. So my supervisor does some digging, and guess who reviewer #5 just so happened to be... Frack academia

broberts01 4y ago
Academia is fundamentally broken. All the incentives are messed up and I have a hard time trusting anything that has been "researched" at all. Really disappointing.
titzer 4y ago
> guess who reviewer #5 just so happened to be... Write to the PC chair and the steering committee of the conference. Those kinds of actions do have consequences. At a minimum you should be able to request a conflict-of-interest review of the situation.
- passwordoops 4y ago
  1- This was 2008 and the guy's retired by now. 2- Outside of engineering (or at least in the Chemistry/Bio field where I came from), conference submissions are abstracts not papers. There may sometimes be special issues where speakers are invited to submit a paper after the fact. But "paper" here means drafting the full manuscript and submitting it to the publisher when you're ready to be published in a future issue. So no steering committee, just an Editor-in-Chief who conducts research in the field. In fact referencing conference abstracts was always frowned upon because they never contained enough information to be properly scrutinized. Speaking of, this whole "reputable papers only happen with conferences" phenomenon was unheard of to me until I joined an engineering company
  - 4y ago

karaterobot 4y ago

> (When one editor started asking authors to add their raw data after they submitted a paper to his journal, half of them declined and retracted their submissions. This suggests, in the editor’s words, “a possibility that the raw data did not exist from the beginning.”) I'm not a researcher, but I've interviewed a lot of researchers in the context of data sharing. I'd go out on a limb and attribute this reluctance not to lack of any dataset whatsoever in most cases, but to a feeling of ownership over your "IP", and an instinct to protect it. In a niche scientific domain, it's not totally paranoid to worry that you could be giving your data to the competition. I heard this again and again, though mostly it was everyone else who did it, not the interview subject... There's also just the work involved in getting the data in shape to send to people, which can be a pain in the butt.

AlbertCory 4y ago

Excellent article. The "experts" very rarely take enough time to actually understand the paper and question its assumptions, and often aren't even qualified to do so. I had one article on the Xerox Star, way back in the day, that passed peer review for TOOIS with just some reasonable suggestions to cite related work. However, this one: https://papers.ssrn.com/sol3/results.cfm https://papers.ssrn.com/sol3/results.cfm (which was cited in an amicus brief to SCOTUS for the CLS Bank case) was rejected by CACM, by an "expert" review at Microsoft Research who admitted he knew nothing about patent law. Microsoft is a big believer in software patents, so he's hardly unbiased. His comments said at one point, because he talked to a patent lawyer "the patent law has changed!" while clearly not knowing that the laws on obviousness had not changed at all.

h-sg 4y ago

I did my time in the peer review trenches trying to give constructive feedback and reproduce results, only for editors to stop asking me to review or saying "don't waste this much time on this"

ynab4 4y ago

"God isn't real! Religion is a farce! Your imaginary sky daddy wont save you." Worships peer reviewed science journals and believes that AI and technology will save us Really makes you think.

bjornsing 4y ago

Scrap peer review and slash research funding. Saves taxpayer money, and saves science.

urthor 4y ago

The system before peer review wasn't particularly good either. Peer review is worst system possible, except for all the others that have been tried. I'm aware that many subfields have good solutions. But, from the outside as a non-researcher, nobody has conceived of a truly effective "whole of science" narrative so far.

YeGoblynQueenne 4y ago

>> (Only one of Einstein’s papers was ever peer-reviewed, by the way, and he was so surprised and upset that he published his paper in a different journal instead.) Yes, well, how shall I put it? Not everyone who attempts to publish a paper is Albert fucking Einstein.

rebeccapaul30 4y ago

I'm so happy to be able to share this testimony with all of you, I invested 6BT, in an online investment bitcoin platform. Scammers stole all of my belongings! I have abandoned everything! I was concerned about breaking my promises to my family this Christmas because I had expenses to cover. I talked to a handful of my friends, but all of their responses were unfavourable. When I went online, I came across an article promoting Spyware Cyber, A hacker group that specialized in cryptocurrency recovery. I made the decision to get in touch with them, and in three days, spyware(@)cybergal(.)com was able to recover all of my stolen bitcoins. It was amazing to have my lost Bitcoins back. Anyone who has likewise lost money in Bitcoin should read this post. and get in touch with them to retrieve back your stolen funds.