21 ms·
Authors’ names have ‘astonishing’ influence on peer reviewers: study
- uptownfunk 4y agoHow is double blind not the norm? there is just too much room for bias
- ghaff 4y agoPast performance is a reasonable indicator in a lot of situations. I've had this discussion in the context of conference submittals. While there are lots of reasons to be inclusive of new presenters, the reality is that I also want to be aware as a conference committee member of applicants who have been very popular in the past.
- hedora 4y agoI've found the opposite. I don't want to hear version 10 of a paper from 2005. Besides, invited speakers are a thing for a reason.
- cperciva 4y agoIf you want a review to be truly double blind, you have to start censoring the paper -- things like "in previous work [1-10] we showed..." makes it obvious who the authors are even if you remove the names from the top of the paper.
- trepetti 4y agoYou do not have to censor: you simply refer to yourselves in the third person to make your shared identity plausibly deniable. A work that is worthy of being of publication should largely be able to stand on its own anyway. This may have the effect of reducing minimum publishable unit (MPU) CV spam papers as a bonus.
- cperciva 4y agoI'm talking about editors censoring papers. Authors could avoid this problem by writing differently, but as a reader I'd prefer they don't; knowing that a body of work is closely related is very useful, especially when it comes to deciding which references to follow up.
- trepetti 4y agoThe double blind process only requires that this be done during the review drafts. The final camera-ready version for publication can use the first person without compromising the integrity of the review process. I don't think the editors would remove self-referential text from a final draft after the paper had been accepted.
- gus_massa 4y agoUsually if you count who has more citations you get the author of the paper (or the advisor, or the leader of the team). Sometimes it's a lifehack to get more citations, but most of the times is just natural because someone of the team was working in tool X and other member of the team in tool Y and now you are adjusting all the details to make X and Y work together and get a new result.
- Calavar 4y agoThere are some computer science conferences that do this. If you cite your prior work, you are supposed to blank out the names in the citation and then attach an anonomyzed version of the prior work so that the reviewers can reference it if necessary. If the paper is accepted, you de-anonomyze the citation in the camera ready version.
- codethief 4y agoThis does not necessarily help. In many fields the reviewers will be familiar with the prior work and therefore recognize it, anonymized or not.
- hedora 4y ago"This work is based on the ideas of [1-10] which showed..." (Also, it is a major red flag to cite 10 papers from one group and no related work from anyone else. Either the topic is completely irrelevant, or you didn't do a cursory literature search - at least skim the references from papers you cite!)
- cozzyd 4y agoIn my field that would be impossible... people work in collaborations and you know pretty much all of the experiments.
- hedora 4y agoThat's usually a consequence of too much nepotism, or some super expensive piece of equipment. Double blind helps the former, though not the latter. In a part of my field that is capital intensive, some well funded newcomers have recently invested a lot, and "broke in", while some incumbent testbeds went away; the "expensive equipment" problem is usually temporary if the field is expanding.
- cozzyd 4y agoIt's just that experiments in astroparticle physics are expensive and have lots of people on them. Like, how is the IceCube Collaboration supposed to write an anonymous paper? Even the most cursory description of the detector would give it away...
- hedora 4y agoWell, maybe there will be many IceCube's later? (Probably not, but I'm routinely shocked when I learn about new giant testbeds in my field...) Alternatively, when IceCube 2 comes out, the old IceCube crowd might be focusing on other stuff, and not paying attention to the IceCube 2 politics. That makes them great peer reviewers (no horse in the race, but knowledgeable).
- cozzyd 4y agoThe proposed IceCube Gen2 is mostly a superset and only slightly disjoint with IceCube (for example, I am not an IceCube collaborator but I am on Gen2...). But the point is that for experiments that are larger than a few PI's, anonymity of authors is basically impossible (since all papers have everyone on them).
- cortesoft 4y ago
- anonym29 4y ago"A Nobel prizewinner is six times more likely than someone less well known to get a thumbs-up for acceptance, finds study." Lifehack discovered: legally change your name to that of a Nobel prizewinner if pursuing academia.
- TheSpiceIsLife 4y agoOld news from 2016 Jobseekers with Anglo-Saxon, easy to pronounce and common names are the most likely to get to the interview stage compared to candidates with unfamiliar names, according to research by the Australian National University published in the Oxford Bulletin of Economics and Statistics. https://www.independent.co.uk/news/business/news/unusual-names-less-likely-to-be-successful-in-job-interviews-research-finds-a6870276.html https://www.independent.co.uk/news/business/news/unusual-nam...
- respinal 4y ago"Nobel and Novice: Author Prominence Affects Peer Review" https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4190976 https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4190976
- deleted 4y ago[deleted]
- sportslife 4y agoSurely they didn't mean actual Anglo-Saxon names, right? Those name are so challenging to the modern english-hearing ear. https://www.behindthename.com/names/usage/anglo-saxon https://www.behindthename.com/names/usage/anglo-saxon
- triceratops 4y agoÆðelwulf wrote a heck of a quicksort on the whiteboard under the gun last week.
- TheSpiceIsLife 4y agoWhat would Žižek say? "No no no, within which historical context does it even make sense to ask this question? No? You see?"
- j7ake 4y agoIs it any different to podcasts, concerts, and books? People will choose what they listen, see, or read heavily based on who the performer is. A performer who has consistently given good content will obviously have a bigger pull than a nobody still trying to get their first break.
- dangom 4y agoPodcasts, concerts and books are something we review subjectively. Scientific articles one would expect would be reviewed objectively.
- nelsondev 4y agoThat expectation goes against human nature. Although I too wish it was true.
- saimiam 4y agoThe data in a paper can be objectively measured, the methodology used to conduct a study - like sample size, cohort composition, cohort groups chosen for comparison - could go both ways, the conclusions drawn are often subjective. If two of the three component parts of a paper are arguably subjective, not surprising that Nobel Prize winners get a pass on the quality of their papers.
- thrown_22 4y agoThe only people who could possibly think that are the ones who have never gotten anything published.
- themitigating 4y agoThe data is objective but the quality of the research is subjective. For example if my sample size was 150 for some study that's subjective whether it was sufficient of size
- hedora 4y agoNot true. There are objective statistical tests regarding sample sizes.
- didgetmaster 4y agoIt seems to me that some subjects are so specialized that the group that makes up one's 'peers' is fairly small. How do they have real anonymized reviews when it becomes easy to recognize the writings of the author? The more papers that someone writes, the easier it would be for their peers to recognize the writing style and other quirks that would give the author away.
- BurningFrog 4y agoFrom what I hear, in many fields the "peers" are better thought of as "rivals".
- deleted 4y ago[deleted]
- matthewdgreen 4y agoIt might happen unconsciously. But in my experience it's not that easy for a reviewer to consciously use writing cues for anything useful. Sometimes it seems very obvious that a given paper is by specific authors (either because of style, or because of how familiar it is with their previous work) but I've had many experiences where I later learned that my supposition was completely wrong. Similarly, when you encounter a paper that doesn't have any obvious cues (which is the overwhelming majority of them) then it's pretty much impossible to tell whether it's an author you admire or someone you've never heard of -- and this is a good thing. Some conferences don't use blind submissions, and yes: I have felt an awful lot of influence there. "Surely [famous Turing-award winning authors] don't need me to double-check their proof."
- yawnxyz 4y agoThis happens all the time in the field I'm in (phages, phage therapy) and the same names appear over and over. Peer review are anonymous but you can easily tell who's reviewed your paper or grant. I'm not sure how to fix this problem though. In our phage newsletter we try to avoid using names and universities to focus on the paper/topic/finding itself, but I keep finding myself looking at author names and affiliations before diving into any paper. I know it's "wrong" and I recognize myself doing it, but I still do it all the time.
- shaburn 4y agoWow. So branding works on human scientists. Shocking.
- insane_dreamer 4y agoThis reminds me of a bias I saw on Quora, where some popular contributor gets tons of upvotes by fans regardless of whether their answer is any good or not (often you have to dig deep into the comments to find out that it's not a good answer). (I should qualify that I don't know if it's still that way; I started using it when it was first launched and then quit 6-7 years ago once it started turning into a fanboi and you-upvote-me-I-upvote-you club.)
- aksjdhmkjasdof 4y agoWell, that's because Quora has distribution mechanics like the home feed and email digests that can amplify content by popular contributors. It's not like Reddit or HN or StackExchange where answers/comments compete directly against each other on the question page. I don't think the situation is analogous.
- SanjayMehta 4y agoIt was worse on Digg, before it "pivoted" into whatever it is now. https://www.wired.com/2012/07/mklopez-digg-power-user-interview/ https://www.wired.com/2012/07/mklopez-digg-power-user-interv...
- bscphil 4y agoIt's probably inherent to any site that allows you to follow individuals. For Twitter that's not much of a problem because most people know better than to rely on Twitter for life advice or product reviews. Letterboxd, on the other hand, manifests exactly the problem you're talking about, and the whole point of the site is to assign purported ratings to film and TV shows. Reddit and IMDb are mostly immune. Which is not to say that they're immune to all problems (astroturfing, brigading, personal biases, etc.).
- Semaphor 4y agoReddit seems to try and change that, though? I don’t really follow what development happens on new reddit, but it seems more user-oriented?
- bergenty 4y agoThat’s what we want in society right? Perks for doing hard work and then being recognized for it. That’s what “street cred” is. Maintaining a reputation is hard work and the fast pass and acknowledgment from the broader society is your reward. As in this person has more than proved himself, let’s not vet him as much.
- TheRealPomax 4y agoThis is literally how one of the biggest scandals in academic publishing occurred[1], so... no? Maybe it's a nice way to treat experts when asking them for their opinion, they are experts after all, but we absolutely don't want to give celebrities more leeway than unknowns in research papers where new claims are being made. You better have done the work properly, again, and again, and again. No free passes, no blind trust. [1] https://en.wikipedia.org/wiki/Diederik_Stapel https://en.wikipedia.org/wiki/Diederik_Stapel
- Null-Set 4y agoIn this case it looks like his good reputation was falsified from the start? They were unable to prove misconduct in his dissertation only because the data had been destroyed. So consistently missing his misconduct would be more to blame here than riding on a reputation.
- TheRealPomax 4y agoBut that ignores the years prior where he had enough reputation to be scrutinized less, which lets him publish more, which made him more famous, which got him scrutinized less, allowing him to keep publishing, boosting his reputation, [etc]. Right up to the very end, this man was able to publish like a celebrity because no on questioned it because he was famous. So let's not bake that into the process: scrutinize all papers equally. Experts don't get a free pass, if they have new claims to make, those claims are just as "I don't believe you yet" as anyone else's.
- hackerlight 4y agoNo. You get gatekeeping and laziness and stale group think and unstated quid pro quo and discrimination. It's worse science. A famous name getting the same paper published easier is a failure of peer review. The whole point of science is that ideas and evidence stands on its own merit. Not celebrity or seniority or power or any other axis that doesn't matter.
- hardtke 4y agoFairness and efficiency are often at odds. As a reviewer, I was sometimes asked to review papers from unknown authors at unknown institutions. My level of effort was likely proportional to how well I knew the quality of the institution from which the paper came. There is a great deal of time involved in understanding a paper well enough to do a full review.
- elcritch 4y agoIt's true, unfortunately. When I was a TA and graded homework I'd learn which students did well over time and generally would put less effort into checking their work because generally it was more likely to be correct. They were also more likely to self-correct, and to present work well, etc. Reviewing papers isn't really too different.
- gms7777 4y agoI agree. I'm in an applied computational field, that attracts a lot of computational and mathematical researchers that don't have knowledge of the applied system. I've learned from experience that often researchers new to the field make a whole lot of poor assumptions and modeling decisions, even if the mathematical part is fine. For better or worse, I'm going to more heavily scrutinize the details of a paper from a lab I haven't heard than I am one that I believe has a strong track record. That doesn't mean I'll give a pass for the latter, I'll still read it with a critical eye. But certain types of trust build up over time.
- spoonjim 4y agoI would expect a much higher than sixfold increase in paper acceptances by winning the Nobel Prize. Why wouldn’t there be a massive lift? Even if you can’t personally see why this result is important you know that one of the biggest contributors to the field thinks it’s interesting. That’s a pretty credible signal in a noisy world.
- aussiesnack 4y agoThat an economist finds human behaviour that every normal human would expect "astonishing" is itself not remotely astonishing.
- nequo 4y agoPeer review is not supposed to be a Twitter popularity contest so these numbers are indeed astonishing.
- aussiesnack 4y agoYet most non-economists would have predicted exactly this. If it's astonishing to you, well, perhaps you've had the misfortune to learn to think like an economist (ie. poorly & very muddy about normative vs actual).
- nequo 4y agoIt would be possible to respond to your comment in earnest but ad hominem like this is more of a psychological cushion than an invitation to genuine discussion.
- aussiesnack 4y agoIn what world is a daft attempt to psychoanalyze your interlocutor not ad hominem? My comment was snarky but not remotely playing the man. I meant it quite literally as a criticism of the thinking at issue. Economistic thought is riddled with confusion, and being surprised that scientists don't behave normatively (ie. not as those engaged in peer review are "supposed to") exhibits perfectly one variant of said confusion. I'm not 'astonished' that non-blinded peer reviewers are influenced by social status. No-one I know would be surprised at all, let alone 'astonished'.
- hedora 4y ago> I'm not 'astonished' that non-blinded peer reviewers are influenced by social status. I am astonished, in the same way I would be astonished to find out that papers that smell like fish are more likely to be accepted for publication at the majority of peer reviewed venues, and that the reason is that reviewers for those publications let their house cats stack rank the submissions.
- anm89 4y agoIs anyone surprised by this?
- themitigating 4y agoWhy weren't you?
- anm89 4y agoMuch of academia seems to have become an outwardly transparent no hold barred status battleground. So this just seems to exactly align with my expected outcome given that assumption. I'm not saying there aren't people in academia who aren't driven by doing good research but I certainly don't see that as the driving force in the US academia.
- themitigating 4y agoDo you have proof of this?
- anm89 4y agoYes I've got objective proof of my qualitative opinions on sociological topics. Let me just dig that right up for you. I didn't realize that HN was an academic journal where all statements should come with proof. Do you have proof that I'm wrong?
- deleted 4y ago[deleted]
- hedora 4y agoI'm surprised that there are still non double blind publications that count for tenure cases. This does explain some of the recent and embarrassing [lack of] retractions of Nature papers though.
- nequo 4y agoIt is nearly impossible to enforce double blind peer review because of the existence of online pre-prints (hosted on arXiv, SSRN, or just the authors' websites). A referee can copy and paste from the text into DuckDuckGo and find the authors.
- rgbrgb 4y agoIn this case an easy 50% solution seems better than nothing. Some people would just comply with the guideline, some would not think of your workaround and it would make peer reviews a little more equitable even if many broke the rule.
- hedora 4y agoI think you are overestimating how much reviewers care which grad student wrote which paper, or who their advisor is. Reviewers are generally overworked (for other reasons), and the main goal is to be constructive and somehow not sound like an idiot (in front of the other reviewers, who might be subject experts and definitely know who you are!) in any of the pile of reviews you need to write after reading a pile of papers. Also, reviewers declare conflicts ahead of paper assignments, which makes accidents less frequent.
- abdullahkhalids 4y agoBesides the sibling comment, in a lot of fields, people just know which group a paper is coming from even without doing any sort of searching or looking at preprints.
- bpodgursky 4y agoYeah it's often pretty obvious that a paper extends / is incremental on a previous body of work (maybe the same code, maybe same dataset) such that it could only have come from one lab.
- prepend 4y agoThat’s odd. I don’t do a lot of peer review, but when I do the authors names are masked. Sometimes I can figure it out based on the citations, but I haven’t reviewed any papers where the author’s name is listed. That being said an author isn’t published randomly in Nature, so I expect subsequent papers from an author to be better, on average, than non-Nature published authors.
- version_five 4y agoElon Musk tweets something stupid and it gets discussed. You or I do it and it's ignored. Celebrity is a thing, nothing odd going on
- quickthrower2 4y agoDiscussed is not the same thing as reviewed though. I agree it is not odd. It is a good thing to know about though and be mindful of. It might be advantageous in closed circumstances, i.e. you can skim a review of a more trusted developer.
- themitigating 4y agoWho wasn't mindful that the concept of bias and popularity exist?
- quickthrower2 4y agoI think many people. Everyone knows. Not everyone is mindful.
- refurb 4y agoWe yeah. At least in science, it’s a small world and reviewers often know the authors personally. And your reputation follows you. If it’s a big-name lab who has an amazing track record, you’re going to review the paper in the context of their entire body of work.
- nashashmi 4y agoI saw a tweet by a researcher who sent her work for peer review and got feedback that she should read more works of xxxxx and revise her work. Great feedback, she thought, because She was xxxxx.
- raister 4y agoSometimes the peer review is the one asking for citing him/hers work, not anybody else's...
- chatterhead 4y agoWe have to come to terms with this. Academia was bought and paid for long ago and the money was used to build an incredibly broken and overly political bureaucratic engine of scholarly and scientific work that doesn't get anywhere near as much peer-review scrutiny as it should and commands far more respect in politics and legal proceedings than we should allow. Universities and experts are the best we can do sometimes so we have to rely on it, but it doesn't mean it's truth or absolute and people like to use it as if it is to sell ideas like global warming instead of educating people on climate change.
- barbariangrunge 4y agoThe 2 academics I know aren’t bought and paid for. Either could make far more money in the private sector than they make now. They work in their fields out of love of research. This was my impression of most of my professors years ago too, especially as we saw so many quit for high paying engineering jobs at companies year to year — the ones who stayed aren’t in it for the money I suspect it’s the same at the top: most senior administrators I bet would make more in senior Fortune 500 jobs
- themitigating 4y agoDo you have any evidence of this?
- Test0129 4y agoHN isn't a journal and so academic sources of evidence really are overkill. However, I'll throw you a bone. Go to your favorite major journal and start looking at the "conflicts of interest" and "grants" section. Once you take money from someone (except for NIST, in my experience) you're basically beholden to try your hardest to get the results they're looking for. Some scientists are moral enough to still return bad results. A lot of scientists aren't. There's a lot of garbage out there, and the worse the journal, the more garbage it gets. Famously, Phillip Morris studies "passed" the scrutiny of several major journals. It's amazing what greasing a few palms will get you.
- 4y ago
- Nokinside 4y agoEvery paper Albert Einstein submitted to anonymous peer review was rejected. (he had only one peer anonymous reviewed paper and it had an error)
- themitigating 4y agoI'm confused because of your second sentence but according to this site https://mindmatters.ai/2020/05/einsteins-only-rejected-paper/ https://mindmatters.ai/2020/05/einsteins-only-rejected-paper... ". Albert Einstein only had one anonymous peer review in his career — and the paper was rejected2. This happened in 1936." How is just one paper submitted anonymously being rejected an indication of a trend?
- Nokinside 4y agoThere was no trend.
- shadowgovt 4y agoI'm not sure what we're supposed to find astonishing about academics (like most of humanity) being vulnerable to reputation biases in their reasoning. As in, that would be the null hypothesis. It would be astonishing if most academics overcame it. EDIT: apparently, the size of the effect (factor of six more likely to be accepted if it came from a Nobel Prizewinner) is larger than anticipated.
- themitigating 4y agoAre people in general bias based on reputation?
- shadowgovt 4y agoYes. In fact, people are so likely to make assumptions about future performance based on past performance that we invented a word for that social construct: "reputation."
- thenoblesunfish 4y agoYes, because it's easier to do that than assess people on their merits. This is especially true when you are reviewing a technical paper (often for free) when you have competing demands on your time and mental energy (research, teaching). It takes hours and hours to properly review a paper, and that is still trusting that the author is acting in good faith - to reproduce their work might in some cases require a large amount of time and money. These are the same reasons that letters of recommendation (written by people you've heard of) are so useful in academia. Lots and lots of bias problems, but the bias isn't the point, as I think is sometimes implied.
- MichaelCollins 4y agoIn general? There is no a single person to ever live without that bias.
- magicalhippo 4y agoA while back I subscribed for The Economist on a whim. I'm not really into economics, but I felt they provided a nice alternative view of what was going on elsewhere in the world compared to what the local news here in Norway reported. One thing I quickly noticed was that the articles were never signed by the authors directly. I found this striking, but also refreshing. It caused me to pay more attention to what was said. And, as I later found out, that seems to be the main intention[1]: The main reason for anonymity, however, is a belief that what is written is more important than who writes it. In the words of Geoffrey Crowther, our editor from 1938 to 1956, anonymity keeps the editor "not the master but the servant of something far greater than himself…it gives to the paper an astonishing momentum of thought and principle." [1]: https://www.economist.com/the-economist-explains/2013/09/04/why-are-the-economists-writers-anonymous https://www.economist.com/the-economist-explains/2013/09/04/...
- LtWorf 4y agoOn the other hand, if an author is consistently bad, signing would allow you to skip the content.
- wallmountedtv 4y agoNorway really has a huge issue of sensationalism in local papers, and it does not help that even the government funded NRK is deep into sensationalism as well. Opening NRK right now gives "Now the government must see the madness in this", "Electricity prices can make going to the cinema more expensive", and "He is so tired" as the top articles for today. I follow so much International/American news thanks to publications like Reuters, AP, and Qz. But no matter how much time or money I use, I can not follow my own country's news. I definitely am not alone in this, and it terrifies me how unworried most of Norway seems to be about this fact as well. Its a ticking time bomb.
- 29athrowaway 4y agohttps://en.wikipedia.org/wiki/SCIgen#Schlangemann https://en.wikipedia.org/wiki/SCIgen#Schlangemann > In 2008, in response to a series of Call-for-Paper e-mails, SCIgen was used to generate a false scientific paper titled Towards the Simulation of E-Commerce, using "Herbert Schlangemann" as the author. The article was accepted at the 2008 International Conference on Computer Science and Software Engineering (CSSE 2008), co-sponsored by the IEEE, to be held in Wuhan, China, and the author was invited to be a session chair on grounds of his fictional Curriculum Vitae.
- emacsen 4y agoWhen I was an undergrad in college, I helped design a study on the role of gender perception in expertise. We had a piece of text that the subjects (undergrads) would read and rate the expertise of. We had the same text but we randomized whether the name would be a commonly male name, a commonly female name, or initials. We also randomized if they'd watch a clip of men's sports beforehand (men's basketball), women's sports beforehand (women's basketball), or no sports. And finally we randomized whether the person giving the subjects instructions would be a young woman (one of my classmates) or a young man (me). Our small study showed what you'd expect- students- men and women students rated male writers more highly than female writers, and initials were right in the middle, though they tended to be more like responses for men. The tester's gender made no difference that we found. The sports thing made a measurable difference, but it didn't reverse the skew. My 18/19 year old self was somewhat skeptical we'd find a difference, but I was totally wrong. It taught me a lot about bias and perception, which this study also shows.
- bitexploder 4y agoAnother interesting bias along those lines: https://en.m.wikipedia.org/wiki/Women-are-wonderful_effect https://en.m.wikipedia.org/wiki/Women-are-wonderful_effect I imagine these biases swing in all sorts of directions depending on the context. Some are intuitive, many are not.
- deleted 4y ago[deleted]
- ganesh7 4y agogood point.
- SapporoChris 4y agoHow do you reconcile the https://en.m.wikipedia.org/wiki/Women-are-wonderful_effect https://en.m.wikipedia.org/wiki/Women-are-wonderful_effect with https://en.wikipedia.org/wiki/Gender_pay_gap https://en.wikipedia.org/wiki/Gender_pay_gap
- Test0129 4y agoVery common for anyone who has ever spent time in academia. When I was still in grad school I noticed a trend in some major mathematical and CS journals where a lot of time it would be the same N people writing articles. This was driven home when my advisor told me "without such and such's name this paper has no chance of making it past review". Non-sense. However, for all the people that say "trust the science" this is the science you're working with. It's politics all the way down just like everything else. For example, if Knuth decided to post some utter garbage chances are it would accepted based on his name alone.
- themitigating 4y agoYou trust science because it has the highest probability of being accurate. What's the alternative? Most people aren't capable of analysis scietific research to see if it's correct. What makes something political? If new research shows that X product is dangerous and republicans, who that company donates to, come out and disagree with the research they just made it political. Now, someone like you, can just say "This research is political" You've created a way to dismiss scientific research simply by disagreeing with it
- colordrops 4y agoYou are completely misinterpreting the meaning of GP's comment. First, they are not talking about government politics, but rather interpersonal politics. Second, they are defending science, and saying that it shouldn't be dismissed for political reasons, rather than the opposite.
- mattwilsonn888 4y agoJesus Christ, the point of science is to REMOVE trust - not be the least worst option. Science that cannot exist without trust is faith.
- fsflover 4y agohttps://lemire.me/blog/2020/07/12/science-is-the-belief-in-the-ignorance-of-experts/ https://lemire.me/blog/2020/07/12/science-is-the-belief-in-t...
- whatever1 4y agoOf course it does. Professors with high caliber research run tight ship and their labs publish consistently high quality work. It does not mean that reviewers lower their bar when assigned a paper from a famous professor, they just know a priori that the work will not be completely scam.
- jll29 4y agoThis is true even when grading school exams: teachers recognize the names of "bright" students and lean towards more positive grades. That's why anonymous grading - and in scientific publishing, double-blind peer review is so important. It's part of the scientific progress just as much as replication, the attempt to re-produce results of a study post-publication by other groups (I wish papers' PDFs had a QR code to a Web page that said "double-blind review by x people, replicated by y groups" - the latter changes over time so it's better tracked externally).
- indus 4y ago“Association of popular/celebrity angels with your startup increases the odds of your Series_A fundraise round.”
- jedberg 4y agoIt blows my mind that peer reviews aren't done blind. Not only is there this geneder bias but also bias of "I know that person they're famous I'm sure their work is good" or "I've rejected their paper in the past". Seems like an easy fix would be to make all peer reviews blind. In fact, for any study that gets federal funding, they should have to publish their hypothesis ahead of time into an escrow system, submit their paper to the same system, and then get blind peer reviews (blind in both directions) where the reviewer gets to see only the initial hypothesis and the paper with the names/institutions removed. And of course all the papers should be available for free. Maybe the government could pay reviewers directly for their time, but I haven't thought that one through yet.
- CodeSgt 4y agoAs a rule I'm generally not a huge fan of taxation, but I'd be happy to see my tax dollars go towards a system like you propose here.
- hannob 4y ago> It blows my mind that peer reviews aren't done blind. I've co-authored a few papers and blind reviews have some surprising consequences, like discussions along the lines of "you can't say that you already wrote about this issue elsewhere and put a link in the paper, because that would unblind it". I was a bit uncomfortable with that, because I like to "cite my sources" (even if the source was myself in this case). This also points to another issue: The more specialized the paper is the less likely the blinding will work. If you know a field by heart then you know who works on what, and probably can guess most authors based on that. I'm not saying I'm against blind review, but while it sounds obvious, it has some issues in practice.
- jedberg 4y agoWhoever told you that about your cites is probably right -- you shouldn't cite yourself while calling out that you are citing yourself. You should cite yourself the same you that site any other work. The second part is probably unavoidable. If you're working on something super specialized when all your reviewers are the six other people who work on it then sure, it'll be unblinded. Not much we can do there sadly.
- keithwinstein 4y agoDouble-anonymous/blind reviewing is completely standard in most areas of computer science, e.g. networking (SIGCOMM, NSDI, IMC, HotNets, MobiCom, MobiSys), systems (OSDI, SOSP, USENIX ATC, HotOS), security (Usenix Security, S&P), machine learning (NeurIPS and ICML), graphics and HCI (SIGGRAPH and CHI), and at least some top-tier theory conferences (like FOCS). So I think most computer scientists would agree with the article's conclusion that double-anonymous reviewing, while flawed, is better than the alternatives. I don't think I've done a non-anonymous submission in 10 years, and as a reviewer, usually I don't have a strong guess about who wrote the paper (and when I think I know, often I turn out to be wrong). It's a little annoying that this news article in Nature Magazine ignores the longstanding widespread prevalence of this practice in a conference-driven (but... we like to think important) academic discipline. :-) But: (1) I don't think the big benefit of double-anonymous reviewing is that a lousy paper from a Nobel laureate (or, from CMU/Berkeley/MIT/Stanford) isn't let in unfairly. To me the big benefit seems to be that reviewers have to review every paper as if it might be from their friends or a famous person, and consequently have to review each paper with due care and try sincerely to understand its contribution, which maybe they wouldn't do if they knew it's from some random place/author they've never heard of or don't think highly of. I think the way it affects judgment may be more in equalizing time/effort spent to read and understand a paper (and the generosity you give a paper because "maybe" it was written by your friend or somebody you respect), rather than a straight-up bias towards liking whatever the faculty at a famous university are writing about this year. (2) While I do think it's true, and a good thing, that double-anonymous reviewing helps "marginalized groups of authors who often struggle to have their work see the world" as the lead researcher says, we should probably acknowledge that authors are not the only beneficiaries of a scientific publication. The interests of the reader matter too -- the journal or conference has some duty to serve them. On the margin, maybe some readers would be more interested to learn what Albert Einstein is thinking about these days, or would like to see a well-balanced conference program that includes a good talk by a known-provocative speaker, instead of one more random (but adequate!) paper from a nobody. I'm not saying we should give a huge weight to this -- it's fine to make people eat their vegetables, but, I don't think we should act like scientific publication is only to give authors a line on their CV and the readers' preference is 100% irrelevant. A scientific journal shouldn't exclusively serve the authors. (Other kinds of media care way too much about what the reader wants, e.g. Facebook giving you whatever it thinks will keep you clicking things on Facebook, but there is probably a happy medium somewhere.) (3) The challenging frontier may be in grant submission and reviewing, where proposers are typically not anonymous to the reviewers, which surely leads to some biases. I have heard about government programs where they did use double-anonymous reviewing and it seemed weird to me. (Probably this is a situation where track record should matter, yet trying to summarize your own track record while remaining effectively anonymous seems really hard...)
- daguava 4y agogreg rutkowski and alphonse mucha are about to become top-tier scholars
- stevebmark 4y agoAlso relevant: "Cognitive ease" from "Thinking Fast And Slow." We often attribute information as higher quality if we're able to recall related information with more ease. Importantly, this happens regardless of the quality of the information. One study they point out is people exposed to random nonsensical words later ranked the words they saw the most frequently as being "better" than ones they saw less. It sounds like a related bias of automatic thinking here. There's some cognitive ease in recognizing the author, so your bias kicks in that this is better than if you didn't recognize the author.
- deleted 4y ago[deleted]
- ogarten 4y agoThis shouldn't be surprising to anyone who has ever worked in academia. Certain names have it much easier to publish than others. It's a self-fulfilling prophecy. Double-blind reviews aren't common in all fields (e.g. everything IEEE) which leads to a lot of bias right from the start: Seniority, country, university, ...
- lpolovets 4y agoThis is how VC firm halos work. When Sequoia or Benchmark invest, tons of investors want to squeeze into the round -- almost regardless of the fundamental attributes of the company.
- kleiba 4y agoI've done research at one of the top universities for NLP in Europe and there's a commonly held believe amongst many of my peers that conference acceptance has more or less degraded to a random process.
- popol12 4y agoAfter reading the article, I feel like the HN title should rather be « Authors’ status » instead of « Authors’ names ». I expected the study to be about gender inequalities but it’s actually about author prestige (is the author famous or not). I often just read HN title without reading the underlying article, and I wonder how many times I’ve got wrong in my interpretation of what the core of the information was
- papito 4y agoI seriously wonder how people would receive the Go language if they had no idea about the people behind it.
- kristopolous 4y agoI wonder if anyone changed their name to something like "Maxwell Einstein" or "Shannon Turing" for an advantage
- techas 4y agoGood that the study is done and it gets published. But the result is obvious and well known. The terrible part is that the solution is also well known (double blind process) but there is no interest to implement it in most journals. Editors have to defend their field from outsiders… editorials do not care as long as money keeps flowing in…
- a3_nm 4y agoThis is an interesting article but it contributes a bit to terminological confusion about reviewing modes. First, the article talks about "double-blind reviewing" as being "logistically hard" because reviewers can "find the paper in a Google search", and talks about a "price tag". By contrast, many conferences around me implement so-called "lightweight double-blind" reviewing, which takes zero effort and just means that the authors and affiliations are not mentioned on the copy of the paper that reviewers read. I think this form of double-blind is a no-brainer, eliminating some bias with essentially no downside; and that discussions about the cost and complexity of double-blind reviewing are a distraction from this immediate improvement. Here is a typical paragraph from a call for papers (here, STACS 2023) describing the policy: As in the previous two years, STACS 2023 will employ a lightweight double-blind reviewing process: submissions should not reveal the identity of the authors in any way. The purpose of the double-blind reviewing is to help PC members and external reviewers come to an initial judgment about the paper without bias, not to make it impossible for them to discover the authors if they were to try. Nothing should be done in the name of anonymity that weakens the submission or makes the job of reviewing the paper more difficult. In particular, important references should not be omitted or anonymized. In addition, authors should feel free to disseminate their ideas or draft versions of their paper as they normally would. For example, authors may post drafts of their papers on the web, submit them to arXiv, and give talks on their research ideas. Second, the article talks about "open review" as meaning "everyone’s identity is public". This is sometimes what the term means, but not always -- for instance the OpenReview.net platform <https://openreview.net/ https://openreview.net/> supports forms of open reviewing where the reviewers are still anonymous. Here "open" means "the discussion happens in the open, and everyone can post comments about the paper and reviews and contribute to the discussion". Here again, I feel that the discussion about "open reviewing" with non-anonymous reviewers is drawing attention away from this model which looks like a net improvement over the status quo.
- epolanski 4y agoAs a former researcher with multiple high-impact publications I can 100% confirm it: - when I was a researcher in Italy for an unknown lab, all of our articles were generally very scrutinized and went through years of reviews before publication - when I worked in Michal Graetzel[1]'s laboratory things get published much more easily on more higher impact journals with less scrutiny. Not only most of the publications didn't really add much to the scientific knowledge, but they didn't even have the very high standards required when I was in Italy Why is this bad? You get more funding the more you publish...So labs that publish easier due to the names involved get much more money...which they can use to publish even more and get even more money...which they can use to publish even more... There's excellent scientists anywhere, really. But funding, fame and politics are extremely asymmetric in academia. [1]https://en.wikipedia.org/wiki/Michael_Gr%C3%A4tzel https://en.wikipedia.org/wiki/Michael_Gr%C3%A4tzel
- jonathanstrange 4y agoI don't understand this. Don't you have blind peer review? In my discipline, most journals even have double blind peer review. I thought that's the standard.
- triknomeister 4y agoMathematics doesn't have double blind peer review unfortunately. It will help in a big number of cases, even if some cases fall off (for example, if you do experiments on a well-known software which only one lab works on, then double blind will have a limited impact.)
- KvanteKat 4y agoAs the linked article points out, practices can vary widely across fields (and in fields where preprints are available, maintaining authour anonymity in a double-blind setting furthermore relies on revieweres not having come across a preprint of the paper that is being reviewed which in turn can easily happen in smaller research-communities).
- epolanski 4y ago
- mfsch 4y agoI’m a bit conflicted about the goal of separating the “hard, cold, objective” science from “soft, warm, subjective” things such as reputation and trust. It may be the case that some publications can be perfectly evaluated based purely on the text of the manuscript, but in most of the work I’ve read and written there are countless little details that are not discussed in the text. The manner in which the authors have handled such details can greatly affect the quality of the research. How carefully did they write the one-off script that produced the figure? How closely did they follow the study protocols? How much did they look out for issues that might undermine the results? Such an evaluation may only be possible after following the work of an author over time and perhaps through direct collaborations. In practice, we may get better outcomes if peer reviewers completely ignore these aspects and evaluate all papers purely based on what’s on the page. But I don’t think it is obvious that any reliance on trust and reputation should be derided as bias to be eliminated.
- minifridge 4y agoSkimming through the article, it fails to mention that journals have some power to assess systematic reviewer biases based on their past history. If reviewers are systematically more permissive towards specific people's work, then good science is not their objective. So, essentially it can be viewed as a diversity problem in the reviewer space. The journals have important responsibility on this and they can not present as neutral observers of this phenomenon where they say : "oh every peer reviewing model has problems " and blame reviewers ethos that they are choosing.
- seydor 4y agoThis is not surprising of course. What has changed probably is that science over the past 20 years has become more "social", but not in a good way. Since there are so many good scientists everywhere, people get ahead by using any means to make their science more public, mostly through conferences, organizing conferences, pursuing various committees and subcommittees, befriending journal editors, the press, and of course social media plays a part in this, but not alone. Antisocial people will have a hard time. Where to start here? What do we want the purpose of publishing and peer review to be? Out of all this publicity dance, which part gets distilled into the solid foundation of science? I do think this whole journals/publishing/conference/apply-for-funding thingy is a bit too ancient and incremental, and more radical solutions would be nice to try. I think fundamental to this is the structure of funding, do we really want small funds going to individual small PIs, or maybe more independence, or less independence ...
- jwmoraes 4y agoIsn’t the article confusing correlation with causation? I read like there’s a correlation between acceptance rate and author’s pedigree, but the article sounds like the pedigree is the cause of the high acceptance. I’m no expert, so please correct if I’m wrong. But for example, how likely is that a Nobel prize winner produces worth-publishing research? Or how likely he is simply good at the skill of paper writing?
- jsemrau 4y agoThis is known. Hence, when we were building the peer-reviewed moderation / curation function for finclout we only show the content. While in selected cases the poster is identifiable, for most users the decision is done only based on content.
- throwaway5371 4y agoalso has astonishing influence to get invited to a job interview i had a friend who changed their name in their CV and got 90% success rate, and was close to 10% before
- kingkawn 4y agoWait a minute, does this mean that the entire conceit that this culture produces a superior form of truth is a hollow lie? Duh
- cycomanic 4y agoI find it ironic that this is published in Nature. In may experience Nature journals are amongst the worst with respect to peer review. In my field they seem to draw from a very limited set of reviewers (often senior academics who have not worked in the specific field in quite a while). We have been criticised with very outdated information. This is worse because editors are not experts, we had a reviewer contradict textbook established science and when we asked for an additional reviewer, they send it back to the same person. Even worse, their goal is not publishing good science, they want to sell journals. While they will never publicly admit it, I know that they take the into account the reputation of an author in their decision to send it out to reviewers (for those who don't know, in the high impact journals like science or nature, the biggest hurdle is typically getting the editors accept the paper and send the paper out to review).
- diedyesterday 4y agoIs that surprising?! That's a kind of optimization with not always optimum results.
- silexia 4y agoNothing Nature publishes now can be trusted as science; they added new rules that they won't publish anything counter to their political opinion.
- 1vuio0pswjnm7 4y agoThis such a strange headline. It is not the authors' names that have the effect, it is the authors' past work. For example, having done work in the past that was the subject of a Nobel Prize might have an "astonishing" influence. Having the name "James" would not not likely have an astonishing influence.