16 ms·

The culture of rejection in computer science publications

249 points by headalgorithm 4y ago

mfidelman 4y ago
All well and good. Then again, I've found the Computer Science literature to be abysmal - for pretty much forever. IMHO, there isn't a single journal that comes close to the quality of papers published in "Science," or "J. of the AMA." I turned off my ACM & IEEE journal subscriptions, years ago. The good stuff used to be in things like MIT AI Lab Memos, BBN Technical Reports, RFCs, theses, and such. The good presentations are at trade shows - and more likely to be in exhibitor booths, and part of show floor demonstrations (the old INTEROP shownets, the LVC demonstrations at I/ITSEC conferences). Personally, I'm a lot more concerned with the review processes for grant & contract proposals - where there's real money on the table, and achieving solid results can be mission- and sometimes life-critical.
Athas 4y ago
I wonder if this culture arises out of necessity. Many conferences claim to not have a set quota of papers they accept, but in practice there is a limitation to how many presentations can be accommodated given the physical and temporal limitations of the venue. The quota may grow slightly over time, either by shrinking the time allotted to every presentation (ICFP is really squeezing it these years, for example) or adding parallel tracks, but there are ultimately fairly hard physical restrictions. This inevitably limits how many submissions can be accepted, which also creates an otherwise unnecessary air of competition, even between otherwise unrelated papers. Journals don't have this restriction; you can always put out more volumes. I suppose I enjoy conferences as much as any CS practitioner, but it does not strike me as a sustainable or scalable publication method.
- xhkkffbf 4y ago
  But journals do have an economic restriction. More pages cost more money and the subscription fees only go so far. I suppose page charges might balance some of the editorial costs, but those are also limiting. They can't be too high or they'll chase away submissions.
  - Athas 4y ago
    Modern journals don't actually print all that much, and I'd expect the marginal cost of publishing additional papers to be very low.
  - musicale 4y ago
    > More pages cost more money and the subscription fees only go so far arxiv.org's budget is instructive: around $2M for 181K new submissions and a digital library of around 2M articles. Works out to about $12 per new submission. For a reviewed journal or conference, presumably all published papers would have to be reviewed, much as they are currently by volunteer reviewers who review all papers before publication or rejection. If Prof. Lee is right then reviewing effort could go down overall due to fewer resubmissions.
    - jononor 4y ago
      12 USD is the current cost per submission. But I do not think that they need to 2x their budget to handle 2x the amount of articles. Marginal costs per article will be lower. But your point stands, this stuff costs real money. So I just donated 100 USD to Arxiv, as they host a few of my papers. And thousands of other articles that I have read, for free and super accessibly.
tresqotheq 4y ago
I am a simple man. When ever I see the words "Toxic Culture" spoken by people somehere, I flee the country..
- mirror_neuron 4y ago
  Correct me if I’m wrong, but I think you’re trying to say that the phrase “toxic culture” signals something exasperating or frustrating to you about the person using it. Is it because you don’t think that a culture can be toxic? Or perhaps that the phrase is overused or misapplied? Or is it something else that you dislike?
  - selimthegrim 4y ago
    He is referring to Goering.
    - mirror_neuron 4y ago
      That’s unexpected! I admit that my search was superficial, but I didn’t find any obvious connection between Goering and the phrase “toxic culture.” Could you elaborate?
      - selimthegrim 4y ago
        https://en.m.wikipedia.org/wiki/Hanns_Johst https://en.m.wikipedia.org/wiki/Hanns_Johst
        mirror_neuron 4y ago
        Fascinating. Thank you.
- Cthulhu_ 4y ago
  You're not "a simple man", you're an agitator lashing out at criticisms of behaviour and culture.
wanderingmind 4y ago
I think we are not far off from virtual conferences where most sessions are presented by DallE generated presenters, speaking with wavenet generated audios in turn are presenting GPT generated papers.
- throwawayacc2 4y ago
  Now that’s a fun hackathon project if I ever saw one!
- mizzao 4y ago
  Basically already happening in many Chinese departments and conferences, just without the AI yet, but of equivalent quality. (I'm ethnic Chinese and this is not meant to be a racially charged comment)
  - LunaSea 4y ago
    Could you give a bit more context about this? I'm not familiar with the subject.
    - lioeters 4y ago
      I think they're just joking that some of the material presented at Chinese academic conferences are close to non-sense, might as well be produced by DALL-E.
      - walleeee 4y ago
        Can't speak to the state of Chinese academia but have had a few similar experiences at American conferences When careers are on the line and one is expected to mischaracterize one's work or its impact, people will tend to do so If this problem exists independently of cultural or institutional factors maybe there are potential solutions with an equally broad range of application
    - mizzao 4y ago
      I think the performative, results-based culture of academia in general (particularly in Asia and not just Chinese) has seriously succumbed to Goodhart's law: https://en.wikipedia.org/wiki/Goodhart%27s_law https://en.wikipedia.org/wiki/Goodhart%27s_law Because you are judged on citations, publication count in prestigious journals, and research dollars spent... you have people citing themselves and their friends in citation rings, gaming the referee process to pick reviewers, and doing unethical things for money. My first experience in this is when I was asked (as a postdoc) to give a keynote for a no-name conference in Vancouver. They were paying so why not? I later realized that my pedigree was being used to lend credence to a conference that was totally cargo cult academia. There were conference sessions, and people presenting, and bunches of people in several rooms, but nothing was actually being said and nothing was being done. They then offered me $3,000 cash in an envelope which I declined to take most of because of ethical reasons (I only wanted to cover my expenses). A very unique, maybe even out-of-body experience to stay the least. If you want to see a crazy instantiation of cargo cult academia, google "extreme learning machines", which is basically a whole field built off of 2-layer feedforward neural networks with 1 randomly initialized layer. The other keynote at this conference was the guy who "started" this field.
RcouF1uZ4gsC 4y ago
> The double-blind review process has eliminated natural biases that may have influenced our reviews in the past, including tendencies to reject based on gender or to accept from the “better” institutions. I don’t know how much double-blind really helps. Especially in a specific field, you can usually tell what person/lab/company wrote the paper without seeing the authors’ names.
- impendia 4y ago
  I once reviewed a math paper for a double-blind journal. Most math journals aren't double blind (as you say, I can often tell who the authors are) -- but this is one focused on expositional articles that undergrads might enjoy. Anyway, I read it, didn't find it all that interesting, and recommended that the paper be rejected. Afterwards, I googled the paper's title and found a signed copy: the author was one of the most respected scientists in our field, who had been a mentor to me and done a huge amount for my career. I was immediately embarrassed: I rejected his work? And right then I had my moment of zen.
  - ghaff 4y ago
    As someone who has been on the conference committee for (non-academic) conferences, I'm not a particular fan of blind selections. Yes, I get the desire to not just give slots to "the usual suspects" which at least historically has often happened at a lot of conferences I attended. But the reality is that there are people who you know based on overwhelming experience will give great talks that attendees will appreciate and learn from. Even if they submitted an abstract that didn't immediately catch your eye among the pile you're going to accept 25% of, do you really want to reject them? Assuming the committee has some commitment to new/less known speakers there are IMO better ways to spread the love than blinding.
    - xdavidliu 4y ago
      > there are people who you know based on overwhelming experience will give great talks that attendees will appreciate and learn from ... do you really want to reject them? If I'm reading correctly, you're saying that sometimes a work deserves to get in based (at least partly) on the author's name recognition (for historically having given good talks) and not solely on the merits of the work itself. I see the point you're trying to make, but something about this argument makes me uncomfortable.
Adiqq 4y ago
In general I agree that low acceptance rate with negative selection based on arguments like "it's not novel", "it's obvious" is toxic. I get it that capacity for conference might be limited, so gatekeeping might be necessity, but personally I like to learn useful knowledge, so something novel, but theoretical might be more boring for me than something well known, but improved in some interesting ways. Also composing solutions is interesting, you should not have to reinvent the wheel, just to get approved, if you want to focus on broader perspective. The whole is greater than sum of the parts.
- xhkkffbf 4y ago
  But at the core of his arguments is that rejection is mean. He's complaining about the "it's not novel" argument now but if everyone takes this to heart, they'll just have to come up with some other reason to reject. There are only so many resources and academia tends to massively overproduce talent.
eatbitseveryday 4y ago
This is a reason I gave up doing systems research. It’s full of cliques and negativity. It became a process whereby we were even trained to accept this rejection frequency as normal and to expect retrying for years. You keep submitting a paper and revising it for months even while working on the next idea (can’t just sit around fixing a paper, PhD work must always continue). Rejections also directly impact a student’s graduation. No accepted papers at decent conferences? Delay graduation. It’s grating and unnecessarily stressful. After graduation I was unmotivated to continue to bother publishing because I don’t care for this culture. I’m saddened by it. There are positions in industry like at Sandia National Labs which require regular publication to remain employed. Again, given good work is done, rejections affect your livelihood. Too much stress. I lost my “spark” thinking the field of CS systems research was one of exploration and community. It exists, but not as a whole. /rant
- Test0129 4y ago
  I left my PhD in computational geometry for similar reasons. I was told flat out by my advisor that the chances of me getting a paper accepted in even a middling journal were low. Not normally low (as in tough review), but so low because the entire field is dominated by a handful of very high profile researchers and mathematicians that essentially gate keep knowledge on it. It was also necessary to publish some papers to even get my PhD. Academia is dead. My biggest most important realization about learning in the last decade is that my concept of the academic was backwards. They aren't free thinking at all. It's drama all the way down. I now havent touched really anything in CS research because this experience completely killed my love for the field, and advanced learning in general.
  - marktangotango 4y ago
    I agree with this sentiment. Although I didn't make it to even applying to grad programs because the professors at my uni were very much "what are you going to do for me" in Thomas Edison invention factory way, when even inquiring about undergrad research. > I now havent touched really anything in CS research because this experience completely killed my love for the field, and advanced learning in general. I wager you'll come around, out of genuine interest, and when you want to work on interesting things, you'll run into novel problems, a lot. But this is sad to me because every few years I'll come up with something novel, and discover someone already did a ph.d thesis on it. I feel like I'd be much further along if I had access to people who really cared about teaching and pushing the boundaries.
impendia 4y ago
I am an academic mathematician -- who has had job applications rejected, papers rejected, grant proposals rejected. Not always, but it's not exactly a rare occurrence. I've also been on the other side, and it also sucks to reject people. It's an unfortunate reality of academia that there are fewer resources (jobs, grant funding, etc.) available, than there are researchers who are prepared to put them to good use. Further, those who are making the decisions have limited time. If you're serving on a hiring committee and get hundreds of job applicants, you can't hope to read all the papers of all the applicants. To deeply read any one of them would take a fair bit of time. We therefore need a signaling mechanism to distinguish the outstanding from the merely very good. It's of course possible to argue about the details of how papers are rejected, as the authors indeed do. But unfortunately the core problem -- an aspiring academic will get rejected often, and it can be extremely demoralizing -- is one we probably can't solve.
- logisticseh 4y ago
  I don't buy the attention filter argument. No one -- and I really do mean no one -- is going to read the entire contents of the proceedings of even just one of these conferences. NeurIPS -- a single CS conference -- is more than twice the size of the Joint Math Meetings. ICRA and ICML are just as large or larger, and AAAI isn't far behind. That's just one sub-field of CS. There are so many papers coming out every year that I simply cannot keep up with two of my own niches. Adding more papers to that firehose wouldn't materially change the situation. I've reviewed for some (high quality) Mathematics journals. Papers tend to be more complete, for sure, but the reviewing is much less rejectionist. I'm not aware of any Mathematics journal with a 10% acceptance rate, and even 20% is probably on the low end. > It's an unfortunate reality of academia that there are fewer resources (jobs, grant funding, etc.) available, than there are researchers who are prepared to put them to good use. I don't think this is true in CS. Universities outside of an elite set really struggle to hire and retain high quality faculty. It's at a crisis level outside of R1. Teaching-oriented institutions have mostly have stopped trying to hire traditional academics; a masters degree with some teaching experience is sufficient. Some of this is due to industry -- high-quality faculty candidates tend to also have 3x-5x offers in industry, and it's hard to turn down a guaranteed early retirement for the grind and uncertainty of the tenure track. But I think some of it is also that students who would make good teachers and mentors lose confidence due to a series of unnecessary paper rejections and decide to nope out of academia. Again, I spend a lot of time around academic mathematics. The rejectionist culture in CS is real. And not just conferences, btw. An NSF program manager started my last review panel by telling us that scores are consistently way lower in CS than in any other field and to please chill out.
MontyCarloHall 4y ago
Add upvotes/downvotes to preprint servers. Weight voting power by the user’s subfield, dynamically determined by who’s upvoting them, PageRank style (so a scientist who’s mostly upvoted by others working in numerical linear algebra would have a lot of voting power on a preprint describing a novel matrix factorization scheme, and very little voting power on a preprint describing a novel thread scheduling algorithm). Take the top N upvoted preprints in a given field and showcase them at a conference for that field. And to avoid hype cycles or gaming the system, make voting cost karma—if you choose to vote, you yourself lose some voting weight in the process, so votes would only be given out sparingly to papers that actually deserve them.
- nautilius 4y ago
  Haha, yeah no way anyone’s going to game that system! No way some clique recruits all of the voting power (collaborators to undergrads) to push their work through, all it takes is LeCun to simply read all papers and his vote will balance everything out!
  - MontyCarloHall 4y ago
    Easy solution: make it so that voting costs karma, so individuals only have limited voting power.
    - nautilius 4y ago
      Easy fix to your solution: get more people to vote. How about all 10000 undergrads in your institution? In your model, a modern day Einstein would have never had a chance; industrialized research, with publication mills and systematic co-authorship permutation is exactly what would flourish under your model.
      - MontyCarloHall 4y ago
        10k undergrads would have few to no highly upvoted publications, and thus little to no voting power, even in aggregate. An even smaller fraction of them would have highly upvoted papers in the specific subfield of the paper they’re voting on, making their aggregated voting power even more worthless. The only way to game the system would be to somehow convince a ton of highly upvoted people in a given subfield to upvote a paper in that particular subfield (and give up some of their voting power accordingly, since voting would cost you karma). Heck, make it so that you can only vote on a certain number of papers each month (5-10?), to make votes even more valuable.
qzx_pierri 4y ago
It doesn't matter what color you are. It doesn't matter where you're from. If you create something like what Dijkstra created back in 56' (undeniable innovation), then no door will remain shut. That's the beauty of Math, CompSci, or STEM as a whole. Not to be too blunt, but I'd rather we keep this soft, "everyone should be accepted" nonsense out of CompSci. The cream will rise to the top. Anyone complaining just needs to improve, or innovate. The author even says the reason for rejection is often "lack of novelty". Me personally, I'm not talented enough to even be in the same conversation as some of the people who are in that "universe", but that's okay - It just wasn't meant to be for me. If I "stay in my own lane", that creates more room for the gifted, and the elite.
- mikkergp 4y ago
  > If you create something like what Dijkstra created back in 56' (undeniable innovation), then no door will remain shut. That's the beauty of Math, CompSci, or STEM as a whole. Don’t get me wrong, this seems like a nice idea, I’d love to live in this world, but hasn’t it been disproven time and again throughout history? This board itself is flooded with engineers with great ideas being dismissed because of “realities of the business” or “changing priorities”, much less “political realities” etc et Al. In fact I’d say the reality is probably opposite,(as a matter of principal, obviously great ideas do eventually make it but) truly great innovative ideas are ignored, it is only the incremental ideas that don’t make anyone uncomfortable that are accepted.
  - qzx_pierri 4y ago
    > This board itself is flooded with engineers with great ideas being dismissed because of “realities of the business” or “changing priorities”, much less “political realities” etc Your argument would be rock solid if those great ideas were only applicable/useful in huge bureaucratic organizations (which is the type of environment where bullshit ideas are given more attention than truly innovative and impactful ideas). We've all probably been there before, and I get your point 100%. However, the next step would be to go where your ideas can be heard. And if the idea is good enough, people will listen. Shouting into an org chart 10 miles deep to narcissistic C level executives has always seemed like a waste of time. So I don't disagree with you, but if you aren't being heard, speak to someone who will listen.
ballenf 4y ago
> The emphasis on novelty has deep roots in academic publishing. It used to be that publishing was expensive, and any repetition came at the expense of other things that could have been published. Today, however, publishing is essentially free. Requiring novelty shows respect for the readers' time. Paper and ink costs were never the primary limiting factors (even if the publishers' claimed otherwise to save face). "repetition came at the expense of other things" -- no, repetition comes at the expense equal to Number of Readers * Time Wasted on Each non-novel paper. If the author just stuck to "novelty is hard to really know" then it's a much stronger argument.
- hiptobecubic 4y ago
  But without reproducibility the paper is basically worthless. There's already a crisis of unreproducible work and the strong novelty bias is basically why.
- PuppyTailWags 4y ago
  I don't think requiring novelty is respect for a reader's time. I think requiring high quality is respect, even if it is adding to existing bodies of evidence. Novel drivel is just drivel regardless of its novelty; it requires high quality analysis and research for anyone to get any benefit from that newness.
- theaeolist 4y ago
  A clear exposition of an obscure and hard to understand result is lack of 'respect for readers' time'? How so?
ath0 4y ago
I'm long past my academia phase, but recently led the PC for an industry conference (accept rate: ~15%). 1. Curation is important both for the physical limits (venues only fit a certain number of people), attention limits (attendees will usually retain only a handful of "nuggets" no matter how packed the agenda is) and interaction limits (you can't meet everyone at a large conference). 2. If the goal of a conference is not just to "stamp" research as somehow "approved", but to encourage discovery and knowledge exchange that deepens a specific area, it's important to apply that curation filter with an eye toward best advancing the goals of the conference. That means not just going for things that are okay, but those that best resonate with other presentations / attendees / research topics. 3. While the size of any one conference has to be fixed, tech has made it infinitely easier to create new conferences and journals with other focus areas. They may not start with the prestige of a larger journal, but if the papers published start to have an impact, it can catalyze an entire subfield of work. Some conferences can be tied exclusively to "novelty" - ACM academic conferences - but others to "incremental advancements" - the bigger industry conferences in security, like Usenix Security and some to "best explaining ideas" - like Enigma. There are new ways to find an audience for your work and create impact - that's part of the job now.
- blagie 4y ago
  I'm well-known in a research community. I'm positioned such that I don't need more academic points. I've mostly stopped publishing in branded prestige academic venues, in part due to rejection rates. My goal in doing work and writing papers is to see them disseminated. The acceptance/rejection process is asinine -- studies show it's basically random. I've had one paper in my whole career where the reviewers did a proper review (e.g. worked through the math). The rest were quick skims. Comments often show the reviewers never read the paper. The stuff that makes it through this process is often nonsense, while very high-quality work is often cut. The very best paper I wrote in my career has never seen the light of day. It was shortened to a 4-page work-in-progress because a reviewer didn't read it (the feedback was literally nonsense: that the sample size was small enough to be anecdotal; I had the largest sample size in the history of the research field). The only impact of this egoistical search for prestige-by-low-accept-rates is that people who have better things to do with their time leave, and that research dissemination is slowed. Those excuses make little sense in the real world: 1) If your conference has a 10% accept rate, it's easy enough to book a bigger venue next year. I've been to conferences with dozens of people, and ones with tens of thousands. It all works well. Bigger ones work better, if anything. 2) PCs aren't thoughtful enough to do that well, and even so, the goal of a conference shouldn't be to select things which resonate with the entrenched PC. That's why many ideas need to wait for a generation of old, conservative professors to die to make it out there. 3) The whole obsession with prestige is stupid and misguided. Journals and conferences ought to have quality bars. Are there typos and grammar errors? Were there clear IRB ethic violations? Did you use error bars on your plots? Was data fabricated? Is the research methodologically sound? Is it coherent and readable? And so on. If it passes those bars, it should be published. If no one reads it / attends a talk, that's okay too -- importance can and should be determined after-the-fact.
krinchan 4y ago
The academic system is completely broken for Computer Science, and I don't really see a way to fix it. The economic realities of the field just make it too risky to allow an exceptionally gifted individual to remain out in the open publishing research that could potentially destroy your business model. Justifying these low acceptance rates as somehow prestigious is really just creating even more perverse incentives that open the academia side of Computer Science to further defunding and brain drain. If you're smart enough to rise, you'll get an offer from the private sector you simply cannot refuse. It doesn't matter if your passion is Academia, they can and will buy you out and own whatever you're working on. This has all led to Computer Science's academia side being something one escapes rather than something you contribute too. The "cream" rising to the top is often less genius and more politically savvy with the right connections on the PC. I'm not necessarily against a selection bias towards "people skills," but to do so and continue to pretend PCs are pure meritocracy is nauseating. It just comes back to the fact that the majority of Comp Sci PhDs have the same story: Halfway into their doctorate program they became severely disillusioned and started jockeying to just graduate and land a private sector job that essentially was just bribe money to keep them from working for the competition.
- logisticseh 4y ago
  > The "cream" rising to the top is often less genius and more politically savvy with the right connections on the PC. I'm generally nauseated when I interact with American CS academics. Every time I attend a conference, PC, or NSF panel, I am so glad I chose industry. It's like IRL twitter. (Europe seems to be better for some reason.) > If you're smart enough to rise, you'll get an offer from the private sector you simply cannot refuse. It doesn't matter if your passion is Academia, they can and will buy you out and own whatever you're working on. IME it's less about "offer you can't refuse" on the industry side and more about "offer you can't take" on the academic side. After 6 years of deferred income I simply could not take a job that paid $80K-$100K in an HCoL area or $65K-$80K in an LCoL area. I had loans to pay back, no 401K, and not enough savings for a down payment. If you want good people to stay in CS academia, I think a few things need to change: 1. First, and most importantly, the faculty culture. I don't really know how to describe the problem, but "the old folks are checked out and the young folks are Twitter personalities" is probably close. What's the point of being in academia if you have to be surrounded by the intellectual equivalent of used car salesmen, especially when you can go to industry and do interesting work without the BS? 2. Double the income of PhD students so that they aren't financially ruined by choosing the academic path. This isn't a super unreasonable request -- they'd still be paid less than their peers in industry while doing what's effectively a full time job. 3. Pay faculty more. Not a lot more... just, like, "at least what my undergrad students make at their first job after graduating". I think if you solve items 2 and 3, then item 1 will take care of itself.
kmmlng 4y ago
I think it's not surprising that conferences have to perform some selection. You need the right amount of participants and talks. If a high rejection rate achieves the right amount, it is hard to argue against it. But why are publications tied to conference attendance anyways? Sure, there also journals, but submitting to a journal tends to be an especially slow process. If you are in a fast-paced field, submitting to a journal is a dangerous game. Why can we not just upload our papers to something like arxiv and then give people the option to vote on papers analogous to reddit submissions, so that promising stuff organically rises to the top. That way it would at least be based on the opinions of a sizable number of judges, not just three preselected peers. Oh no, but what about peer review. What about it? Is it difficult to get past peer review at a top conference? Yes. Is it difficult to get past peer review in general? No. You can publish anything you want already, you will just have to jump through senseless hoops to do it. Why not skip the hoops and just upload it somewhere? We can still have journals and conferences that select high quality material from the uploaded papers and it will be an honour to be featured in one of those. You can still use features at conferences and journals as a bad metric to judge the quality of researchers, but the actual publishing will be decoupled from these institutions.
- Eridrus 4y ago
  ArXiv is fine for what it does, but it does not provide any sort of dissemination support. IMO the best thing about CS conferences is the poster track where you can walk by hundreds of posters, and the information is (when done well) much more easily digestible than papers, and you get to ask questions, and these are nowhere near their limits.
snickerbockers 4y ago
>Serving on a PC is a yeoman’s service, and the community owes them a debt of gratitude. However, I believe that a toxic culture has emerged. This blog is a call for PCs to change their priorities. does that acronym not stand for what i think it does?
- warkdarrior 4y ago
  PC stands for Program Committee, the selected group of people in charge of accepting and rejecting paper submissions.
- secondcoming 4y ago
  Probably not, but I don't know what it actually does stand for. It's definitely not Personal Computer
  - Hackbraten 4y ago
    The acronym is not web-search friendly either. Been looking through this thread and search engines. Nothing helped. For example, not a single one of these [1] makes sense to me in that context. [1]: https://www.allacronyms.com/PC/university https://www.allacronyms.com/PC/university
    - warkdarrior 4y ago
      Some times you will see the acronym TPC for Technical Program Committee. Searching for 'conference TPC' returns some relevant hits.
    - xdavidliu 4y ago
      this is one of my pet peeves. You and dozens of other people probably spent collectively hundreds to thousands of minutes and brain cells wondering what PC stood for, while the people who originally typed "PC" instead of program committee saved maybe a fifth of a second.
      - snickerbockers 4y ago
m1117 4y ago
No, theres not!
seydor 4y ago
the most toxic ... if you exclude all the others
magpi3 4y ago
Academia as a whole is such an interesting place. I value it highly because I value learning. I love school, I loved the conversations and deep dives into interesting waters that nourished me in both college and graduate school, and I loved the opportunity to be fully immersed, without needing to get a job, in an atmosphere of learning. But the rest of it? The petty hierarchies, the papers born of countless hours of hard work by extremely talented people that few people will ever read, the wage labor of adjuncts, the elitism, the comfortable cowardice of tenure - that all should be burned in fire. I once read a book - I wish I could remember its name - that noted that even though we live in a democratic republic fueled by at least somewhat meritocratic capitalism, institutions from the feudal era still exist, the most notable being the church and the university. That was eye opening when I read it, and to this day it has jaded my view of academia and the people who tether themselves to it. All of those noble, high minded academics, fighting for their place in a feudal structure that they don't dare challenge. I know I sound a bit like an asshole when I write this, but: I can't truly look up to anyone who would resign themselves to a structure like that.
- bo1024 4y ago
  If the last paragraph is your perspective on being a pawn in an academic structure, I would love to hear what you think of being a pawn in a corporation.
  - blacksmithgu 4y ago
    Not OP, but at at least most corporate pawns are cynically aware of how artificial the system is and that you need to play games to get ahead (or just coast along if you don't care). Academia has a luster of meritocracy when in reality you need to game things just as hard to become successful.
  - magpi3 4y ago
    I suppose a big enough corporation can feel the same, at least in terms of internal politics, but the difference for me is the tenure (well-paid, almost impossible to fire) and adjunct (close to minimum wage, no health insurance, completely disposable) dynamics. Indefensible. And I think the people with tenure don't object to this model simply because they enjoy their privilege.
dekhn 4y ago
Nothing has made me happier than to simply leave the academic/publishing side of CS and focus entirely on things that I find interesting and can do on a single desktop computer. If I share, I share and give people my code and ideas freely, but don't publish. Why would I want to take all my hard work and beg somebody with power to put it on a piece of paper? If I have something useful to say I'll put it in arxiv. Peer reviewers just wasted my time.
hannob 4y ago
Honestly this only touches a small fraction of how absurd the whole publishing system with conferences is. I mean just thing about some obvious issues: Large parts of computer science limit how much science they can publish (and thus effectively share with others) by the number of conferences people want to organize. There's also a very obvious discrimination issue, as most "high tier" conferences are either in the US or (to a lesser extent) in the EU. And it's pretty crazy that people do transatlantic flights to go to a conference in order to publish a paper, even if they don't really want to go to the conference.
adamsmith143 4y ago
The lack of Novelty point feels off the mark, at least in some fields. At least in ML its extremely common for a paper to just report a new benchmark on some well known dataset with virtually no new real contributions other than more time spent hyperparemeter turning
FabHK 4y ago
One side effect I didn't see mentioned in the article: One professor of mine spoke of the LPU, the least publishable unit. So, if you're lucky enough to have some novel ideas, and build something nice out of it, don't put it all into one coherent and easily digested journal paper! The number of publications counts. Instead, chop it into little pieces that are just "novel" or noteworthy enough (LPUs), and publish them separately. Publication list inflation accomplished; and scientific progress/intelligibility/successful communication be damned.
- grayclhn 4y ago
  Honestly, I think this is often said cynically but is a good practice overall. Would you rather have to read and understand one giant commit reflecting 2 years of work, or 10 well-documented and logically complete individual commits?
  - blacksmithgu 4y ago
    The issue is that novel paper ideas will be split across multiple years (and even multiple conferences), making it much harder to actually see the whole picture for a reader. Each little piece of the paper will often also be bloated with unnecessary extra detail in order to reach the threshold for "minimum publishable paper".
    - holidaygoose 4y ago
      Using the same code analogy as the parent, this is like code with unnecessary extra commenting. Seems okay to me.
    - grayclhn 4y ago
      Splitting up a groundbreaking idea into so many papers that the idea is lost is 1) going beyond a “minimal publishable unit” and 2) not in the authors’ interest, since getting credit for a groundbreaking idea in a correspondingly prestigious outlet is much better than getting credit for 2 or 3 bad ideas. I’m sure there’s a level of novelty where 2 irrelevant papers is better for the author than 1 single paper, but I don’t think we should design academic publishing around slightly-better-than-mediocre contributions.
faangiq 4y ago
CS has a ton of people in it and they produce a lot of garbage papers. Oh you retrained your ML model? Boom, new paper.
- 22SAS 4y ago
  I completely agree with this. I have seen papers produced by a bunch of newly minted PhD's and the first thought that came in my mind was "WTF is this? There's nothing in this that's new or makes sense". A lot of publishing, from outside the top research universities, seems to be mainly at showing a higher publication count to get more funding and impress tenure committees. Some of the professors, who do this, make you wonder how they get their PhD in the first place. Back in my masters, I was giving a presentation on a paper from FB, where they were going over scheduling policies they had been using in their hadoop clusters. The professor reviewing asked me a question "You're taking about scheduling tasks in distributed environments, you're talking about Hadoop, Hadoop is for files what scheduling does it do?". Some of us there had a big "WTF" plastered on our face. Couldn't even argue cause this prof. thought he was a hotshot and would make life hell for students who did. In his own words to a student "I can write a compiler, I can write an operating system, but I can't make head or tail of what you're doing".
raphlinus 4y ago
This topic is close to my heart. I do research on the fundamentals of 2D graphics, some of it cutting edge (especially GPU techniques), others just selecting the best known techniques (for example, right now I'm doing a bit of a deep dive into robust cubic and quartic polynomial root finders, not really academically publishable, but potentially hugely useful for others working on similar problems). I made a serious attempt to submit some of my GPU monoid work to academic conferences earlier this year, got rejected. As someone who doesn't have the "publish or perish" incentive of actually being in academia, it's really just not worth it. I'm also doing inquiry into UI, for example architectural patterns in reactive systems. The current state of academic literature on this topic seems to be terrible (though please point out counterexamples!). Most of what's written is marketing material for UI frameworks - there's an explosion of those, especially in the JavaScript world, but very little synthesis of the core concepts. I wouldn't even attempt to try to submit an academic paper on this topic, though I think it would likely be useful to the world. It feels like there should be a space to publish work that is not novel in an academic sense, but useful. Wikipedia is not it (their articles on cubic and quartic equations are garbage when it comes to numerical concerns), Stack Overflow is not it, academic conferences and journals are not it. I'm using my blog, generally successfully, but it feels there should be a more systematic approach.
- YorkshireSeason 4y ago
  I found myself in a somewhat similar situation. I learned to thing long-term: (1) Publish initial ideas not in top conferences, but in second rate venues. Most publications in top conferences are, clearly, refinements of ideas that were first presented in inchoate form, in less prestigious workshops. Paradoxically, prestigious conferences don't like brand new ideas, but they like polished papers. And they like concrete open problems being solved. If a paper starts "We are solving the longstanding open problem from [17]" then that dramatically increases acceptance probability: most submissions to prestigious conferences are not rejected b/c they are wrong, but because it's unclear why they are more siginficant than all the other submissions. Solving a concrete open problem gives you social proof that the problem is significant (others worked on it) and hard (people failed). How do you create concrete open problems? By leaving them as concrete open problems in a previous paper. (2) Create your own conferences. All now famous conferences started as small, low prestige workshops 10-20 years ago, e.g. the "Oakland" security workshop (now: IEEE Symposium on Security and Privacy). I learned this from my post-doc mentor, who is now a famous scientist with a whole research tradition to his name, that he created, bootstrapping from a first workshop paper. PS, is your monoid work online, is it this https://arxiv.org/abs/2205.11659 https://arxiv.org/abs/2205.11659 ? I'm interested in monoids on GPUs.
kazinator 4y ago
This one has an easy fix! If you're in computing and want to produce garbage that is accepted anyway, don't write papers: write software.
- cauefcr 4y ago
  There are PR reviews worse than the #2 reviewer of most papers, so it's all about new software or forks, don't even bother upstreaming. /s
paulpauper 4y ago
The emphasis on novelty has deep roots in academic publishing. It used to be that publishing was expensive, and any repetition came at the expense of other things that could have been published. Today, however, publishing is essentially free. Yes, publishing is cheap or free, but attention is still scarce.
- mirker 4y ago
  In hot fields, you have arXiv papers 2-3 generations ahead of peer review. Some have more citations than typical accepted papers. Peer review does not limit people’s attention.
hedora 4y ago
I’d like to see a histogram of the number of resubmissions a paper goes through before acceptance. There are definitely papers that get accepted on the first try, and others that get resubmitted 5+ times. It doesn’t take many papers in the second group to drag acceptance rates down.
sgt101 4y ago
A lot of this is caused by the bad faith of some state actors. Many computer science conferences are under systematic assault from these places, which systematically swamp the PC's with submissions. What needs to happen is : - Regionalisation; make conferences regional only. So that submission can only come from that area or small group of nations. This will reduce travel demands and increase plurality. - Sharp constraints on personal submission: one and only one paper as an author by anyone. - Block outs : you get in one year, you skip a year.
yodsanklai 4y ago
I don't see a "culture of rejection". There's rejection whenever there's competition and a selection. There are tons of computer science conferences and not all of them are elitist. Even mediocre papers get published. Should we have all the submissions in the world accepted at the top conferences so that no researcher feels left behind?
bcantrill 4y ago
It's a good article, but it's not making the (obvious?) jump: the conference model in computer science is broken. No other discipline does it this way, and it creates for ourselves this terrible program committee problem. The journal model has plenty of its own flaws, but it at least allows for iterative work. (That is, instead of rejecting a work outright, a journal can work with an author on the flaws, even if that requires doing substantial new work.) This is not to lionize the journal model, which has plenty of its own flaws -- and indeed, I personally think computer science should use its current laggard status as an opportunity to find a wholly new model, preferably one that is much more amenable to practitioner participation.[0] [0] https://www.usenix.org/conference/atc16/technical-sessions/presentation/cantrill https://www.usenix.org/conference/atc16/technical-sessions/p...
raindear 4y ago
A solution is a site for publishing papers, with community discussions. People can rate/upvote papers, and these ratings can be used to build several metrics for papers, authors and reviewers.

249 points by headalgorithm 4y ago

All well and good. Then again, I've found the Computer Science literature to be abysmal - for pretty much forever. IMHO, there isn't a single journal that comes close to the quality of papers published in "Science," or "J. of the AMA." I turned off my ACM & IEEE journal subscriptions, years ago. The good stuff used to be in things like MIT AI Lab Memos, BBN Technical Reports, RFCs, theses, and such. The good presentations are at trade shows - and more likely to be in exhibitor booths, and part of show floor demonstrations (the old INTEROP shownets, the LVC demonstrations at I/ITSEC conferences). Personally, I'm a lot more concerned with the review processes for grant & contract proposals - where there's real money on the table, and achieving solid results can be mission- and sometimes life-critical.

Athas 4y ago

I wonder if this culture arises out of necessity. Many conferences claim to not have a set quota of papers they accept, but in practice there is a limitation to how many presentations can be accommodated given the physical and temporal limitations of the venue. The quota may grow slightly over time, either by shrinking the time allotted to every presentation (ICFP is really squeezing it these years, for example) or adding parallel tracks, but there are ultimately fairly hard physical restrictions. This inevitably limits how many submissions can be accepted, which also creates an otherwise unnecessary air of competition, even between otherwise unrelated papers. Journals don't have this restriction; you can always put out more volumes. I suppose I enjoy conferences as much as any CS practitioner, but it does not strike me as a sustainable or scalable publication method.

xhkkffbf 4y ago
But journals do have an economic restriction. More pages cost more money and the subscription fees only go so far. I suppose page charges might balance some of the editorial costs, but those are also limiting. They can't be too high or they'll chase away submissions.
- Athas 4y ago
  Modern journals don't actually print all that much, and I'd expect the marginal cost of publishing additional papers to be very low.
- musicale 4y ago
  > More pages cost more money and the subscription fees only go so far arxiv.org's budget is instructive: around $2M for 181K new submissions and a digital library of around 2M articles. Works out to about $12 per new submission. For a reviewed journal or conference, presumably all published papers would have to be reviewed, much as they are currently by volunteer reviewers who review all papers before publication or rejection. If Prof. Lee is right then reviewing effort could go down overall due to fewer resubmissions.
  - jononor 4y ago
    12 USD is the current cost per submission. But I do not think that they need to 2x their budget to handle 2x the amount of articles. Marginal costs per article will be lower. But your point stands, this stuff costs real money. So I just donated 100 USD to Arxiv, as they host a few of my papers. And thousands of other articles that I have read, for free and super accessibly.

tresqotheq 4y ago

I am a simple man. When ever I see the words "Toxic Culture" spoken by people somehere, I flee the country..

mirror_neuron 4y ago
Correct me if I’m wrong, but I think you’re trying to say that the phrase “toxic culture” signals something exasperating or frustrating to you about the person using it. Is it because you don’t think that a culture can be toxic? Or perhaps that the phrase is overused or misapplied? Or is it something else that you dislike?
- selimthegrim 4y ago
  He is referring to Goering.
  - mirror_neuron 4y ago
    That’s unexpected! I admit that my search was superficial, but I didn’t find any obvious connection between Goering and the phrase “toxic culture.” Could you elaborate?
    - selimthegrim 4y ago
      https://en.m.wikipedia.org/wiki/Hanns_Johst https://en.m.wikipedia.org/wiki/Hanns_Johst
      - mirror_neuron 4y ago
        Fascinating. Thank you.
Cthulhu_ 4y ago
You're not "a simple man", you're an agitator lashing out at criticisms of behaviour and culture.

wanderingmind 4y ago

I think we are not far off from virtual conferences where most sessions are presented by DallE generated presenters, speaking with wavenet generated audios in turn are presenting GPT generated papers.

throwawayacc2 4y ago
Now that’s a fun hackathon project if I ever saw one!
mizzao 4y ago
Basically already happening in many Chinese departments and conferences, just without the AI yet, but of equivalent quality. (I'm ethnic Chinese and this is not meant to be a racially charged comment)
- LunaSea 4y ago
  Could you give a bit more context about this? I'm not familiar with the subject.
  - lioeters 4y ago
    I think they're just joking that some of the material presented at Chinese academic conferences are close to non-sense, might as well be produced by DALL-E.
    - walleeee 4y ago
      Can't speak to the state of Chinese academia but have had a few similar experiences at American conferences When careers are on the line and one is expected to mischaracterize one's work or its impact, people will tend to do so If this problem exists independently of cultural or institutional factors maybe there are potential solutions with an equally broad range of application
  - mizzao 4y ago
    I think the performative, results-based culture of academia in general (particularly in Asia and not just Chinese) has seriously succumbed to Goodhart's law: https://en.wikipedia.org/wiki/Goodhart%27s_law https://en.wikipedia.org/wiki/Goodhart%27s_law Because you are judged on citations, publication count in prestigious journals, and research dollars spent... you have people citing themselves and their friends in citation rings, gaming the referee process to pick reviewers, and doing unethical things for money. My first experience in this is when I was asked (as a postdoc) to give a keynote for a no-name conference in Vancouver. They were paying so why not? I later realized that my pedigree was being used to lend credence to a conference that was totally cargo cult academia. There were conference sessions, and people presenting, and bunches of people in several rooms, but nothing was actually being said and nothing was being done. They then offered me $3,000 cash in an envelope which I declined to take most of because of ethical reasons (I only wanted to cover my expenses). A very unique, maybe even out-of-body experience to stay the least. If you want to see a crazy instantiation of cargo cult academia, google "extreme learning machines", which is basically a whole field built off of 2-layer feedforward neural networks with 1 randomly initialized layer. The other keynote at this conference was the guy who "started" this field.

RcouF1uZ4gsC 4y ago

> The double-blind review process has eliminated natural biases that may have influenced our reviews in the past, including tendencies to reject based on gender or to accept from the “better” institutions. I don’t know how much double-blind really helps. Especially in a specific field, you can usually tell what person/lab/company wrote the paper without seeing the authors’ names.

impendia 4y ago
I once reviewed a math paper for a double-blind journal. Most math journals aren't double blind (as you say, I can often tell who the authors are) -- but this is one focused on expositional articles that undergrads might enjoy. Anyway, I read it, didn't find it all that interesting, and recommended that the paper be rejected. Afterwards, I googled the paper's title and found a signed copy: the author was one of the most respected scientists in our field, who had been a mentor to me and done a huge amount for my career. I was immediately embarrassed: I rejected his work? And right then I had my moment of zen.
- ghaff 4y ago
  As someone who has been on the conference committee for (non-academic) conferences, I'm not a particular fan of blind selections. Yes, I get the desire to not just give slots to "the usual suspects" which at least historically has often happened at a lot of conferences I attended. But the reality is that there are people who you know based on overwhelming experience will give great talks that attendees will appreciate and learn from. Even if they submitted an abstract that didn't immediately catch your eye among the pile you're going to accept 25% of, do you really want to reject them? Assuming the committee has some commitment to new/less known speakers there are IMO better ways to spread the love than blinding.
  - xdavidliu 4y ago
    > there are people who you know based on overwhelming experience will give great talks that attendees will appreciate and learn from ... do you really want to reject them? If I'm reading correctly, you're saying that sometimes a work deserves to get in based (at least partly) on the author's name recognition (for historically having given good talks) and not solely on the merits of the work itself. I see the point you're trying to make, but something about this argument makes me uncomfortable.

Adiqq 4y ago

In general I agree that low acceptance rate with negative selection based on arguments like "it's not novel", "it's obvious" is toxic. I get it that capacity for conference might be limited, so gatekeeping might be necessity, but personally I like to learn useful knowledge, so something novel, but theoretical might be more boring for me than something well known, but improved in some interesting ways. Also composing solutions is interesting, you should not have to reinvent the wheel, just to get approved, if you want to focus on broader perspective. The whole is greater than sum of the parts.

xhkkffbf 4y ago
But at the core of his arguments is that rejection is mean. He's complaining about the "it's not novel" argument now but if everyone takes this to heart, they'll just have to come up with some other reason to reject. There are only so many resources and academia tends to massively overproduce talent.

eatbitseveryday 4y ago

This is a reason I gave up doing systems research. It’s full of cliques and negativity. It became a process whereby we were even trained to accept this rejection frequency as normal and to expect retrying for years. You keep submitting a paper and revising it for months even while working on the next idea (can’t just sit around fixing a paper, PhD work must always continue). Rejections also directly impact a student’s graduation. No accepted papers at decent conferences? Delay graduation. It’s grating and unnecessarily stressful. After graduation I was unmotivated to continue to bother publishing because I don’t care for this culture. I’m saddened by it. There are positions in industry like at Sandia National Labs which require regular publication to remain employed. Again, given good work is done, rejections affect your livelihood. Too much stress. I lost my “spark” thinking the field of CS systems research was one of exploration and community. It exists, but not as a whole. /rant

Test0129 4y ago
I left my PhD in computational geometry for similar reasons. I was told flat out by my advisor that the chances of me getting a paper accepted in even a middling journal were low. Not normally low (as in tough review), but so low because the entire field is dominated by a handful of very high profile researchers and mathematicians that essentially gate keep knowledge on it. It was also necessary to publish some papers to even get my PhD. Academia is dead. My biggest most important realization about learning in the last decade is that my concept of the academic was backwards. They aren't free thinking at all. It's drama all the way down. I now havent touched really anything in CS research because this experience completely killed my love for the field, and advanced learning in general.
- marktangotango 4y ago
  I agree with this sentiment. Although I didn't make it to even applying to grad programs because the professors at my uni were very much "what are you going to do for me" in Thomas Edison invention factory way, when even inquiring about undergrad research. > I now havent touched really anything in CS research because this experience completely killed my love for the field, and advanced learning in general. I wager you'll come around, out of genuine interest, and when you want to work on interesting things, you'll run into novel problems, a lot. But this is sad to me because every few years I'll come up with something novel, and discover someone already did a ph.d thesis on it. I feel like I'd be much further along if I had access to people who really cared about teaching and pushing the boundaries.

impendia 4y ago

I am an academic mathematician -- who has had job applications rejected, papers rejected, grant proposals rejected. Not always, but it's not exactly a rare occurrence. I've also been on the other side, and it also sucks to reject people. It's an unfortunate reality of academia that there are fewer resources (jobs, grant funding, etc.) available, than there are researchers who are prepared to put them to good use. Further, those who are making the decisions have limited time. If you're serving on a hiring committee and get hundreds of job applicants, you can't hope to read all the papers of all the applicants. To deeply read any one of them would take a fair bit of time. We therefore need a signaling mechanism to distinguish the outstanding from the merely very good. It's of course possible to argue about the details of how papers are rejected, as the authors indeed do. But unfortunately the core problem -- an aspiring academic will get rejected often, and it can be extremely demoralizing -- is one we probably can't solve.

logisticseh 4y ago
I don't buy the attention filter argument. No one -- and I really do mean no one -- is going to read the entire contents of the proceedings of even just one of these conferences. NeurIPS -- a single CS conference -- is more than twice the size of the Joint Math Meetings. ICRA and ICML are just as large or larger, and AAAI isn't far behind. That's just one sub-field of CS. There are so many papers coming out every year that I simply cannot keep up with two of my own niches. Adding more papers to that firehose wouldn't materially change the situation. I've reviewed for some (high quality) Mathematics journals. Papers tend to be more complete, for sure, but the reviewing is much less rejectionist. I'm not aware of any Mathematics journal with a 10% acceptance rate, and even 20% is probably on the low end. > It's an unfortunate reality of academia that there are fewer resources (jobs, grant funding, etc.) available, than there are researchers who are prepared to put them to good use. I don't think this is true in CS. Universities outside of an elite set really struggle to hire and retain high quality faculty. It's at a crisis level outside of R1. Teaching-oriented institutions have mostly have stopped trying to hire traditional academics; a masters degree with some teaching experience is sufficient. Some of this is due to industry -- high-quality faculty candidates tend to also have 3x-5x offers in industry, and it's hard to turn down a guaranteed early retirement for the grind and uncertainty of the tenure track. But I think some of it is also that students who would make good teachers and mentors lose confidence due to a series of unnecessary paper rejections and decide to nope out of academia. Again, I spend a lot of time around academic mathematics. The rejectionist culture in CS is real. And not just conferences, btw. An NSF program manager started my last review panel by telling us that scores are consistently way lower in CS than in any other field and to please chill out.

MontyCarloHall 4y ago

Add upvotes/downvotes to preprint servers. Weight voting power by the user’s subfield, dynamically determined by who’s upvoting them, PageRank style (so a scientist who’s mostly upvoted by others working in numerical linear algebra would have a lot of voting power on a preprint describing a novel matrix factorization scheme, and very little voting power on a preprint describing a novel thread scheduling algorithm). Take the top N upvoted preprints in a given field and showcase them at a conference for that field. And to avoid hype cycles or gaming the system, make voting cost karma—if you choose to vote, you yourself lose some voting weight in the process, so votes would only be given out sparingly to papers that actually deserve them.

nautilius 4y ago
Haha, yeah no way anyone’s going to game that system! No way some clique recruits all of the voting power (collaborators to undergrads) to push their work through, all it takes is LeCun to simply read all papers and his vote will balance everything out!
- MontyCarloHall 4y ago
  Easy solution: make it so that voting costs karma, so individuals only have limited voting power.
  - nautilius 4y ago
    Easy fix to your solution: get more people to vote. How about all 10000 undergrads in your institution? In your model, a modern day Einstein would have never had a chance; industrialized research, with publication mills and systematic co-authorship permutation is exactly what would flourish under your model.
    - MontyCarloHall 4y ago
      10k undergrads would have few to no highly upvoted publications, and thus little to no voting power, even in aggregate. An even smaller fraction of them would have highly upvoted papers in the specific subfield of the paper they’re voting on, making their aggregated voting power even more worthless. The only way to game the system would be to somehow convince a ton of highly upvoted people in a given subfield to upvote a paper in that particular subfield (and give up some of their voting power accordingly, since voting would cost you karma). Heck, make it so that you can only vote on a certain number of papers each month (5-10?), to make votes even more valuable.

qzx_pierri 4y ago

It doesn't matter what color you are. It doesn't matter where you're from. If you create something like what Dijkstra created back in 56' (undeniable innovation), then no door will remain shut. That's the beauty of Math, CompSci, or STEM as a whole. Not to be too blunt, but I'd rather we keep this soft, "everyone should be accepted" nonsense out of CompSci. The cream will rise to the top. Anyone complaining just needs to improve, or innovate. The author even says the reason for rejection is often "lack of novelty". Me personally, I'm not talented enough to even be in the same conversation as some of the people who are in that "universe", but that's okay - It just wasn't meant to be for me. If I "stay in my own lane", that creates more room for the gifted, and the elite.

mikkergp 4y ago
> If you create something like what Dijkstra created back in 56' (undeniable innovation), then no door will remain shut. That's the beauty of Math, CompSci, or STEM as a whole. Don’t get me wrong, this seems like a nice idea, I’d love to live in this world, but hasn’t it been disproven time and again throughout history? This board itself is flooded with engineers with great ideas being dismissed because of “realities of the business” or “changing priorities”, much less “political realities” etc et Al. In fact I’d say the reality is probably opposite,(as a matter of principal, obviously great ideas do eventually make it but) truly great innovative ideas are ignored, it is only the incremental ideas that don’t make anyone uncomfortable that are accepted.
- qzx_pierri 4y ago
  > This board itself is flooded with engineers with great ideas being dismissed because of “realities of the business” or “changing priorities”, much less “political realities” etc Your argument would be rock solid if those great ideas were only applicable/useful in huge bureaucratic organizations (which is the type of environment where bullshit ideas are given more attention than truly innovative and impactful ideas). We've all probably been there before, and I get your point 100%. However, the next step would be to go where your ideas can be heard. And if the idea is good enough, people will listen. Shouting into an org chart 10 miles deep to narcissistic C level executives has always seemed like a waste of time. So I don't disagree with you, but if you aren't being heard, speak to someone who will listen.

ballenf 4y ago

> The emphasis on novelty has deep roots in academic publishing. It used to be that publishing was expensive, and any repetition came at the expense of other things that could have been published. Today, however, publishing is essentially free. Requiring novelty shows respect for the readers' time. Paper and ink costs were never the primary limiting factors (even if the publishers' claimed otherwise to save face). "repetition came at the expense of other things" -- no, repetition comes at the expense equal to Number of Readers * Time Wasted on Each non-novel paper. If the author just stuck to "novelty is hard to really know" then it's a much stronger argument.

hiptobecubic 4y ago
But without reproducibility the paper is basically worthless. There's already a crisis of unreproducible work and the strong novelty bias is basically why.
PuppyTailWags 4y ago
I don't think requiring novelty is respect for a reader's time. I think requiring high quality is respect, even if it is adding to existing bodies of evidence. Novel drivel is just drivel regardless of its novelty; it requires high quality analysis and research for anyone to get any benefit from that newness.
theaeolist 4y ago
A clear exposition of an obscure and hard to understand result is lack of 'respect for readers' time'? How so?

ath0 4y ago

I'm long past my academia phase, but recently led the PC for an industry conference (accept rate: ~15%). 1. Curation is important both for the physical limits (venues only fit a certain number of people), attention limits (attendees will usually retain only a handful of "nuggets" no matter how packed the agenda is) and interaction limits (you can't meet everyone at a large conference). 2. If the goal of a conference is not just to "stamp" research as somehow "approved", but to encourage discovery and knowledge exchange that deepens a specific area, it's important to apply that curation filter with an eye toward best advancing the goals of the conference. That means not just going for things that are okay, but those that best resonate with other presentations / attendees / research topics. 3. While the size of any one conference has to be fixed, tech has made it infinitely easier to create new conferences and journals with other focus areas. They may not start with the prestige of a larger journal, but if the papers published start to have an impact, it can catalyze an entire subfield of work. Some conferences can be tied exclusively to "novelty" - ACM academic conferences - but others to "incremental advancements" - the bigger industry conferences in security, like Usenix Security and some to "best explaining ideas" - like Enigma. There are new ways to find an audience for your work and create impact - that's part of the job now.

blagie 4y ago
I'm well-known in a research community. I'm positioned such that I don't need more academic points. I've mostly stopped publishing in branded prestige academic venues, in part due to rejection rates. My goal in doing work and writing papers is to see them disseminated. The acceptance/rejection process is asinine -- studies show it's basically random. I've had one paper in my whole career where the reviewers did a proper review (e.g. worked through the math). The rest were quick skims. Comments often show the reviewers never read the paper. The stuff that makes it through this process is often nonsense, while very high-quality work is often cut. The very best paper I wrote in my career has never seen the light of day. It was shortened to a 4-page work-in-progress because a reviewer didn't read it (the feedback was literally nonsense: that the sample size was small enough to be anecdotal; I had the largest sample size in the history of the research field). The only impact of this egoistical search for prestige-by-low-accept-rates is that people who have better things to do with their time leave, and that research dissemination is slowed. Those excuses make little sense in the real world: 1) If your conference has a 10% accept rate, it's easy enough to book a bigger venue next year. I've been to conferences with dozens of people, and ones with tens of thousands. It all works well. Bigger ones work better, if anything. 2) PCs aren't thoughtful enough to do that well, and even so, the goal of a conference shouldn't be to select things which resonate with the entrenched PC. That's why many ideas need to wait for a generation of old, conservative professors to die to make it out there. 3) The whole obsession with prestige is stupid and misguided. Journals and conferences ought to have quality bars. Are there typos and grammar errors? Were there clear IRB ethic violations? Did you use error bars on your plots? Was data fabricated? Is the research methodologically sound? Is it coherent and readable? And so on. If it passes those bars, it should be published. If no one reads it / attends a talk, that's okay too -- importance can and should be determined after-the-fact.

krinchan 4y ago

The academic system is completely broken for Computer Science, and I don't really see a way to fix it. The economic realities of the field just make it too risky to allow an exceptionally gifted individual to remain out in the open publishing research that could potentially destroy your business model. Justifying these low acceptance rates as somehow prestigious is really just creating even more perverse incentives that open the academia side of Computer Science to further defunding and brain drain. If you're smart enough to rise, you'll get an offer from the private sector you simply cannot refuse. It doesn't matter if your passion is Academia, they can and will buy you out and own whatever you're working on. This has all led to Computer Science's academia side being something one escapes rather than something you contribute too. The "cream" rising to the top is often less genius and more politically savvy with the right connections on the PC. I'm not necessarily against a selection bias towards "people skills," but to do so and continue to pretend PCs are pure meritocracy is nauseating. It just comes back to the fact that the majority of Comp Sci PhDs have the same story: Halfway into their doctorate program they became severely disillusioned and started jockeying to just graduate and land a private sector job that essentially was just bribe money to keep them from working for the competition.

logisticseh 4y ago
> The "cream" rising to the top is often less genius and more politically savvy with the right connections on the PC. I'm generally nauseated when I interact with American CS academics. Every time I attend a conference, PC, or NSF panel, I am so glad I chose industry. It's like IRL twitter. (Europe seems to be better for some reason.) > If you're smart enough to rise, you'll get an offer from the private sector you simply cannot refuse. It doesn't matter if your passion is Academia, they can and will buy you out and own whatever you're working on. IME it's less about "offer you can't refuse" on the industry side and more about "offer you can't take" on the academic side. After 6 years of deferred income I simply could not take a job that paid $80K-$100K in an HCoL area or $65K-$80K in an LCoL area. I had loans to pay back, no 401K, and not enough savings for a down payment. If you want good people to stay in CS academia, I think a few things need to change: 1. First, and most importantly, the faculty culture. I don't really know how to describe the problem, but "the old folks are checked out and the young folks are Twitter personalities" is probably close. What's the point of being in academia if you have to be surrounded by the intellectual equivalent of used car salesmen, especially when you can go to industry and do interesting work without the BS? 2. Double the income of PhD students so that they aren't financially ruined by choosing the academic path. This isn't a super unreasonable request -- they'd still be paid less than their peers in industry while doing what's effectively a full time job. 3. Pay faculty more. Not a lot more... just, like, "at least what my undergrad students make at their first job after graduating". I think if you solve items 2 and 3, then item 1 will take care of itself.

kmmlng 4y ago

I think it's not surprising that conferences have to perform some selection. You need the right amount of participants and talks. If a high rejection rate achieves the right amount, it is hard to argue against it. But why are publications tied to conference attendance anyways? Sure, there also journals, but submitting to a journal tends to be an especially slow process. If you are in a fast-paced field, submitting to a journal is a dangerous game. Why can we not just upload our papers to something like arxiv and then give people the option to vote on papers analogous to reddit submissions, so that promising stuff organically rises to the top. That way it would at least be based on the opinions of a sizable number of judges, not just three preselected peers. Oh no, but what about peer review. What about it? Is it difficult to get past peer review at a top conference? Yes. Is it difficult to get past peer review in general? No. You can publish anything you want already, you will just have to jump through senseless hoops to do it. Why not skip the hoops and just upload it somewhere? We can still have journals and conferences that select high quality material from the uploaded papers and it will be an honour to be featured in one of those. You can still use features at conferences and journals as a bad metric to judge the quality of researchers, but the actual publishing will be decoupled from these institutions.

Eridrus 4y ago
ArXiv is fine for what it does, but it does not provide any sort of dissemination support. IMO the best thing about CS conferences is the poster track where you can walk by hundreds of posters, and the information is (when done well) much more easily digestible than papers, and you get to ask questions, and these are nowhere near their limits.

snickerbockers 4y ago

>Serving on a PC is a yeoman’s service, and the community owes them a debt of gratitude. However, I believe that a toxic culture has emerged. This blog is a call for PCs to change their priorities. does that acronym not stand for what i think it does?

warkdarrior 4y ago
PC stands for Program Committee, the selected group of people in charge of accepting and rejecting paper submissions.
secondcoming 4y ago
Probably not, but I don't know what it actually does stand for. It's definitely not Personal Computer
- Hackbraten 4y ago
  The acronym is not web-search friendly either. Been looking through this thread and search engines. Nothing helped. For example, not a single one of these [1] makes sense to me in that context. [1]: https://www.allacronyms.com/PC/university https://www.allacronyms.com/PC/university
  - warkdarrior 4y ago
    Some times you will see the acronym TPC for Technical Program Committee. Searching for 'conference TPC' returns some relevant hits.
  - xdavidliu 4y ago
    this is one of my pet peeves. You and dozens of other people probably spent collectively hundreds to thousands of minutes and brain cells wondering what PC stood for, while the people who originally typed "PC" instead of program committee saved maybe a fifth of a second.
    - snickerbockers 4y ago

m1117 4y ago

No, theres not!

seydor 4y ago

the most toxic ... if you exclude all the others

magpi3 4y ago

Academia as a whole is such an interesting place. I value it highly because I value learning. I love school, I loved the conversations and deep dives into interesting waters that nourished me in both college and graduate school, and I loved the opportunity to be fully immersed, without needing to get a job, in an atmosphere of learning. But the rest of it? The petty hierarchies, the papers born of countless hours of hard work by extremely talented people that few people will ever read, the wage labor of adjuncts, the elitism, the comfortable cowardice of tenure - that all should be burned in fire. I once read a book - I wish I could remember its name - that noted that even though we live in a democratic republic fueled by at least somewhat meritocratic capitalism, institutions from the feudal era still exist, the most notable being the church and the university. That was eye opening when I read it, and to this day it has jaded my view of academia and the people who tether themselves to it. All of those noble, high minded academics, fighting for their place in a feudal structure that they don't dare challenge. I know I sound a bit like an asshole when I write this, but: I can't truly look up to anyone who would resign themselves to a structure like that.

bo1024 4y ago
If the last paragraph is your perspective on being a pawn in an academic structure, I would love to hear what you think of being a pawn in a corporation.
- blacksmithgu 4y ago
  Not OP, but at at least most corporate pawns are cynically aware of how artificial the system is and that you need to play games to get ahead (or just coast along if you don't care). Academia has a luster of meritocracy when in reality you need to game things just as hard to become successful.
- magpi3 4y ago
  I suppose a big enough corporation can feel the same, at least in terms of internal politics, but the difference for me is the tenure (well-paid, almost impossible to fire) and adjunct (close to minimum wage, no health insurance, completely disposable) dynamics. Indefensible. And I think the people with tenure don't object to this model simply because they enjoy their privilege.

dekhn 4y ago

Nothing has made me happier than to simply leave the academic/publishing side of CS and focus entirely on things that I find interesting and can do on a single desktop computer. If I share, I share and give people my code and ideas freely, but don't publish. Why would I want to take all my hard work and beg somebody with power to put it on a piece of paper? If I have something useful to say I'll put it in arxiv. Peer reviewers just wasted my time.

hannob 4y ago

Honestly this only touches a small fraction of how absurd the whole publishing system with conferences is. I mean just thing about some obvious issues: Large parts of computer science limit how much science they can publish (and thus effectively share with others) by the number of conferences people want to organize. There's also a very obvious discrimination issue, as most "high tier" conferences are either in the US or (to a lesser extent) in the EU. And it's pretty crazy that people do transatlantic flights to go to a conference in order to publish a paper, even if they don't really want to go to the conference.

adamsmith143 4y ago

The lack of Novelty point feels off the mark, at least in some fields. At least in ML its extremely common for a paper to just report a new benchmark on some well known dataset with virtually no new real contributions other than more time spent hyperparemeter turning

FabHK 4y ago

One side effect I didn't see mentioned in the article: One professor of mine spoke of the LPU, the least publishable unit. So, if you're lucky enough to have some novel ideas, and build something nice out of it, don't put it all into one coherent and easily digested journal paper! The number of publications counts. Instead, chop it into little pieces that are just "novel" or noteworthy enough (LPUs), and publish them separately. Publication list inflation accomplished; and scientific progress/intelligibility/successful communication be damned.

grayclhn 4y ago
Honestly, I think this is often said cynically but is a good practice overall. Would you rather have to read and understand one giant commit reflecting 2 years of work, or 10 well-documented and logically complete individual commits?
- blacksmithgu 4y ago
  The issue is that novel paper ideas will be split across multiple years (and even multiple conferences), making it much harder to actually see the whole picture for a reader. Each little piece of the paper will often also be bloated with unnecessary extra detail in order to reach the threshold for "minimum publishable paper".
  - holidaygoose 4y ago
    Using the same code analogy as the parent, this is like code with unnecessary extra commenting. Seems okay to me.
  - grayclhn 4y ago
    Splitting up a groundbreaking idea into so many papers that the idea is lost is 1) going beyond a “minimal publishable unit” and 2) not in the authors’ interest, since getting credit for a groundbreaking idea in a correspondingly prestigious outlet is much better than getting credit for 2 or 3 bad ideas. I’m sure there’s a level of novelty where 2 irrelevant papers is better for the author than 1 single paper, but I don’t think we should design academic publishing around slightly-better-than-mediocre contributions.

faangiq 4y ago

CS has a ton of people in it and they produce a lot of garbage papers. Oh you retrained your ML model? Boom, new paper.

22SAS 4y ago
I completely agree with this. I have seen papers produced by a bunch of newly minted PhD's and the first thought that came in my mind was "WTF is this? There's nothing in this that's new or makes sense". A lot of publishing, from outside the top research universities, seems to be mainly at showing a higher publication count to get more funding and impress tenure committees. Some of the professors, who do this, make you wonder how they get their PhD in the first place. Back in my masters, I was giving a presentation on a paper from FB, where they were going over scheduling policies they had been using in their hadoop clusters. The professor reviewing asked me a question "You're taking about scheduling tasks in distributed environments, you're talking about Hadoop, Hadoop is for files what scheduling does it do?". Some of us there had a big "WTF" plastered on our face. Couldn't even argue cause this prof. thought he was a hotshot and would make life hell for students who did. In his own words to a student "I can write a compiler, I can write an operating system, but I can't make head or tail of what you're doing".

raphlinus 4y ago

This topic is close to my heart. I do research on the fundamentals of 2D graphics, some of it cutting edge (especially GPU techniques), others just selecting the best known techniques (for example, right now I'm doing a bit of a deep dive into robust cubic and quartic polynomial root finders, not really academically publishable, but potentially hugely useful for others working on similar problems). I made a serious attempt to submit some of my GPU monoid work to academic conferences earlier this year, got rejected. As someone who doesn't have the "publish or perish" incentive of actually being in academia, it's really just not worth it. I'm also doing inquiry into UI, for example architectural patterns in reactive systems. The current state of academic literature on this topic seems to be terrible (though please point out counterexamples!). Most of what's written is marketing material for UI frameworks - there's an explosion of those, especially in the JavaScript world, but very little synthesis of the core concepts. I wouldn't even attempt to try to submit an academic paper on this topic, though I think it would likely be useful to the world. It feels like there should be a space to publish work that is not novel in an academic sense, but useful. Wikipedia is not it (their articles on cubic and quartic equations are garbage when it comes to numerical concerns), Stack Overflow is not it, academic conferences and journals are not it. I'm using my blog, generally successfully, but it feels there should be a more systematic approach.

YorkshireSeason 4y ago
I found myself in a somewhat similar situation. I learned to thing long-term: (1) Publish initial ideas not in top conferences, but in second rate venues. Most publications in top conferences are, clearly, refinements of ideas that were first presented in inchoate form, in less prestigious workshops. Paradoxically, prestigious conferences don't like brand new ideas, but they like polished papers. And they like concrete open problems being solved. If a paper starts "We are solving the longstanding open problem from [17]" then that dramatically increases acceptance probability: most submissions to prestigious conferences are not rejected b/c they are wrong, but because it's unclear why they are more siginficant than all the other submissions. Solving a concrete open problem gives you social proof that the problem is significant (others worked on it) and hard (people failed). How do you create concrete open problems? By leaving them as concrete open problems in a previous paper. (2) Create your own conferences. All now famous conferences started as small, low prestige workshops 10-20 years ago, e.g. the "Oakland" security workshop (now: IEEE Symposium on Security and Privacy). I learned this from my post-doc mentor, who is now a famous scientist with a whole research tradition to his name, that he created, bootstrapping from a first workshop paper. PS, is your monoid work online, is it this https://arxiv.org/abs/2205.11659 https://arxiv.org/abs/2205.11659 ? I'm interested in monoids on GPUs.

kazinator 4y ago

This one has an easy fix! If you're in computing and want to produce garbage that is accepted anyway, don't write papers: write software.

cauefcr 4y ago
There are PR reviews worse than the #2 reviewer of most papers, so it's all about new software or forks, don't even bother upstreaming. /s

paulpauper 4y ago

The emphasis on novelty has deep roots in academic publishing. It used to be that publishing was expensive, and any repetition came at the expense of other things that could have been published. Today, however, publishing is essentially free. Yes, publishing is cheap or free, but attention is still scarce.

mirker 4y ago
In hot fields, you have arXiv papers 2-3 generations ahead of peer review. Some have more citations than typical accepted papers. Peer review does not limit people’s attention.

hedora 4y ago

I’d like to see a histogram of the number of resubmissions a paper goes through before acceptance. There are definitely papers that get accepted on the first try, and others that get resubmitted 5+ times. It doesn’t take many papers in the second group to drag acceptance rates down.

sgt101 4y ago

A lot of this is caused by the bad faith of some state actors. Many computer science conferences are under systematic assault from these places, which systematically swamp the PC's with submissions. What needs to happen is : - Regionalisation; make conferences regional only. So that submission can only come from that area or small group of nations. This will reduce travel demands and increase plurality. - Sharp constraints on personal submission: one and only one paper as an author by anyone. - Block outs : you get in one year, you skip a year.

yodsanklai 4y ago

I don't see a "culture of rejection". There's rejection whenever there's competition and a selection. There are tons of computer science conferences and not all of them are elitist. Even mediocre papers get published. Should we have all the submissions in the world accepted at the top conferences so that no researcher feels left behind?

bcantrill 4y ago

It's a good article, but it's not making the (obvious?) jump: the conference model in computer science is broken. No other discipline does it this way, and it creates for ourselves this terrible program committee problem. The journal model has plenty of its own flaws, but it at least allows for iterative work. (That is, instead of rejecting a work outright, a journal can work with an author on the flaws, even if that requires doing substantial new work.) This is not to lionize the journal model, which has plenty of its own flaws -- and indeed, I personally think computer science should use its current laggard status as an opportunity to find a wholly new model, preferably one that is much more amenable to practitioner participation.[0] [0] https://www.usenix.org/conference/atc16/technical-sessions/presentation/cantrill https://www.usenix.org/conference/atc16/technical-sessions/p...

raindear 4y ago

A solution is a site for publishing papers, with community discussions. People can rate/upvote papers, and these ratings can be used to build several metrics for papers, authors and reviewers.