22 ms·
Right, it's a low bar, and it's meant to be a low bar, and that's fine. It's not no bar. Peer review adds some credibility to a paper. And the venue does as we
by tptacek 3y ago
Right, it's a low bar, and it's meant to be a low bar, and that's fine.
It's not no bar. Peer review adds some credibility to a paper. And the venue does as well. It's just less credibility than the popular imagination assumes.
- AlbertCory 3y agoThis had a refreshing degree of realism. Naturally, any perfection-minded person will be offended by that. A decent journal will stop using reviewers who do a lousy job, and complaints by authors about their paper's reviewers ought to be listened to and not dismissed as sour grapes.
- godelski 3y ago> A decent journal will stop using reviewers who do a lousy job, and complaints by authors about their paper's reviewers ought to be listened to and not dismissed as sour grapes. As a data point in ML conferences, I had a reviewer give me a strong reject with only a few sentences. First was a generic "not novel" with no further explanation. The rest was a complaint that my paper had serious flaws and pointed to an intentionally redacted link and a broken cross-reference. We reported them to the AC and the other two reviewers were borderline and weak accept, but both had low confidence (of course the inane reviewer was high confidence). The result was that the reviewer's rebuttal was much more angry and the weak accept said "authors addressed my concerns, but I've decided to lower my score." I tell this story because at a certain point it is not a problem with bad reviewers, it is a problem with bad management. The most important thing that a system can have is self correcting behavior. A system that gives a bad faith actor a slap on the wrist and allows them to then escalate their bad behavior and influence others is not a well functioning system. Granted, this is just one example, but I've seen many others (such as a theory paper being rejected for lack of experiments). I've started collecting author tweets I find about these kinds of situations, but I think there's a lot more that aren't easily conveyed on social media. I do really want to promote discussion of these things because I think we should always be improving our systems. You're right that nothing is every perfect, but we can still recognize flaws without being consumed with the impossible goal of perfectionism.
- AlbertCory 3y agoIndeed, my paper to CACM based on this: https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2399580 https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2399580 was shitcanned by a Microsoft reviewer (if you're the one and you're reading this: GFY) First of all, he ignored it for six months, and it wasn't until I begged the editor that he got it moving. Then he gave one round of critique, and when I addressed it, he said, "the patent law changed" and closed the process. He knew nothing about patents. As I said, the editors should not defer to these douchebags. Fortunately I don't need pubs for my career, but some people do.
- godelski 3y agoFunny enough, none of my first author works have been "published" but since they are on arxiv I have a competitive citation count and h-index (most cites are from my first author works). Only group upset at me is my grad school. It's extremely frustrating because I have all indications of doing good work and clearly the community agrees. I think the funniest part of it all is that I've offered to show them my reviews and so we can discuss if I am doing something wrong or "just having bad luck." But no one wants to even entertain the idea that the system is dysfunctional. I will never understand why in CS we use a conference system as our indicator of merit. It is an antagonistic zero sum game. There is no recourse for giving a bad faith review and you are incentivized to reject works. It's easier to reject and thus less time consuming -- every paper has flaws and limitations, just point at them and ignore context--, you marginally increase the chance of your own paper getting in, and venues use acceptance rates as the main measure to indicate their level of prestige (if I cared enough I'd just get LLMs to spam papers to abuse this). Every part of the system that does something good and beneficial to the scientific community relies entirely on people acting in good faith and against other incentives they have, favoring the purity of science. That clearly doesn't scale and clearly sets a stage for bad actors to overwhelm. I think we only continue this because they've successfully convinced good faith actors that there is no other way and that it isn't as bad as it is. I just wonder how much money and time is lost due to all of this (before we even account for the ridiculousness of charging for what arxiv does for free. The paper is given to them for free as well as the reviewing is done for free, and much of the organizing committee and all that is also free labor. Something this important shouldn't have so many similarities to a scam that seeks to extract money from the government and universities).