6 ms·

Sure, peer review is peer review. It depends on your peers. I'm a chemist, who has considerable established competition. So our reviwers know the systems and s

by xorbax 3y ago

Sure, peer review is peer review. It depends on your peers. I'm a chemist, who has considerable established competition. So our reviwers know the systems and studied the method in grad school and sweated over it as a postdoc and had to innovate for cash in a professional capacity while doing something other than what we were doing. They'd excoriate for vague methods, or poor explanations, or general nonsense. You had to explain everything and make sure to refence the Big boys you were close to but definitely not stealing from. I think ML is unformed enough that there isn't that sort of public stricture as a random field in chemistry. Feel lucky. We don't even get to hide behind beautiful LaTeX, because there's too much benchwork for anybody's boss to ever give a fuck about it.

godelski 3y ago
> You had to explain everything and make sure to refence the Big boys you were close to That sounds problematic > there isn't that sort of public stricture as a random field in chemistry. Feel lucky. It's pure noise over here. I don't feel lucky, I feel frustrated. I get 3 reviewers and 3 completely different reasons to reject, often nonsensical (not joking, recently was rejected because a broken reference link to the appendix). Then submit to another conference and it is another 3 reviewers with 3 different reasons to reject that are completely different from the previous ones. Often asking me why I didn't cite arxiv papers that were released after the submission deadline. ACs don't care as long as reviewers agree because they gotta keep that acceptance rate low. It's playing the fucking lottery except your graduation and career depends on it despite people frequently saying conferences don't matter they'll still critique on lack of top tier publications. Your system doesn't sound great and also sounds frustrating, but I'll trade you. Regardless, I don't have faith in either system to significantly determine if a work is good (I do believe they can identify bad works, just not good works).
- bschne 3y ago
  > I get 3 reviewers and 3 completely different reasons to reject, often nonsensical Could one problem with these kinds of reviews be that the reviewers have absolutely zero skin in the game? Like, you can write complete nonsense and nobody will blame you personally for it, you have no stake in the outcome of the submission/paper what so ever, and you're much more incentivized to come up with reasons to reject than to say "good enough". I realize the latter part can be good in theory, as it sets a high bar, but it also often feels like this goes awry when not kept in check somehow.
  - godelski 3y ago
    > Could one problem with these kinds of reviews be that the reviewers have absolutely zero skin in the game? Actually, it is worse. They probably have skin in the game, and incentives to reject! Most CS fields publish through a conference system. This is a zero or one shot system (if you get a rebuttal phase, _if_ you want to count that). These conferences have acceptance rates that they "need to maintain" to keep their rankings. In other words, you are competing against all other papers being submitted to the same conference, not just papers with similar topics. Even if it is only a little, rejecting a paper actually increases the odds that your submission makes it through. But yeah, you're in the right ball park. This is also why you will quite frequently see conference review guidelines blatantly violated and why you will see area chairs and metareviewers just not care. They all have incentives to encourage rejection, let alone be impartial. I think the system works when the community is small and there is accountability among peers. Accountability creates a dampening effect on bad behavior. But at scale, you only need a few bad actors to setup a feedback system and to not just spoil the entire barrel, but the entire shipment. Edit: I should also add that there's an additional negative incentive. You are not judged by how often you review, how many reviews you perform, or how good your review is (hard to measure). So reviewing ends up taking away time from the very limited time you have to do work that you are actually evaluated on. This is likely why reviews are so rushed. There's a feedback loop too, since many will see that others rush when reviewing their work so they get tired and end up rushing when reviewing the work of others. "If they aren't going to give my work their time, why should I have their work my time?" thinking grows.
- deleted 3y ago
  [deleted]
- YeGoblynQueenne 3y ago
  >> I don't feel lucky, I feel frustrated. I get 3 reviewers and 3 completely different reasons to reject, often nonsensical (not joking, recently was rejected because a broken reference link to the appendix). The people who review your papers in conferences and ask you why you didn't cite future arxiv papers are the same people who put their work on arxiv and cite each other's preprints. You can't rely on the process of "peer-review" on arxiv any more than you can rely on the conference peer-reviews because they're performed by the same people, and they're people who don't know what they're doing. The sad truth is that the vast majority of the researchers in the machine learning community haven't got a clue what the hell they're doing, nor do they understand what anyone else is doing. The typical machine learning paper is poorly motivated, vaguely written, and makes no claims, nor presents any results, other than "our system beats some other systems". As to reproducibility, hell if we know whether any of that work is really reproducible. Everybody who references it ends up doing something completely different anyway and they just cite prior work as an excuse to avoid doing their job and properly motivating their work. The people who write those papers eventually get to be reviewers (by sheer luck), or sub-reviewers. They have no idea how to write a good paper, so they have no idea how to write a good review, either. And they couldn't recognise a good paper if it jumped up and bit them in the cojones. I love to cite Geoff Hinton on this one: GH: One big challenge the community faces is that if you want to get a paper published in machine learning now it's got to have a table in it, with all these different data sets across the top, and all these different methods along the side, and your method has to look like the best one. If it doesn’t look like that, it’s hard to get published. I don't think that's encouraging people to think about radically new ideas. Now if you send in a paper that has a radically new idea, there's no chance in hell it will get accepted, because it's going to get some junior reviewer who doesn't understand it. Or it’s going to get a senior reviewer who's trying to review too many papers and doesn't understand it first time round and assumes it must be nonsense. Anything that makes the brain hurt is not going to get accepted. And I think that's really bad. https://www.wired.com/story/googles-ai-guru-computers-think-more-like-brains/ https://www.wired.com/story/googles-ai-guru-computers-think-... So the problem is not arxiv or not arxiv, the problem is that peers in peer review lack expertise and knowledge and they can't do their job well.
  - 3y ago
- cryptonector 3y ago
  > > You had to explain everything and make sure to refence the Big boys you were close to > That sounds problematic Bingo, it is problematic. It's part of how "the Big boys" get their academic and compensation (investment in their firms) rewards. You have to play the publish-or-perish, paper rank game to get ahead and stay ahead in academia, and this leads to all sorts of problems. Authors don't want to question "the Big boys" because that leads to their papers not getting published because the "the Big boys" and their bootlicker wannabes are the reviewers and they will exact their tribute. Make it to "the Big boys" club and now you're a gatekeeper and now you're also responsible for perpetuating this system. It's why Nature-style peer review needs to become a thing of the past. I'm not saying that "popular" (for a value of "popular" that involves peers at large, not the public at large) peer review is / will be without problems. But it seems to me that it will -at least for a while- be less corruptible.
  - godelski 3y ago
    Yeah, this is a hill I will die on. I love researching, but once I grab my PhD I do not plan to push to journals/conferences unless it is requested in a job. It just holds no meaning and I'm tired of pretending it does. Perpetuating the system harms my fellow researchers, kills innovation, and just kicks the growing can down the road. The other hill I will die on is that we shouldn't refer to journal/conference publishing as "peer review." This is one form of peer review, but there are MANY more. And as far as I'm concerned, 3 randos that briefly skim my paper in an adversarial setting (zero sum) looking to reject works barely constitutes peer review. Peer review is what happens when your peers read your work, test it, build upon it, replicate it, etc. We need to stop this language because it helps no one.
    - cryptonector 3y ago
      > Peer review is what happens when your peers ... I would add: when your peers' review commentary is public.
      - godelski 3y ago
        I love when this happens tbh, but it is rather rare. Often I can tell that a paper was rejected for dumb reasons and that gives me a good signal to actually read it.
bschne 3y ago
> (...) make sure to refence the Big boys you were close to but definitely not stealing from Much as giving credit is important, this is unfortunately also how you end up with intro sections littered with references to the same ten papers everyone's read already just to back up some extremely vague/general statement.
- godelski 3y ago
  I hate this. Honestly, I'd love to see short papers. Just cite what is relevant and not much else. But I see citations exploding. Look at this paper[0], it is 34 pages in total and 5.5 of those are the bibliography! It has almost 100 references! ~14 pages are images (not figures... images). Nearly an entire page worth of material is just the citations! Not the text referencing the citations, but the citations themselves. It is absolute chaos. No, this isn't a survey paper. So it is serving no purpose other than greasing their peers. [0] https://arxiv.org/abs/2303.11435 https://arxiv.org/abs/2303.11435

by xorbax 3y ago

godelski 3y ago

> You had to explain everything and make sure to refence the Big boys you were close to That sounds problematic > there isn't that sort of public stricture as a random field in chemistry. Feel lucky. It's pure noise over here. I don't feel lucky, I feel frustrated. I get 3 reviewers and 3 completely different reasons to reject, often nonsensical (not joking, recently was rejected because a broken reference link to the appendix). Then submit to another conference and it is another 3 reviewers with 3 different reasons to reject that are completely different from the previous ones. Often asking me why I didn't cite arxiv papers that were released after the submission deadline. ACs don't care as long as reviewers agree because they gotta keep that acceptance rate low. It's playing the fucking lottery except your graduation and career depends on it despite people frequently saying conferences don't matter they'll still critique on lack of top tier publications. Your system doesn't sound great and also sounds frustrating, but I'll trade you. Regardless, I don't have faith in either system to significantly determine if a work is good (I do believe they can identify bad works, just not good works).

bschne 3y ago
> I get 3 reviewers and 3 completely different reasons to reject, often nonsensical Could one problem with these kinds of reviews be that the reviewers have absolutely zero skin in the game? Like, you can write complete nonsense and nobody will blame you personally for it, you have no stake in the outcome of the submission/paper what so ever, and you're much more incentivized to come up with reasons to reject than to say "good enough". I realize the latter part can be good in theory, as it sets a high bar, but it also often feels like this goes awry when not kept in check somehow.
- godelski 3y ago
  > Could one problem with these kinds of reviews be that the reviewers have absolutely zero skin in the game? Actually, it is worse. They probably have skin in the game, and incentives to reject! Most CS fields publish through a conference system. This is a zero or one shot system (if you get a rebuttal phase, _if_ you want to count that). These conferences have acceptance rates that they "need to maintain" to keep their rankings. In other words, you are competing against all other papers being submitted to the same conference, not just papers with similar topics. Even if it is only a little, rejecting a paper actually increases the odds that your submission makes it through. But yeah, you're in the right ball park. This is also why you will quite frequently see conference review guidelines blatantly violated and why you will see area chairs and metareviewers just not care. They all have incentives to encourage rejection, let alone be impartial. I think the system works when the community is small and there is accountability among peers. Accountability creates a dampening effect on bad behavior. But at scale, you only need a few bad actors to setup a feedback system and to not just spoil the entire barrel, but the entire shipment. Edit: I should also add that there's an additional negative incentive. You are not judged by how often you review, how many reviews you perform, or how good your review is (hard to measure). So reviewing ends up taking away time from the very limited time you have to do work that you are actually evaluated on. This is likely why reviews are so rushed. There's a feedback loop too, since many will see that others rush when reviewing their work so they get tired and end up rushing when reviewing the work of others. "If they aren't going to give my work their time, why should I have their work my time?" thinking grows.
deleted 3y ago
[deleted]
YeGoblynQueenne 3y ago
>> I don't feel lucky, I feel frustrated. I get 3 reviewers and 3 completely different reasons to reject, often nonsensical (not joking, recently was rejected because a broken reference link to the appendix). The people who review your papers in conferences and ask you why you didn't cite future arxiv papers are the same people who put their work on arxiv and cite each other's preprints. You can't rely on the process of "peer-review" on arxiv any more than you can rely on the conference peer-reviews because they're performed by the same people, and they're people who don't know what they're doing. The sad truth is that the vast majority of the researchers in the machine learning community haven't got a clue what the hell they're doing, nor do they understand what anyone else is doing. The typical machine learning paper is poorly motivated, vaguely written, and makes no claims, nor presents any results, other than "our system beats some other systems". As to reproducibility, hell if we know whether any of that work is really reproducible. Everybody who references it ends up doing something completely different anyway and they just cite prior work as an excuse to avoid doing their job and properly motivating their work. The people who write those papers eventually get to be reviewers (by sheer luck), or sub-reviewers. They have no idea how to write a good paper, so they have no idea how to write a good review, either. And they couldn't recognise a good paper if it jumped up and bit them in the cojones. I love to cite Geoff Hinton on this one: GH: One big challenge the community faces is that if you want to get a paper published in machine learning now it's got to have a table in it, with all these different data sets across the top, and all these different methods along the side, and your method has to look like the best one. If it doesn’t look like that, it’s hard to get published. I don't think that's encouraging people to think about radically new ideas. Now if you send in a paper that has a radically new idea, there's no chance in hell it will get accepted, because it's going to get some junior reviewer who doesn't understand it. Or it’s going to get a senior reviewer who's trying to review too many papers and doesn't understand it first time round and assumes it must be nonsense. Anything that makes the brain hurt is not going to get accepted. And I think that's really bad. https://www.wired.com/story/googles-ai-guru-computers-think-more-like-brains/ https://www.wired.com/story/googles-ai-guru-computers-think-... So the problem is not arxiv or not arxiv, the problem is that peers in peer review lack expertise and knowledge and they can't do their job well.
- 3y ago
cryptonector 3y ago
> > You had to explain everything and make sure to refence the Big boys you were close to > That sounds problematic Bingo, it is problematic. It's part of how "the Big boys" get their academic and compensation (investment in their firms) rewards. You have to play the publish-or-perish, paper rank game to get ahead and stay ahead in academia, and this leads to all sorts of problems. Authors don't want to question "the Big boys" because that leads to their papers not getting published because the "the Big boys" and their bootlicker wannabes are the reviewers and they will exact their tribute. Make it to "the Big boys" club and now you're a gatekeeper and now you're also responsible for perpetuating this system. It's why Nature-style peer review needs to become a thing of the past. I'm not saying that "popular" (for a value of "popular" that involves peers at large, not the public at large) peer review is / will be without problems. But it seems to me that it will -at least for a while- be less corruptible.
- godelski 3y ago
  Yeah, this is a hill I will die on. I love researching, but once I grab my PhD I do not plan to push to journals/conferences unless it is requested in a job. It just holds no meaning and I'm tired of pretending it does. Perpetuating the system harms my fellow researchers, kills innovation, and just kicks the growing can down the road. The other hill I will die on is that we shouldn't refer to journal/conference publishing as "peer review." This is one form of peer review, but there are MANY more. And as far as I'm concerned, 3 randos that briefly skim my paper in an adversarial setting (zero sum) looking to reject works barely constitutes peer review. Peer review is what happens when your peers read your work, test it, build upon it, replicate it, etc. We need to stop this language because it helps no one.
  - cryptonector 3y ago
    > Peer review is what happens when your peers ... I would add: when your peers' review commentary is public.
    - godelski 3y ago
      I love when this happens tbh, but it is rather rare. Often I can tell that a paper was rejected for dumb reasons and that gives me a good signal to actually read it.

bschne 3y ago

> (...) make sure to refence the Big boys you were close to but definitely not stealing from Much as giving credit is important, this is unfortunately also how you end up with intro sections littered with references to the same ten papers everyone's read already just to back up some extremely vague/general statement.

godelski 3y ago
I hate this. Honestly, I'd love to see short papers. Just cite what is relevant and not much else. But I see citations exploding. Look at this paper[0], it is 34 pages in total and 5.5 of those are the bibliography! It has almost 100 references! ~14 pages are images (not figures... images). Nearly an entire page worth of material is just the citations! Not the text referencing the citations, but the citations themselves. It is absolute chaos. No, this isn't a survey paper. So it is serving no purpose other than greasing their peers. [0] https://arxiv.org/abs/2303.11435 https://arxiv.org/abs/2303.11435