5 ms·

The thing that established journals do better than arXiv is clarity You can upload whatever trash methods you want, but a normal journal will have at least one

by xorbax 3y ago

The thing that established journals do better than arXiv is clarity You can upload whatever trash methods you want, but a normal journal will have at least one guy who tells you to wipe your ass and make your bullshit presentable in public, if only because they're expected to be gatekeepers for a minimum standard of supposed reproducibility.

godelski 3y ago
In practice I'm not seeing much of a difference. Maybe it is just being in ML, where if you wait for conferences you're far behind. If a paper is that shitty, it is usually very apparent. Like if a paper isn't in latex you know... I mean there's a lot of garbage in conferences and journals too, I just haven't found it to be a meaningful signal. And it is still silly that people call it "peer review." Peer review is not 3-4 randos briefly glancing my work in an adversarial setting who say my work is not novel because it is the same as some unrelated work that they didn't read either; peer review is the grad student building on top of my work, peer review is lucidrains rewriting my work from scratch, it is Ross Wightman integrating it into timm and retraining, it is the forks that use my work for projects (hobby or professional). Peer review is peers looking and reviewing. More peer review happens on Twitter than these conferences. You can say these conferences and journals are a form of peer review, but we gotta stop saying that just because something is a preprint that it isn't peer reviewed. That's just incorrect. Peer review is when peers review.
- xorbax 3y ago
  Sure, peer review is peer review. It depends on your peers. I'm a chemist, who has considerable established competition. So our reviwers know the systems and studied the method in grad school and sweated over it as a postdoc and had to innovate for cash in a professional capacity while doing something other than what we were doing. They'd excoriate for vague methods, or poor explanations, or general nonsense. You had to explain everything and make sure to refence the Big boys you were close to but definitely not stealing from. I think ML is unformed enough that there isn't that sort of public stricture as a random field in chemistry. Feel lucky. We don't even get to hide behind beautiful LaTeX, because there's too much benchwork for anybody's boss to ever give a fuck about it.
  - godelski 3y ago
    > You had to explain everything and make sure to refence the Big boys you were close to That sounds problematic > there isn't that sort of public stricture as a random field in chemistry. Feel lucky. It's pure noise over here. I don't feel lucky, I feel frustrated. I get 3 reviewers and 3 completely different reasons to reject, often nonsensical (not joking, recently was rejected because a broken reference link to the appendix). Then submit to another conference and it is another 3 reviewers with 3 different reasons to reject that are completely different from the previous ones. Often asking me why I didn't cite arxiv papers that were released after the submission deadline. ACs don't care as long as reviewers agree because they gotta keep that acceptance rate low. It's playing the fucking lottery except your graduation and career depends on it despite people frequently saying conferences don't matter they'll still critique on lack of top tier publications. Your system doesn't sound great and also sounds frustrating, but I'll trade you. Regardless, I don't have faith in either system to significantly determine if a work is good (I do believe they can identify bad works, just not good works).
    - bschne 3y ago
      > I get 3 reviewers and 3 completely different reasons to reject, often nonsensical Could one problem with these kinds of reviews be that the reviewers have absolutely zero skin in the game? Like, you can write complete nonsense and nobody will blame you personally for it, you have no stake in the outcome of the submission/paper what so ever, and you're much more incentivized to come up with reasons to reject than to say "good enough". I realize the latter part can be good in theory, as it sets a high bar, but it also often feels like this goes awry when not kept in check somehow.
      - godelski 3y ago
        > Could one problem with these kinds of reviews be that the reviewers have absolutely zero skin in the game? Actually, it is worse. They probably have skin in the game, and incentives to reject! Most CS fields publish through a conference system. This is a zero or one shot system (if you get a rebuttal phase, _if_ you want to count that). These conferences have acceptance rates that they "need to maintain" to keep their rankings. In other words, you are competing against all other papers being submitted to the same conference, not just papers with similar topics. Even if it is only a little, rejecting a paper actually increases the odds that your submission makes it through. But yeah, you're in the right ball park. This is also why you will quite frequently see conference review guidelines blatantly violated and why you will see area chairs and metareviewers just not care. They all have incentives to encourage rejection, let alone be impartial. I think the system works when the community is small and there is accountability among peers. Accountability creates a dampening effect on bad behavior. But at scale, you only need a few bad actors to setup a feedback system and to not just spoil the entire barrel, but the entire shipment. Edit: I should also add that there's an additional negative incentive. You are not judged by how often you review, how many reviews you perform, or how good your review is (hard to measure). So reviewing ends up taking away time from the very limited time you have to do work that you are actually evaluated on. This is likely why reviews are so rushed. There's a feedback loop too, since many will see that others rush when reviewing their work so they get tired and end up rushing when reviewing the work of others. "If they aren't going to give my work their time, why should I have their work my time?" thinking grows.
- DoctorOetker 3y ago
  I agree with the core of your message, and you are far from alone with this opinion. The history of science is a sequence of profound platitudes, each time refining the prior ones. It is a part of awakening to pick the lowest hanging fruits first, just to get a foot in the door. While its not theoretically impossible to go straight from ignorance to -say- the postulates of quantum field theory, its incredibly unlikely to scale such a huge step at once, both for scientists at the frontier as well as each generation of students catching up. In this sense the profound platitudes are quasi necessary to build up an understanding of the world around us. Individually each outdated platitude may appear so very wrong when looking back with 20 / 20 hindsight, and most of these individual platitudes could plausibly have been skipped by scaling 2 or 3 steps at a time, but no one could scale all of them at once. Literally respect and review mean the same thing. To view or observe anew (re-). One may objectively claim that each step or correction of a prior platitude was an improvement of our worldview, even though in hindsight each step of correction also contributed the next erroneous stumbling block to be scaled. Throughout history policy decisions (from household decisions, to court decisions, to budget decisions, to research direction decisions, etc...) humans sought ways to settle matters. The desire for Finality is universal and justified. However the expectation that this universe came with a manual is not. A core tenet of the scientific mindset is the recognition that this finality is a temporary illusion. Selling this illusion of a recipe for finality is profitable and what journals and gatekeepers have gravitated toward, especially since information became dirt cheap to duplicate.
  - godelski 3y ago
    I think you missed my point. I have no problem citing, and actively encourage it. I do have a problem with someone rejecting my work because I failed to cite a work which would have been impossible for me to cite. The key is that it was released (as a preprint) AFTER my paper. Not only is it in bad taste to make concurrent works a requisite (happy to update btw), but it is ludicrous to require that I have a time machine. This isn't about respect and me complaining about pomp and circumstance. This is me complaining about the frequency that I have been criticized for not having a time machine. The point is that a system that allows people to reject works due to lack of access to a functioning time machine is not a system we should support.
dheera 3y ago
> make your bullshit presentable in public Triple integrals and sigmas over sets, instead of clearly stated limits, and that are not analytically solvable, is not "presentable". Scientists do this stuff on purpose to make their papers harder to read. > expected to be gatekeepers for a minimum standard of supposed reproducibility They don't enforce this at all. Otherwise they would demand code and training data be released for ML papers.
- godelski 3y ago
  > Scientists do this stuff on purpose to make their papers harder to read. Not just that, but if your paper has math that the reviewer doesn't understand they're more likely to think the work is good and rigorous. It's not like they read it anyways. > and training data be released for ML papers. Other than checkpoints and hyper-parameters, what do you want? The wandb logs? I do try to encourage people to save all relevant training parameters in checkpoints (I personally do). This even includes seeds.
  - dheera 3y ago
    > what do you want? Hyperparameters yes, but also the data used for training. I should be able to reproduce the checkpoint bit-for-bit by training from scratch. If their training process is not deterministic, also release the random seed used.
    - godelski 3y ago
      Oh yeah, that I agree. I'm kinda upset Google is frequently pushing papers with JFT and 30 different versions of it and making conclusions based on pre-training with it. This isn't really okay for publication. Plus it breaks double blind! I'd be okay if say CVPR enforced that they train on public datasets and can only add proprietary after acceptance (but you've seen my views on these venues anyways). All ML training is non-deterministic. That's kinda the point. But yeah, people should include seeds AND random states. People forget the latter. I also don't know why people just don't throw args (including current iteration and important metrics) into their checkpoints. We share this frustration.
crucialfelix 3y ago
That could be an LLM prompt. Cleanliness and structure analysis checks for papers would be very useful tool.
willis936 3y ago
[dead]
xmcqdpt2 3y ago
Eh, not really. There are exceptions [1] but most journals don't expect the reviewers to even attempt to reproduce results, which makes sense given how specialized and expensive scientific experiments often are. As a reviewer on open code papers I would usually try to run the provided code, it didn't always work and that wasn't always addressed before publication. (I was also usually the only one who even tried.) Usually peer review is more about making sure the work is novel and interesting, fits the journal's audience and doesn't have any glaring flaws. Not entirely unlike code review: if it builds, merge it, and we can address problems in a future PR. Those are basically the reviewer instructions you get from most journals IME. [1] OrgSyn famously requires a reproduction from one of its editors lab before it accepts any paper, http://www.orgsyn.org/about.aspx http://www.orgsyn.org/about.aspx It has a very high reputation amongst chemists, even if it's "impact rating" is low. High impact journals are not usually considered the most accurate.
- hn_throwaway_99 3y ago
  I don't think you are really arguing against what the parent poster was saying. That is, I interpreted the parent commenter as saying that journals require that submissions at the very least be in a clear, understandable, "your paper must be at least verifiable (or falsifiable)" format, not that they actually attempt to reproduce the results.
  - davidktr 3y ago
    (not OP) Verifiability/falsifiability are big words, mostly it is not clear what that means in a specific case. Crucially, that is not what journals/editors/reviewers do. They check if they find the contribution convincing, novel, and in line with the discipline's community standards, nothing more.
    - xorbax 3y ago
      No, I think a half decent paper is expected to 3ither explain their methods or reference a paper that explains their method You don't get to handwave away the instructions of your experiment in my mind. Maybe other fields are fine without that, but I would never write a paper that doesn't clearly explain how I made samples or reference a paper which does. To do otherwise is bad science. It's not about a reviewer replicating it, it's about anybody replicating it in a year or 30 years.
zamalek 3y ago
> if only because they're expected to be gatekeepers for a minimum standard of supposed reproducibility. There are decades of beta-amyloid papers that show that this is false. Like any for-profit entity, premium journals' primary purpose is to satiate shareholders - it is their feduciary responsibility to get away with as much as possible in the name of profit. That's not to say that open access doesn't have equally as concerning issues: which of the two is better or worse is an extremely difficult call to make.
- lazide 3y ago
  Just because someone wiped their ass doesn't mean they're not still full of shit. It just means it's less stinky/noticable. I think I did take this analogy too far though.

by xorbax 3y ago

godelski 3y ago

In practice I'm not seeing much of a difference. Maybe it is just being in ML, where if you wait for conferences you're far behind. If a paper is that shitty, it is usually very apparent. Like if a paper isn't in latex you know... I mean there's a lot of garbage in conferences and journals too, I just haven't found it to be a meaningful signal. And it is still silly that people call it "peer review." Peer review is not 3-4 randos briefly glancing my work in an adversarial setting who say my work is not novel because it is the same as some unrelated work that they didn't read either; peer review is the grad student building on top of my work, peer review is lucidrains rewriting my work from scratch, it is Ross Wightman integrating it into timm and retraining, it is the forks that use my work for projects (hobby or professional). Peer review is peers looking and reviewing. More peer review happens on Twitter than these conferences. You can say these conferences and journals are a form of peer review, but we gotta stop saying that just because something is a preprint that it isn't peer reviewed. That's just incorrect. Peer review is when peers review.

xorbax 3y ago
Sure, peer review is peer review. It depends on your peers. I'm a chemist, who has considerable established competition. So our reviwers know the systems and studied the method in grad school and sweated over it as a postdoc and had to innovate for cash in a professional capacity while doing something other than what we were doing. They'd excoriate for vague methods, or poor explanations, or general nonsense. You had to explain everything and make sure to refence the Big boys you were close to but definitely not stealing from. I think ML is unformed enough that there isn't that sort of public stricture as a random field in chemistry. Feel lucky. We don't even get to hide behind beautiful LaTeX, because there's too much benchwork for anybody's boss to ever give a fuck about it.
- godelski 3y ago
  > You had to explain everything and make sure to refence the Big boys you were close to That sounds problematic > there isn't that sort of public stricture as a random field in chemistry. Feel lucky. It's pure noise over here. I don't feel lucky, I feel frustrated. I get 3 reviewers and 3 completely different reasons to reject, often nonsensical (not joking, recently was rejected because a broken reference link to the appendix). Then submit to another conference and it is another 3 reviewers with 3 different reasons to reject that are completely different from the previous ones. Often asking me why I didn't cite arxiv papers that were released after the submission deadline. ACs don't care as long as reviewers agree because they gotta keep that acceptance rate low. It's playing the fucking lottery except your graduation and career depends on it despite people frequently saying conferences don't matter they'll still critique on lack of top tier publications. Your system doesn't sound great and also sounds frustrating, but I'll trade you. Regardless, I don't have faith in either system to significantly determine if a work is good (I do believe they can identify bad works, just not good works).
  - bschne 3y ago
    > I get 3 reviewers and 3 completely different reasons to reject, often nonsensical Could one problem with these kinds of reviews be that the reviewers have absolutely zero skin in the game? Like, you can write complete nonsense and nobody will blame you personally for it, you have no stake in the outcome of the submission/paper what so ever, and you're much more incentivized to come up with reasons to reject than to say "good enough". I realize the latter part can be good in theory, as it sets a high bar, but it also often feels like this goes awry when not kept in check somehow.
    - godelski 3y ago
      > Could one problem with these kinds of reviews be that the reviewers have absolutely zero skin in the game? Actually, it is worse. They probably have skin in the game, and incentives to reject! Most CS fields publish through a conference system. This is a zero or one shot system (if you get a rebuttal phase, _if_ you want to count that). These conferences have acceptance rates that they "need to maintain" to keep their rankings. In other words, you are competing against all other papers being submitted to the same conference, not just papers with similar topics. Even if it is only a little, rejecting a paper actually increases the odds that your submission makes it through. But yeah, you're in the right ball park. This is also why you will quite frequently see conference review guidelines blatantly violated and why you will see area chairs and metareviewers just not care. They all have incentives to encourage rejection, let alone be impartial. I think the system works when the community is small and there is accountability among peers. Accountability creates a dampening effect on bad behavior. But at scale, you only need a few bad actors to setup a feedback system and to not just spoil the entire barrel, but the entire shipment. Edit: I should also add that there's an additional negative incentive. You are not judged by how often you review, how many reviews you perform, or how good your review is (hard to measure). So reviewing ends up taking away time from the very limited time you have to do work that you are actually evaluated on. This is likely why reviews are so rushed. There's a feedback loop too, since many will see that others rush when reviewing their work so they get tired and end up rushing when reviewing the work of others. "If they aren't going to give my work their time, why should I have their work my time?" thinking grows.
DoctorOetker 3y ago
I agree with the core of your message, and you are far from alone with this opinion. The history of science is a sequence of profound platitudes, each time refining the prior ones. It is a part of awakening to pick the lowest hanging fruits first, just to get a foot in the door. While its not theoretically impossible to go straight from ignorance to -say- the postulates of quantum field theory, its incredibly unlikely to scale such a huge step at once, both for scientists at the frontier as well as each generation of students catching up. In this sense the profound platitudes are quasi necessary to build up an understanding of the world around us. Individually each outdated platitude may appear so very wrong when looking back with 20 / 20 hindsight, and most of these individual platitudes could plausibly have been skipped by scaling 2 or 3 steps at a time, but no one could scale all of them at once. Literally respect and review mean the same thing. To view or observe anew (re-). One may objectively claim that each step or correction of a prior platitude was an improvement of our worldview, even though in hindsight each step of correction also contributed the next erroneous stumbling block to be scaled. Throughout history policy decisions (from household decisions, to court decisions, to budget decisions, to research direction decisions, etc...) humans sought ways to settle matters. The desire for Finality is universal and justified. However the expectation that this universe came with a manual is not. A core tenet of the scientific mindset is the recognition that this finality is a temporary illusion. Selling this illusion of a recipe for finality is profitable and what journals and gatekeepers have gravitated toward, especially since information became dirt cheap to duplicate.
- godelski 3y ago
  I think you missed my point. I have no problem citing, and actively encourage it. I do have a problem with someone rejecting my work because I failed to cite a work which would have been impossible for me to cite. The key is that it was released (as a preprint) AFTER my paper. Not only is it in bad taste to make concurrent works a requisite (happy to update btw), but it is ludicrous to require that I have a time machine. This isn't about respect and me complaining about pomp and circumstance. This is me complaining about the frequency that I have been criticized for not having a time machine. The point is that a system that allows people to reject works due to lack of access to a functioning time machine is not a system we should support.

dheera 3y ago

> make your bullshit presentable in public Triple integrals and sigmas over sets, instead of clearly stated limits, and that are not analytically solvable, is not "presentable". Scientists do this stuff on purpose to make their papers harder to read. > expected to be gatekeepers for a minimum standard of supposed reproducibility They don't enforce this at all. Otherwise they would demand code and training data be released for ML papers.

godelski 3y ago
> Scientists do this stuff on purpose to make their papers harder to read. Not just that, but if your paper has math that the reviewer doesn't understand they're more likely to think the work is good and rigorous. It's not like they read it anyways. > and training data be released for ML papers. Other than checkpoints and hyper-parameters, what do you want? The wandb logs? I do try to encourage people to save all relevant training parameters in checkpoints (I personally do). This even includes seeds.
- dheera 3y ago
  > what do you want? Hyperparameters yes, but also the data used for training. I should be able to reproduce the checkpoint bit-for-bit by training from scratch. If their training process is not deterministic, also release the random seed used.
  - godelski 3y ago
    Oh yeah, that I agree. I'm kinda upset Google is frequently pushing papers with JFT and 30 different versions of it and making conclusions based on pre-training with it. This isn't really okay for publication. Plus it breaks double blind! I'd be okay if say CVPR enforced that they train on public datasets and can only add proprietary after acceptance (but you've seen my views on these venues anyways). All ML training is non-deterministic. That's kinda the point. But yeah, people should include seeds AND random states. People forget the latter. I also don't know why people just don't throw args (including current iteration and important metrics) into their checkpoints. We share this frustration.

crucialfelix 3y ago

That could be an LLM prompt. Cleanliness and structure analysis checks for papers would be very useful tool.

willis936 3y ago

[dead]

xmcqdpt2 3y ago

Eh, not really. There are exceptions [1] but most journals don't expect the reviewers to even attempt to reproduce results, which makes sense given how specialized and expensive scientific experiments often are. As a reviewer on open code papers I would usually try to run the provided code, it didn't always work and that wasn't always addressed before publication. (I was also usually the only one who even tried.) Usually peer review is more about making sure the work is novel and interesting, fits the journal's audience and doesn't have any glaring flaws. Not entirely unlike code review: if it builds, merge it, and we can address problems in a future PR. Those are basically the reviewer instructions you get from most journals IME. [1] OrgSyn famously requires a reproduction from one of its editors lab before it accepts any paper, http://www.orgsyn.org/about.aspx http://www.orgsyn.org/about.aspx It has a very high reputation amongst chemists, even if it's "impact rating" is low. High impact journals are not usually considered the most accurate.

hn_throwaway_99 3y ago
I don't think you are really arguing against what the parent poster was saying. That is, I interpreted the parent commenter as saying that journals require that submissions at the very least be in a clear, understandable, "your paper must be at least verifiable (or falsifiable)" format, not that they actually attempt to reproduce the results.
- davidktr 3y ago
  (not OP) Verifiability/falsifiability are big words, mostly it is not clear what that means in a specific case. Crucially, that is not what journals/editors/reviewers do. They check if they find the contribution convincing, novel, and in line with the discipline's community standards, nothing more.
  - xorbax 3y ago
    No, I think a half decent paper is expected to 3ither explain their methods or reference a paper that explains their method You don't get to handwave away the instructions of your experiment in my mind. Maybe other fields are fine without that, but I would never write a paper that doesn't clearly explain how I made samples or reference a paper which does. To do otherwise is bad science. It's not about a reviewer replicating it, it's about anybody replicating it in a year or 30 years.

zamalek 3y ago

> if only because they're expected to be gatekeepers for a minimum standard of supposed reproducibility. There are decades of beta-amyloid papers that show that this is false. Like any for-profit entity, premium journals' primary purpose is to satiate shareholders - it is their feduciary responsibility to get away with as much as possible in the name of profit. That's not to say that open access doesn't have equally as concerning issues: which of the two is better or worse is an extremely difficult call to make.

lazide 3y ago
Just because someone wiped their ass doesn't mean they're not still full of shit. It just means it's less stinky/noticable. I think I did take this analogy too far though.