6 ms·
As an academic, I have not so secretly been enjoying the shit storm of revelations lately. Science got along fine for thousands of years without Journals and Co
by godelski 3y ago
As an academic, I have not so secretly been enjoying the shit storm of revelations lately. Science got along fine for thousands of years without Journals and Conferences deciding what's "correct" and not. It's been a failed experiment. Journals started as a way to help distribute works to other researchers to improve upon the old method of just mailing it to one another. Arxiv is great for peer review. Your work is actually fucking available and not behind some pay wall, where your peers can... review it. Only thing would be better is if we were using OpenReview so we could track discussions, but I'll admit that could get messy real quick as anyone that's open sourced their research work will tell you (lots of questions like "I trained my model, how do I test it?" and "I'm getting a cuda out of memory error, how do I fix this?").
Arxiv/preprints are peer review
- xorbax 3y agoThe thing that established journals do better than arXiv is clarity You can upload whatever trash methods you want, but a normal journal will have at least one guy who tells you to wipe your ass and make your bullshit presentable in public, if only because they're expected to be gatekeepers for a minimum standard of supposed reproducibility.
- godelski 3y agoIn practice I'm not seeing much of a difference. Maybe it is just being in ML, where if you wait for conferences you're far behind. If a paper is that shitty, it is usually very apparent. Like if a paper isn't in latex you know... I mean there's a lot of garbage in conferences and journals too, I just haven't found it to be a meaningful signal. And it is still silly that people call it "peer review." Peer review is not 3-4 randos briefly glancing my work in an adversarial setting who say my work is not novel because it is the same as some unrelated work that they didn't read either; peer review is the grad student building on top of my work, peer review is lucidrains rewriting my work from scratch, it is Ross Wightman integrating it into timm and retraining, it is the forks that use my work for projects (hobby or professional). Peer review is peers looking and reviewing. More peer review happens on Twitter than these conferences. You can say these conferences and journals are a form of peer review, but we gotta stop saying that just because something is a preprint that it isn't peer reviewed. That's just incorrect. Peer review is when peers review.
- xorbax 3y agoSure, peer review is peer review. It depends on your peers. I'm a chemist, who has considerable established competition. So our reviwers know the systems and studied the method in grad school and sweated over it as a postdoc and had to innovate for cash in a professional capacity while doing something other than what we were doing. They'd excoriate for vague methods, or poor explanations, or general nonsense. You had to explain everything and make sure to refence the Big boys you were close to but definitely not stealing from. I think ML is unformed enough that there isn't that sort of public stricture as a random field in chemistry. Feel lucky. We don't even get to hide behind beautiful LaTeX, because there's too much benchwork for anybody's boss to ever give a fuck about it.
- godelski 3y ago> You had to explain everything and make sure to refence the Big boys you were close to That sounds problematic > there isn't that sort of public stricture as a random field in chemistry. Feel lucky. It's pure noise over here. I don't feel lucky, I feel frustrated. I get 3 reviewers and 3 completely different reasons to reject, often nonsensical (not joking, recently was rejected because a broken reference link to the appendix). Then submit to another conference and it is another 3 reviewers with 3 different reasons to reject that are completely different from the previous ones. Often asking me why I didn't cite arxiv papers that were released after the submission deadline. ACs don't care as long as reviewers agree because they gotta keep that acceptance rate low. It's playing the fucking lottery except your graduation and career depends on it despite people frequently saying conferences don't matter they'll still critique on lack of top tier publications. Your system doesn't sound great and also sounds frustrating, but I'll trade you. Regardless, I don't have faith in either system to significantly determine if a work is good (I do believe they can identify bad works, just not good works).
- bschne 3y ago> I get 3 reviewers and 3 completely different reasons to reject, often nonsensical Could one problem with these kinds of reviews be that the reviewers have absolutely zero skin in the game? Like, you can write complete nonsense and nobody will blame you personally for it, you have no stake in the outcome of the submission/paper what so ever, and you're much more incentivized to come up with reasons to reject than to say "good enough". I realize the latter part can be good in theory, as it sets a high bar, but it also often feels like this goes awry when not kept in check somehow.
- dheera 3y ago> make your bullshit presentable in public Triple integrals and sigmas over sets, instead of clearly stated limits, and that are not analytically solvable, is not "presentable". Scientists do this stuff on purpose to make their papers harder to read. > expected to be gatekeepers for a minimum standard of supposed reproducibility They don't enforce this at all. Otherwise they would demand code and training data be released for ML papers.
- godelski 3y ago> Scientists do this stuff on purpose to make their papers harder to read. Not just that, but if your paper has math that the reviewer doesn't understand they're more likely to think the work is good and rigorous. It's not like they read it anyways. > and training data be released for ML papers. Other than checkpoints and hyper-parameters, what do you want? The wandb logs? I do try to encourage people to save all relevant training parameters in checkpoints (I personally do). This even includes seeds.
- dheera 3y ago> what do you want? Hyperparameters yes, but also the data used for training. I should be able to reproduce the checkpoint bit-for-bit by training from scratch. If their training process is not deterministic, also release the random seed used.
- godelski 3y agoOh yeah, that I agree. I'm kinda upset Google is frequently pushing papers with JFT and 30 different versions of it and making conclusions based on pre-training with it. This isn't really okay for publication. Plus it breaks double blind! I'd be okay if say CVPR enforced that they train on public datasets and can only add proprietary after acceptance (but you've seen my views on these venues anyways). All ML training is non-deterministic. That's kinda the point. But yeah, people should include seeds AND random states. People forget the latter. I also don't know why people just don't throw args (including current iteration and important metrics) into their checkpoints. We share this frustration.
- crucialfelix 3y agoThat could be an LLM prompt. Cleanliness and structure analysis checks for papers would be very useful tool.
- willis936 3y ago[dead]
- xmcqdpt2 3y agoEh, not really. There are exceptions [1] but most journals don't expect the reviewers to even attempt to reproduce results, which makes sense given how specialized and expensive scientific experiments often are. As a reviewer on open code papers I would usually try to run the provided code, it didn't always work and that wasn't always addressed before publication. (I was also usually the only one who even tried.) Usually peer review is more about making sure the work is novel and interesting, fits the journal's audience and doesn't have any glaring flaws. Not entirely unlike code review: if it builds, merge it, and we can address problems in a future PR. Those are basically the reviewer instructions you get from most journals IME. [1] OrgSyn famously requires a reproduction from one of its editors lab before it accepts any paper, http://www.orgsyn.org/about.aspx http://www.orgsyn.org/about.aspx It has a very high reputation amongst chemists, even if it's "impact rating" is low. High impact journals are not usually considered the most accurate.
- hn_throwaway_99 3y agoI don't think you are really arguing against what the parent poster was saying. That is, I interpreted the parent commenter as saying that journals require that submissions at the very least be in a clear, understandable, "your paper must be at least verifiable (or falsifiable)" format, not that they actually attempt to reproduce the results.
- davidktr 3y ago(not OP) Verifiability/falsifiability are big words, mostly it is not clear what that means in a specific case. Crucially, that is not what journals/editors/reviewers do. They check if they find the contribution convincing, novel, and in line with the discipline's community standards, nothing more.
- xorbax 3y agoNo, I think a half decent paper is expected to 3ither explain their methods or reference a paper that explains their method You don't get to handwave away the instructions of your experiment in my mind. Maybe other fields are fine without that, but I would never write a paper that doesn't clearly explain how I made samples or reference a paper which does. To do otherwise is bad science. It's not about a reviewer replicating it, it's about anybody replicating it in a year or 30 years.
- zamalek 3y ago> if only because they're expected to be gatekeepers for a minimum standard of supposed reproducibility. There are decades of beta-amyloid papers that show that this is false. Like any for-profit entity, premium journals' primary purpose is to satiate shareholders - it is their feduciary responsibility to get away with as much as possible in the name of profit. That's not to say that open access doesn't have equally as concerning issues: which of the two is better or worse is an extremely difficult call to make.
- lazide 3y agoJust because someone wiped their ass doesn't mean they're not still full of shit. It just means it's less stinky/noticable. I think I did take this analogy too far though.
- whimsicalism 3y ago> Science got along fine for thousands of years without Journals and Conferences deciding what's "correct" and not The 'republic of letters' was much more similar to the journal-style approach than to modern arxiv, IMO.
- godelski 3y agoThat was one way, but it definitely was not the only way. A lot was people writing letters to one another and sharing papers by word of mouth. There will always be elitist groups, yeah, but we shouldn't support them. It's about doing good work, not gatekeeping. Reviewing is supposed to be critical, but that's not the same as adversarial. Science is about progressing human knowledge and we need to ask if this system is accomplishing that or if we're just caught it Goodhart's nightmare.
- SiempreViernes 3y agoThe work of keeping spurious results out is pretty important for the building of reliable knowledge though...
- godelski 3y agoI'm not seeing good evidence that journals or conferences do any better of a job at this than pre-print servers do. They do have a slight edge, but that is not that great. The edge is because the default position is to reject and so identifying just a small percentage of good papers does affect this outcome, but that's not a meaningful signal. Have an extremely high false positive reject rate and a high false positive accept rate doesn't make the signal meaningful, it just means it is very noisy. I don't give this very low signal any meaning because the truth is that good works filter to the top regardless of the journals. They rise because peers share them, not because they got the stamp mark from a journal.
- heliophobicdude 3y agoHas there been any serious discussion about using GitHub for peer reviews? Specifically the pull request functionality?
- rsfern 3y agoJOSS does something like this and it’s awesome. I’m not sure about the pull request workflow though, how would that work for review? Is your concept more for collaboration? https://joss.theoj.org/ https://joss.theoj.org/
- godelski 3y agoYeah, but more in the CS world and specifically with ML. There are still plenty of CS groups that don't make their source available.
- einpoklum 3y agoGitHub would not be relevant in this respect because: * It's owned by a (single) commercial corporation, Microsoft. * There is censorship both by content and in some respects by country of origin. * The code is closed. but otherwise it's an interesting idea. The C++ standardization committee uses GitHub to track papers submitted to them, see: https://github.com/cplusplus/papers https://github.com/cplusplus/papers
- tensor 3y agoI don't understand how pull requests would help with reviews. Peer review is generally looking at the methodology and analysis, not simple typos or wording that might be corrected by a pull request.
- efficax 3y agoscience in the modern sense is far less than 1000 years old, and emerged in tandem with systems of publication which no did not use the contemporary review system but did have editors who decided whether or not to publish submissions. not defending the peer review process as it currently exists but it’s not a bizarre outlier in the history of science
- danielbln 3y agoAlso, the scientific method took shape in what, the 17th century? Thousands of years, that's quite the stretch for "science".
- xorbax 3y agoEventually somebody will say "what if arXive, but we had really smart people who were volunteer experts read this bullshit first and decide if it was good enough? It would be like arXive but way more efficient for the end user!"
- godelski 3y agoThat's not far from how it actually works. 90% of what I read is from arxiv, and before it has been accepted into a journal or conference. Most of the time that I'm visiting a conference website is to submit a paper or get the proper citation for the paper I'm writing. Like I'm not sure what you expect, just dumb people to read arxiv? If that were the case then no one would be submitting works to arxiv. The reason works go there is because researchers are reading them. They then talk about the works to their peers, post on twitter, blog, or whatever. And guess what, that's also exactly how works that are published in journals and conferences get passed around because there's a million of them too and no way for people to sort them. No one is sitting around ranking papers. That process happens by citations. Who do you think the end user is? Laymen?
- SiempreViernes 3y agoAre you aware that the origin of Arxiv was as a pre-print archive? People read stuff on there because it was freely accessible drafts of published work, and in my field that is still the most common practice.
- drtgh 3y agoThe popular Journals have an strange business, · By one side the Journal reclaim a high amount of money for being able TO TRY to publish on them (APCs fee), more if one want to be published for to allow free access, as to read the publication in such Journals require subscription, are pay walled. · By other side, the reviewers of such papers, academics and researchers from other universities, don't receive economical compensation for doing the review, while the taken APCs fee its supposed for doing it? only in few cases the review is compensated by some programs of the reviewer's university or other 3rd parties. Then, what is for all those pay-walls and publish-fees? Sounds like a big filter... for what? for who? Because can be seen it's not working just as a mere "quality" filter. Being a paper published in one of those popular Journals, or being done directly in medias like arXiv, both sources needs the same amount of "grain of salt" about what is being read, until others replicate experiments, or contrast theories. So IMHO, outsider point of view, from time to time I have to use Sci-Hub in fact, I understand if they decide to don't publish in such popular Journals. I consider absolutely legitimate when they do it in distribution channels like arXiv. What I really would like it were exists an open research platform without pay walled papers, like happens in arXiv, and in addition, peer reviews through the platform were possible, as to read such reviews, and also, equally important, to have available debates about the paper, like if it were hacker news format.
- blablabla123 3y ago> Sounds like a big filter... for what? for who? Because can be seen it's not working just as a mere "quality" filter. To me it seems it keeps not only a wall around papers and journals. But also about whole academia which is probably an unwanted side effect since inside and outside of academia there seems to be consensus that knowledge transfer could be better. Also it seems everybody who can rather does research outside (AI/ML, Chemistry, Pharmacy).
- godelski 3y agoI mostly agree with you. But I absolutely despise debates. They are more often than not worthless. They are based on charisma and how good you are at language, not facts. Discussion is the right word. I've often thought of pushing directly to OpenReview rather than just arxiv, but discussions can get off base quickly and novices can dominate (while academic Twitter is great, this happens there a lot). It is not our job as a researcher to communicate our ideas to laymen, that's the job of science communicators. Sure, it is great and a bonus, but please stop asking on my github how to do something that is in the pytorch intro tutorial. Read the tutorial or attend my class. Open discussions are nice in theory, but HN and Reddit are good examples at how these systems often get over burdened with noise. There's a catch-22 and so it is a tough call.
- EvgeniyZh 3y ago> Science got along fine for thousands of years without Journals and Conferences deciding what's "correct" and not. And was available only to rich bored people who could self teach themselves
- _aavaa_ 3y agoRight, because a 40$-per-article paywall has done wonders for accessibility.
- EvgeniyZh 3y agoI guess you're just trolling, but in case you're serious, yes, being able to group together as scientists (university) and pay a single (or small amount) of entities to get all the scientific results, peer reviewed, has done wonders. Being able as university/funding agency to evaluate someone's scientific output without actually reading their papers (because other experts did it) allowed scaling. And to be honest, I do not know a single person how pays $40 for a paper, there are so many ways to get it for free (including just asking the authors). There is space for improvements, but "get rid of journals and conferences and hope scientists will somehow figure it out" is not a solution
- godelski 3y agoAnd pre-print servers have even accelerated science. Because frankly this is a MCMC system because the nature of the game is everyone stumbling around in the dark. I am a scientist and I'm all for "get rid of journals and conferences BECAUSE we already do figure it out." It isn't "somehow" we're already fucking doing it. We aren't going to journals and searching them, we go to arxiv. We read what our peers share with us. We read what comes out of other groups that we know that are doing similar work to us. Journals and conferences aren't helping us do science, they are helping us advance our careers because we live in a publish or perish ecosystem that quantifies our work based on the prestige of a conference who is prestigious because they reject 80% of works. Nowhere in here is a measure for the actual quality of a work. That is often a difficult task and can only be done by peers, which is already happening. Identifying bad works is easy but identifying good works is hard. I'm sorry, but we already live in the system that you're suggesting is silly and it's been this way for centuries. Journals are just people capitalizing on our work and labor. Conferences at least put us in the same room, but they have no business claiming that they can accurately ascertain the quality of a work.
- evouga 3y agoArxiv has its own problems. You can find dozens of proofs on the arxiv that P=NP and that P!=NP. What does peer “review” look like here? Most people won’t bother to write an article specifically rebutting one random incorrect proof. Now somebody tomorrow posts a proof that P!=NP. Will people pay attention? If it’s a “big name” or somebody at a university with a good PR office then yes; otherwise no. Doesn’t seem ideal to me.
- godelski 3y agoHow many of those works actually hit the front page of HN? How many do you hear of? The reason it is little to none is because peer review happens. If someone posted that to HN it would likely be trashed very quickly and fail to make it to the front page. That's peer review in action. Now does shit get through sometimes? Yeah. But I'm not convinced this is at a meaningfully higher rate than any top tier journal or conference, if at all.
- fsckboy 3y ago> Science got along fine for thousands of years doesn't upend your point, but just for the record, what we call science has only been around for about 500 years
- ayakang31415 3y agoAlso, I believe we should focus more on rigorous replication process rather than publishing new things from scientific researchers. Peer review is not enough to check the claims as it merely checks on reasoning of the science instead of experimental confirmation unless it is a very simple and short experiment. In other words, instead of directing funds to exploratory research, more money should be spent on multiple experimental verifications by independent researchers as I believe there are so many papers that need retraction due to unsubstantiated claims.