4 ms·
> Journal papers are limited by length and so was our assignment I have always thought this was a bit silly and that it creates really weird effects that also
by godelski 3y ago
> Journal papers are limited by length and so was our assignment
I have always thought this was a bit silly and that it creates really weird effects that also decrease readability. An interesting point is that reviewers are not required to read the appendix of works. So everything is required to be in the front matter. This is a bit silly when we do things like research graphics or do generative works and such. You want to include images and samples but then your space is eaten up. What if you want to discuss analysis on those images and explore some? You could easily do this on a blog but you're forced to throw this into the appendix. But then a reviewer can ask a question that's explained there and your work can still get rejected because it isn't in the front matter. Another weird incentive is that people end up padding works to fit page limits. This is because if you turn in a shorter paper reviewers will frequently reject your work the same way your boss might not think you're working if they don't see you at your desk.
We live in the 21st century and we still publish like it's the 15th. Computers gave us the ability to embed images, which is why there are so many more graphs and charts now, and it's not like more pages cost more. So just remove it. Some papers should be only a few pages and there's nothing wrong with that. Some papers should be far larger and there's nothing wrong with that. It's just weird to set these up considering they were likely created under other constraints but momentum continued and we back justify the continued decisions (there is something to be said about readability, but that can just be a reason to reject).
Side note: CS groups typically publish in conferences
- outrun86 3y agoDistill.pub was one effort to modernize publishing in CS. Chris Olah wrote some thoughts [1] about why he didn’t feel it was tenable. Seems like the primary challenge was the additional effort and skill involved in crafting rich-content/interactive material. [1] https://distill.pub/2021/distill-hiatus/ https://distill.pub/2021/distill-hiatus/
- godelski 3y agoHonestly, I don't get why we don't just submit to OpenReview and call it a day. Paper is visible and distributed. There are comment sections where peer review can not just happen, but happen in the open (added bonus!). You can iterate and even see the difference between submissions. What is the conference/journal providing that isn't covered here? A stamp of approval? From a well known noisy system that creates other disincentives?
- sideshowb 3y agoPromotion and filtering I guess? What does a record label provide when you can just upload music to Spotify?
- godelski 3y ago> What does a record label provide when you can just upload music to Spotify? I believe this is an illustrative example in support of my proposition, not against. Many artists are in fact turning away from record labels in favor of self publishing. Similarly for books. But I will say that I still think there's value and so I'll expand on my ideas about conferences. I think they should exist, but be focused on meet and greets. So instead of being an indicator of the validity of work, have them invite authors to speak about their works. Allow others to sign up for poster sessions. How to do that appropriately does need to be worked out, but there's nothing wrong with it simply being under recommendation from the advisement of the organizing members. Yes, there will still be preferential bias, but I do mean "still" because we do have preferential biases towards certain institutions and labs. This would just make it a bit more explicit that they are not the arbitrators of quality but just treated as a "reward." Importantly I think this allows opening the doors for different kinds of research that are not incentivized by our systems. Most important being reproduction
- vladms 3y agoNot sure the openness of the review would solve so many problems of the system. For example would not touch like reproducibility and data and code availability. Then you will need moderation (or do you imagine that things will be civilized between people on the internet?) and would need to manage various possibilities of bullying/targeting/etc. Of course these things can happen now, but difference would be between a potentially fully automated and simple system and something very clunky (be friends with an editor, convince him to report who are the reviewers, manage to recognize another of his papers, etc.)
- godelski 3y ago> For example would not touch like reproducibility and data and code availability. These are different issues, which are certainly important. But I do think in some way this would help. OpenReview does allow you to post comments many months after. Effectively think about this as a GitHub issues page. It certainly could be organized better but it is better than what exists now. OR also has links for code and community implementations (as does arxiv now). Here's an example that has all these things[0]. Granted data is missing, but I don't see why this can't also be integrated, but would need to also push cultural norms. > Then you will need moderation I think OR has this a bit solved, similarly arxiv. They are not anonymous accounts and are tied to your ORCID record. Arxiv requires you to have a verifier that is already someone with an arxiv account. Yes, this can be abused, but it is also an easier moderation problem that say Reddit or HN even. I think if you're posting bullying comments under a named profile, then it is good that that is visible so others can see. Mind you, bullying does already exist but it is just behind closed doors. It is worse now because only the Area Chair can take action and often they are over worked and works do get dismissed (which results in A LOT of wasted time, and money) because of this bullying. The larger the field, the more noise too and the more this happens. It is just far less common to see people bullying in public than behind closed doors. I must stress though, that there is no perfect system here. There is no system that can make the amount of bullying 0. So we have to be careful in our critiques because there will always be valid critiques that are in fact of concern (like this one) but are fundamentally unsolvable. The question then becomes if we improve upon the existing frameworks and if whatever costs have been made are worth the added benefits. So I just want to make sure that this idea isn't killed because an impossible bar, despite the critique being valid. Edit: I'd actually add that this system encourages reproduction. Because if we still measure on citations and number of publications this means that reproduction works can still count towards those metrics and thus someone's career advancements. The whole conference/journal system currently discourages such effort in favor of the absurdly nebulous novelty concept (which also makes papers noisy). My proposal would also allow for the publication of failures, which is also an important thing for academics. [0] https://openreview.net/forum?id=Hkxzx0NtDB https://openreview.net/forum?id=Hkxzx0NtDB
- jltsiren 3y agoPage limits force you to focus. As a researcher, you are often expected to communicate your ideas in 1 page, 3 pages, 10 pages, or 30 pages, for various purposes. If a journal asks for a 10-page paper, you write a 10-page paper. If a conference asks for a 1-page abstract, you write a 1-page abstract. Most people reading a paper are not interested in going through all the details, and those details should usually not be in the main paper. It's also easier to find reviewers for short papers than for long ones. Some the issues you mention are specific to CS conferences. Because there is only time for 1-2 rounds of reviews, the reviews focus more on accepting/rejecting the paper and less on clearing any misunderstandings before judging it. Conferences are are also more likely to have one-size-fits-all page limits, while journals often have several catagories of papers with different expectations of length.
- sideshowb 3y agoI think desirability of page limits is very subject specific. Some people will just waffle if you don't give them a page limit. Other times it means there's not room for the technical details.
- godelski 3y agoBut the reviewers can reject if it isn't enough or reject if it is too much. What I'm arguing is the alignment mechanism already exists. The page limit is over constraining
- godelski 3y ago> Page limits force you to focus. This can be solved in better ways, which is, in fact, reviewers. I'm okay with a soft requirement but a standardization is what I'm getting at as being problematic. Some papers are noisy because they should be 3 pages but are 10. Some papers are noisy because they are 10 pages and should be 30. There is no universal rule, and that's what I'm getting at. > It's also easier to find reviewers for short papers than for long ones. That's a separate problem that needs to be addressed, but is not easy. > Some the issues you mention are specific to CS conferences. Yes, but the author here is CS and we are on a CS focused website. But in general what I said isn't specific to conferences. If conferences are the problem then let's abandon them in favor of good science instead of keeping them around (or turn them into being meetup focused). Certainly the lack of back and forth between authors and reviewers is not a meaningful review process (most author rebuttals are limited to one page and often reviewers are not aligned in critiques). Are we all on the same team (better science) or strictly competing against one another?