5 ms·

Thank you for these comments. I appreciate them and I'll consider them in my next draft. However, I would like to point out a few things; just so that we have t

• somethingsome 2y ago
  1. When I said we revise the paper between two submissions, I wasn't implying it was becoming "better". The message was that there is no general consensus around what should be expanded and what might be concise. Someone believes you should discuss prior work more, someone thinks the main algorithm requires more elaboration, someone wants you to talk more about BayesOpt etc., but you just have <10 pages in the main paper, and putting this stuff in the Appendix, or citing source, doesn't seem to be good enough in many cases (another comment in a sibling thread gives an example wrt GANs, and my experiences have been no different). That's exactly my point, the reviews do not converge because the message is too diffuse or not justified enough. I recently had a paper rejected because it was too difficult to understand, it was on 4 pages, now it's sent to a better journal and was expanded to 20 pages. The content was too big for a 4 pages content, we couldn't fit enough justifications. But in your paper you still have many places where the text could be shorter and clearer, gaining at least 1 page of content. Learning to write good research takes a lot of time, and a phd is the place where ideally this should happens. It's difficult, but you'll get there if you work on it enough! Read best paper awards of good conferences, notice how much material is there in the same number of pages, and reverse engineer what they did to make the paper clear, concise and easy to follow. 2. You say you randomly picked a few sentences to read; that's good for a casual discussion but that should not be how a review process functions. Some of the best reviewers I've encountered (and I hope I am continuing in that tradition) come back to say something like "I see what you're getting at, but your intro. doesn't sell it well enough; think about writing it like this ...". Rejecting based on random skimming is exactly one of the things I'm calling out. Let's face it - like a lot of things, high quality reviewing is hard. It isn't supposed to be quick or easy. You cannot choose who will read. But even for the more throughout readers, if it's difficult to understand / missing justifications from the beginning, they will give a bad review, even if they read the whole thing. Reading should be like a conversation with the author, if I find the conversation with the author through the paper too sloppy or erratic, I will not understand the message, that's what happens when I ask more justifications on some part to the author. It's because I couldn't follow the logic enough or I was not agreeing with some part, so I require more justifications. 3. Predicting how much to elaborate: this is probably an extension of the first point, but I feel like this has become way harder in the recent years. The rule that mostly works seems to be that if its not a trending topic explain it as much as you can, because cited background material is overlooked. This is unfair for areas that are not trending - the goal of research should be to situate itself closer to "explore" on the "explore-exploit" spectrum, but the review system today heavily favors "exploit". And like I mentioned, a page limit means that the publication game stacked against people not working on mainstream ideas. This should not be the case. I agree, there are no more general experts, everyone works in a very niche subfield, you don't get people that know the sota. Learning the good tradeoff is difficult. My threshold is: don't explain the math unless it's not self obvious why. For example for some equation I can give more insights on how it affects my method and if a parameter of the equation is very important to my method, a complete analysis of its effects and analogies and experiments to see its impact. I try to make the main story line as crystal clear as possible, if I deviate too much, it's better on a second paper. My experiments should reflect not trivial things. Finally I make sure the abstract corresponds to the text. I mainly don't work in deep learning, so by default my topics are extremely hard to find reviewers, I feel the pain. But it's my work to make them understand what I'm achieving and why it's important. Hope that helps :)
  • abhgh 2y ago
    I feel like we are debating slightly different perspectives, and with that lens I agree with what you say. Here is the difference in perspectives (and this is decoupled from this particular paper): your take is that today the reviews work in a certain way and here are some things we could do to maximize our chances of acceptance. My take is that reviews shouldn't work this way. To take some examples: 1. > You cannot choose who will read. Specifically no, but generally, yes. I'd expect the reviewer to understand ML. And if this is not the brand of ML they're familiar with, I'd expect them to put in the work to familiarize themselves during the review process, in the interest of fairness. After all are we not seeking out qualified reviewers for the review process? This is not just anyone who stumbles across a paper on the internet. 2. > message is too diffuse Any message would appear diffuse/opaque/abstract to someone unfamiliar with the area. This is exactly why an objective review process must equalize such communicative biases. This is partly facilitated by the conference picking the right reviewers and with their review-assignments, and partly it is also the duty a reviewer to fill in whatever gaps of comprehension that remain. 3. > Read best paper awards of good conferences, notice how much material is there in the same number of pages, and reverse engineer what they did to make the paper clear, concise and easy to follow. Good general advice but you are preaching to the choir. I do read best papers from various conferences and I run reading groups where we discuss papers from ongoing conferences. I run an applied ML research group in the industry - this pretty much comes with the job. Further, I don't think that best papers are head-and-shoulders above non-best papers; they are often voted to the top because they solve a broadly known problem, or they further the understanding of such a problem. Writing plays some role here, but is not the discriminative factor. 4. Requiring justifications. Yes, there is a rebuttal phase for that. Just to be doubly clear, I am not saying papers (and this paper) can't be improved. But that is not the argument I am making.
    • somethingsome 2y ago
      I totally understand your take, and you are right in some aspect. Reviews SHOULD NOT WORK IN THIS WAY. I'm totally agreeing on that, but consider this, the population is increasing, the number of topics are increasing, we are no more in a system where the review system work as it was designed to. When it was designed, only a few people got the chance to read, and information was not easily available. So, people became experts, and knowing the sota was mandatory. Now, due to the high quantity of (good and bad) researches, you cannot expect the review system to work properly. But you are still stuck in this system. So consider what is important: - do you want to write and hope that by chance the right people will read it, and they will be educated enough in your topic and have enough time at their disposal. Or - Do you think your idea is good and should be more known. If it's the later, it's your work to make your idea as clear as possible so that any (good) researcher can understand it, and therefore use it. We must work in the reality of the current system if we want to spread interesting ideas to the community. The publication system is a social system, and it evolves with the people inside, you want to write to spread knowledge. How can you do that if the probability is that the reader will not fully understand? The time is very limited and I always have many things to do, I only read what I filter as worthy enough. That filter is based on the quality of writing. If some paper is important but badly written, it will automatically fall in my 'if I have time to read it' and most of the time it will never reach the 'to read' category, because there are many many paper well written with good ideas inside. We work in a biased system. It's extremely difficult to find reviewers, we do what we can with what we have. I also was infuriated when I got a review saying 'you didn't explain structure from motion' in a conference with a topic on structure from motion. But the reality is this. If I want my papers to be read, I must adapt to my audience. > Read best paper awards of good conferences With that sentence, I did not mean 'read it for knowledge' but read it with the lenses of the writer, why did they present the topic in this way, what makes this paper clear and another on the same topic not clear at all. Reverse engineer the writing style. It's not about knowledge of the content of the paper, it's about communication. Best paper do not always have the best ideas inside, but they are presented in a way that even if the topic is difficult they provide insights on it. And often those insights are what readers want to read. The maths are not important, the important thing is the insight you give to the readers. That insight can be translated in their field of they internalized it enough.