5 ms·

I feel like we are debating slightly different perspectives, and with that lens I agree with what you say. Here is the difference in perspectives (and this is d

by abhgh 2y ago

I feel like we are debating slightly different perspectives, and with that lens I agree with what you say. Here is the difference in perspectives (and this is decoupled from this particular paper): your take is that today the reviews work in a certain way and here are some things we could do to maximize our chances of acceptance. My take is that reviews shouldn't work this way. To take some examples: 1. > You cannot choose who will read. Specifically no, but generally, yes. I'd expect the reviewer to understand ML. And if this is not the brand of ML they're familiar with, I'd expect them to put in the work to familiarize themselves during the review process, in the interest of fairness. After all are we not seeking out qualified reviewers for the review process? This is not just anyone who stumbles across a paper on the internet. 2. > message is too diffuse Any message would appear diffuse/opaque/abstract to someone unfamiliar with the area. This is exactly why an objective review process must equalize such communicative biases. This is partly facilitated by the conference picking the right reviewers and with their review-assignments, and partly it is also the duty a reviewer to fill in whatever gaps of comprehension that remain. 3. > Read best paper awards of good conferences, notice how much material is there in the same number of pages, and reverse engineer what they did to make the paper clear, concise and easy to follow. Good general advice but you are preaching to the choir. I do read best papers from various conferences and I run reading groups where we discuss papers from ongoing conferences. I run an applied ML research group in the industry - this pretty much comes with the job. Further, I don't think that best papers are head-and-shoulders above non-best papers; they are often voted to the top because they solve a broadly known problem, or they further the understanding of such a problem. Writing plays some role here, but is not the discriminative factor. 4. Requiring justifications. Yes, there is a rebuttal phase for that. Just to be doubly clear, I am not saying papers (and this paper) can't be improved. But that is not the argument I am making.

somethingsome 2y ago
I totally understand your take, and you are right in some aspect. Reviews SHOULD NOT WORK IN THIS WAY. I'm totally agreeing on that, but consider this, the population is increasing, the number of topics are increasing, we are no more in a system where the review system work as it was designed to. When it was designed, only a few people got the chance to read, and information was not easily available. So, people became experts, and knowing the sota was mandatory. Now, due to the high quantity of (good and bad) researches, you cannot expect the review system to work properly. But you are still stuck in this system. So consider what is important: - do you want to write and hope that by chance the right people will read it, and they will be educated enough in your topic and have enough time at their disposal. Or - Do you think your idea is good and should be more known. If it's the later, it's your work to make your idea as clear as possible so that any (good) researcher can understand it, and therefore use it. We must work in the reality of the current system if we want to spread interesting ideas to the community. The publication system is a social system, and it evolves with the people inside, you want to write to spread knowledge. How can you do that if the probability is that the reader will not fully understand? The time is very limited and I always have many things to do, I only read what I filter as worthy enough. That filter is based on the quality of writing. If some paper is important but badly written, it will automatically fall in my 'if I have time to read it' and most of the time it will never reach the 'to read' category, because there are many many paper well written with good ideas inside. We work in a biased system. It's extremely difficult to find reviewers, we do what we can with what we have. I also was infuriated when I got a review saying 'you didn't explain structure from motion' in a conference with a topic on structure from motion. But the reality is this. If I want my papers to be read, I must adapt to my audience. > Read best paper awards of good conferences With that sentence, I did not mean 'read it for knowledge' but read it with the lenses of the writer, why did they present the topic in this way, what makes this paper clear and another on the same topic not clear at all. Reverse engineer the writing style. It's not about knowledge of the content of the paper, it's about communication. Best paper do not always have the best ideas inside, but they are presented in a way that even if the topic is difficult they provide insights on it. And often those insights are what readers want to read. The maths are not important, the important thing is the insight you give to the readers. That insight can be translated in their field of they internalized it enough.
- godelski 2y ago
  > I also was infuriated when I got a review saying 'you didn't explain structure from motion' in a conference with a topic on structure from motion. But the reality is this. If I want my papers to be read, I must adapt to my audience. Honestly, I hate this take. I don’t think it’s good for science or academia. Papers do not need be readable to everyone. The point is to be readable by other experts in your niche. Otherwise, I don’t know who to write to, and that’s exactly the same problem the parent is having. Writing to too broad of an audience also makes papers unnecessarily long. You have to spend more time motivating the work and more time on the background. This has spinoff effects where reviewers can demand you cite them, contributing to the citation mining nightmare. I’ve seen 8 page papers with 100+ references (the paper I referenced has 78). This is more what we expect from a survey paper. When background sections are minimal you can’t justify asking unless you are critical to the exact problem being solved. Every paper rewrite is time and money that should be better spent on research or other activities. Every rewrite is an additional submission into the next conference. I don’t think labs are submitting 20+ papers per round because they wrote 20 papers in that 3 months (with some exceptions) but rather because they wrote a bunch and are recycling works from the previous few years. This increases reviewer load as well. The question then is how people enter a topic. Truth be told, I don’t think it’s any easier than when papers had under 30. For reference, that one Cybenko paper we all know has under 30 references but is 10.5 pages. What I think we should do instead is allow citing of blogs and encourage people to write tutorials. I think this would actually be a really useful task for 2nd and 3rd year PhD students. You learn a lot when writing those things and that’s the stage where you should be entering expert level at your niche. The problem is that we have no incentivize to do any of the other critical tasks in academia. This is why I personally hate it. We are hyper focused on this novelty thing but in reality that doesn’t exist and is highly subjective. It just encourages obscurification, which we’ve routinely seen from high profile labs. We work in ML, how are we not all keenly aware of reward hacking and knock-on effects?! I honestly think the fact that we cannot get our house in order is evidence that we can’t safely build safe AGI yet. This is certainly magnitudes easier of a task, not to mention has significant reward (selfishly, it highly benefits us too!). Everyone feels that something is off but no one wants to do anything about it. We’ve only implemented half added measures that are more about signaling. Can’t let an LLM review for you? But the author is responsible for proving the reviewer used one? We’re all ML experts… we all know this isn’t possible except in the most cases. It’s as if you got shot while blindfolded and the police won’t investigate until you bring them evidence of who shot you and with what kind of gun. It shouldn’t matter if a review is bad because it was written by an LLM or because it was by a human. Just like it shouldn’t matter if you were shot or stabbed. > The maths are not important, the important thing is the insight you give to the readers. I also hate this take. The math often __is__ the insight. I agree that a lot of papers have needless math (look at any diffusion paper or any paper with attention copy pasting the same equations). But other works need them. The reason to use math is the same reason we program. If there was an easier way to communicate, we would (note: math isn’t just symbols, it can be words too). Math and programming are hard because they are languages that are precise and dense. The precision is important when communicating. Yes, it might take longer to parse but it is unambiguous when interpreted (it is also easier to parse when you’re trained and in the habit. Just like any other language). I think we lost our way in academia. We got caught up in by excitement. We let the bureaucrats take too much control and dictate the universities. We got lost in our egos (definitely not new) and too focused on prestige and fame. Our systems should be fighting these things, not enabling or encouraging them. Yes, the people at the top benefit from these systems, but the truth is that even they benefit from fixing things.

by abhgh 2y ago

somethingsome 2y ago

I totally understand your take, and you are right in some aspect. Reviews SHOULD NOT WORK IN THIS WAY. I'm totally agreeing on that, but consider this, the population is increasing, the number of topics are increasing, we are no more in a system where the review system work as it was designed to. When it was designed, only a few people got the chance to read, and information was not easily available. So, people became experts, and knowing the sota was mandatory. Now, due to the high quantity of (good and bad) researches, you cannot expect the review system to work properly. But you are still stuck in this system. So consider what is important: - do you want to write and hope that by chance the right people will read it, and they will be educated enough in your topic and have enough time at their disposal. Or - Do you think your idea is good and should be more known. If it's the later, it's your work to make your idea as clear as possible so that any (good) researcher can understand it, and therefore use it. We must work in the reality of the current system if we want to spread interesting ideas to the community. The publication system is a social system, and it evolves with the people inside, you want to write to spread knowledge. How can you do that if the probability is that the reader will not fully understand? The time is very limited and I always have many things to do, I only read what I filter as worthy enough. That filter is based on the quality of writing. If some paper is important but badly written, it will automatically fall in my 'if I have time to read it' and most of the time it will never reach the 'to read' category, because there are many many paper well written with good ideas inside. We work in a biased system. It's extremely difficult to find reviewers, we do what we can with what we have. I also was infuriated when I got a review saying 'you didn't explain structure from motion' in a conference with a topic on structure from motion. But the reality is this. If I want my papers to be read, I must adapt to my audience. > Read best paper awards of good conferences With that sentence, I did not mean 'read it for knowledge' but read it with the lenses of the writer, why did they present the topic in this way, what makes this paper clear and another on the same topic not clear at all. Reverse engineer the writing style. It's not about knowledge of the content of the paper, it's about communication. Best paper do not always have the best ideas inside, but they are presented in a way that even if the topic is difficult they provide insights on it. And often those insights are what readers want to read. The maths are not important, the important thing is the insight you give to the readers. That insight can be translated in their field of they internalized it enough.

godelski 2y ago
> I also was infuriated when I got a review saying 'you didn't explain structure from motion' in a conference with a topic on structure from motion. But the reality is this. If I want my papers to be read, I must adapt to my audience. Honestly, I hate this take. I don’t think it’s good for science or academia. Papers do not need be readable to everyone. The point is to be readable by other experts in your niche. Otherwise, I don’t know who to write to, and that’s exactly the same problem the parent is having. Writing to too broad of an audience also makes papers unnecessarily long. You have to spend more time motivating the work and more time on the background. This has spinoff effects where reviewers can demand you cite them, contributing to the citation mining nightmare. I’ve seen 8 page papers with 100+ references (the paper I referenced has 78). This is more what we expect from a survey paper. When background sections are minimal you can’t justify asking unless you are critical to the exact problem being solved. Every paper rewrite is time and money that should be better spent on research or other activities. Every rewrite is an additional submission into the next conference. I don’t think labs are submitting 20+ papers per round because they wrote 20 papers in that 3 months (with some exceptions) but rather because they wrote a bunch and are recycling works from the previous few years. This increases reviewer load as well. The question then is how people enter a topic. Truth be told, I don’t think it’s any easier than when papers had under 30. For reference, that one Cybenko paper we all know has under 30 references but is 10.5 pages. What I think we should do instead is allow citing of blogs and encourage people to write tutorials. I think this would actually be a really useful task for 2nd and 3rd year PhD students. You learn a lot when writing those things and that’s the stage where you should be entering expert level at your niche. The problem is that we have no incentivize to do any of the other critical tasks in academia. This is why I personally hate it. We are hyper focused on this novelty thing but in reality that doesn’t exist and is highly subjective. It just encourages obscurification, which we’ve routinely seen from high profile labs. We work in ML, how are we not all keenly aware of reward hacking and knock-on effects?! I honestly think the fact that we cannot get our house in order is evidence that we can’t safely build safe AGI yet. This is certainly magnitudes easier of a task, not to mention has significant reward (selfishly, it highly benefits us too!). Everyone feels that something is off but no one wants to do anything about it. We’ve only implemented half added measures that are more about signaling. Can’t let an LLM review for you? But the author is responsible for proving the reviewer used one? We’re all ML experts… we all know this isn’t possible except in the most cases. It’s as if you got shot while blindfolded and the police won’t investigate until you bring them evidence of who shot you and with what kind of gun. It shouldn’t matter if a review is bad because it was written by an LLM or because it was by a human. Just like it shouldn’t matter if you were shot or stabbed. > The maths are not important, the important thing is the insight you give to the readers. I also hate this take. The math often __is__ the insight. I agree that a lot of papers have needless math (look at any diffusion paper or any paper with attention copy pasting the same equations). But other works need them. The reason to use math is the same reason we program. If there was an easier way to communicate, we would (note: math isn’t just symbols, it can be words too). Math and programming are hard because they are languages that are precise and dense. The precision is important when communicating. Yes, it might take longer to parse but it is unambiguous when interpreted (it is also easier to parse when you’re trained and in the habit. Just like any other language). I think we lost our way in academia. We got caught up in by excitement. We let the bureaucrats take too much control and dictate the universities. We got lost in our egos (definitely not new) and too focused on prestige and fame. Our systems should be fighting these things, not enabling or encouraging them. Yes, the people at the top benefit from these systems, but the truth is that even they benefit from fixing things.