4 ms·

> This is leading to quite irresponsible reviews. Instead, authored and credited reviews might lead to more responsible reviews, or reviewers respectfully decli

> This is leading to quite irresponsible reviews. Instead, authored and credited reviews might lead to more responsible reviews, or reviewers respectfully declining when they might not know the topic. That misses the whole point of anonymous reviews, or even anonymous voting. You want your work to stand by itself. Otherwise you're blasting open a very corrupt door where appeals to authority, petty politics, careerism, and funding play a role in how a paper is approved. > Instead, in CS, there is a tendency to hide behind an abrasive negative review when the reality is that the reviewer does not understand the paper. Isn't the whole point of a paper to present a topic in a clear and understandable way, so that your peers are able to cut through the bullshit? If you pick a journal to publish your paper, which means you explicitly want the paper's editors to go through each and every single line of text you wrote to poke holes, but in the end once those holes are picked you complain that the journal you picked is not the right one for your flawless paper and that they all suck and their problem is that they don't understand your genius, what does this say about you and your work? When you pick a journal you pick the subset of your peers to review your work. If your peers point out problems then why not listen to them? Your comment reads a whole lot like "the fox and the grapes". https://en.wikipedia.org/wiki/The_Fox_and_the_Grapes https://en.wikipedia.org/wiki/The_Fox_and_the_Grapes

auggierose 4y ago
First, journals and conferences are different. I would expect more valuable feedback from journals than from conference reviews. Second, journals are overloaded, the journal reviewers unpaid, and frankly, they are not doing a good job anymore in reviewing work. Feedback that amounts to "maybe submit first to a conference" is not that valuable. Effectively, you have to have the idea, then execute it perfectly, then do a major marketing effort to popularise your work, and why exactly would you at the end of this still submit to a journal? Given that the journal either wants the rights to your work, or charges you for the pleasure of publishing with them. At this point, a journal is just another marketing step, and for marketing you pay, I guess.
CrazyStat 4y ago
> You want your work to stand by itself. Otherwise you're blasting open a very corrupt door where appeals to authority, petty politics, careerism, and funding play a role in how a paper is approved. This door is already open, and impossible to close [1]. For any reviewer closely familiar with the area the paper is in it is trivial to guess which group the paper came out of 9 times out of 10. Style, approach, citations, and just knowing what the other people in your area are working on all contribute. To achieve any sort of blindness you have to go to reviewers who don't know that area as well... which, I suppose, does decrease the potential for corruption--at the cost of the reviewers not understanding the paper as well. > Isn't the whole point of a paper to present a topic in a clear and understandable way, so that your peers are able to cut through the bullshit? Even a well written paper often requires some effort to understand, due to conceptual novelties, mathematical complexities, etc. Unfortunately there's little incentive for reviewers to put in much mental effort. Some of them will. Many won't. > When you pick a journal you pick the subset of your peers to review your work. If your peers point out problems then why not listen to them? Because the "problems" are often bullshit. You're assuming the reviewers are acting in good faith and finding legitimate problems with the paper, but they're often not--usually not out of malice, just laziness. There's no incentive to put any effort into a review above the minimum to convince the editor that your review is plausible. This doesn't take much since the editor is probably only going to glance briefly at the reviews. If you can finish your review in an hour instead of a day you can spend the rest of the day writing your own papers or doing other work that actually furthers your career. [1] In all but the largest, most active areas, where there are so many people working that nobody can keep up.
- bumby 4y ago
  It would be interesting to see if there is a distinction between review and acceptance statistics depending on timeframe. For example, in fields where the peer-review is done primarily by academics are there better reviews during academic breaks vs. say, exam season? Does this translate to more accepted work because the reviewer can spend more time digesting the content?
bumby 4y ago
>Isn't the whole point of a paper to present a topic in a clear and understandable way, so that your peers are able to cut through the bullshit? Yes, but there is unfortunately a wide variability in the knowledge of reviewers. It's not uncommon to have reviewers directly contradict each other about the shortcomings of the paper. E.g., Reviewer 1: "Section A needs to be condensed, this level of background is unnecessary. Readers will find this as common knowledge." Reviewer 2: "Section A should be expanded as all readers may not be knowledgeable in this area." Particularly for grad students, this can make the process feel arbitrary and the feedback of limited value. There also can be an inordinate amount of stylistic review comments (e.g., "however" should be changed to "nonetheless"). These often come across as the reviewer feels the need to critique something because they don't have sustentative critiques. Good critiques strengthen a paper, bad feedback seems to do nothing but check the process boxes. I actually agree with the OP that one potential solution is to encourage reviewers to decline reviewing papers that they don't have the requisite knowledge to provide good feedback. Unfortunately, this decision is often only offered based on reading the title and abstract.
- humanistbot 4y ago
  In some publications (mostly CS conferences), there are associate chairs or 'meta-reviewers' whose sole job is to manage the peer review for a paper. They will read the reviews and if there isn't consensus, they'll ask the reviewers to read the other reviews and have a discussion. It is usually 3 reviewers and 1 meta-reviewer per paper, so there are often 2-1 splits. The meta-reviewer doesn't just send the reviews to the authors with no commentary. They write a summary of the reviews with a recommendation about what to do next. If there are contradictions and no consensus is reached, it is their job to decide which concerns the authors need to respond to in order to get it published. They aren't bound by the reviews, they can disregard one if they feel like it is unfair, although sometimes the rules say they must get another reviewer to write a 4th review.

auggierose 4y ago

First, journals and conferences are different. I would expect more valuable feedback from journals than from conference reviews. Second, journals are overloaded, the journal reviewers unpaid, and frankly, they are not doing a good job anymore in reviewing work. Feedback that amounts to "maybe submit first to a conference" is not that valuable. Effectively, you have to have the idea, then execute it perfectly, then do a major marketing effort to popularise your work, and why exactly would you at the end of this still submit to a journal? Given that the journal either wants the rights to your work, or charges you for the pleasure of publishing with them. At this point, a journal is just another marketing step, and for marketing you pay, I guess.

CrazyStat 4y ago

> You want your work to stand by itself. Otherwise you're blasting open a very corrupt door where appeals to authority, petty politics, careerism, and funding play a role in how a paper is approved. This door is already open, and impossible to close [1]. For any reviewer closely familiar with the area the paper is in it is trivial to guess which group the paper came out of 9 times out of 10. Style, approach, citations, and just knowing what the other people in your area are working on all contribute. To achieve any sort of blindness you have to go to reviewers who don't know that area as well... which, I suppose, does decrease the potential for corruption--at the cost of the reviewers not understanding the paper as well. > Isn't the whole point of a paper to present a topic in a clear and understandable way, so that your peers are able to cut through the bullshit? Even a well written paper often requires some effort to understand, due to conceptual novelties, mathematical complexities, etc. Unfortunately there's little incentive for reviewers to put in much mental effort. Some of them will. Many won't. > When you pick a journal you pick the subset of your peers to review your work. If your peers point out problems then why not listen to them? Because the "problems" are often bullshit. You're assuming the reviewers are acting in good faith and finding legitimate problems with the paper, but they're often not--usually not out of malice, just laziness. There's no incentive to put any effort into a review above the minimum to convince the editor that your review is plausible. This doesn't take much since the editor is probably only going to glance briefly at the reviews. If you can finish your review in an hour instead of a day you can spend the rest of the day writing your own papers or doing other work that actually furthers your career. [1] In all but the largest, most active areas, where there are so many people working that nobody can keep up.

bumby 4y ago
It would be interesting to see if there is a distinction between review and acceptance statistics depending on timeframe. For example, in fields where the peer-review is done primarily by academics are there better reviews during academic breaks vs. say, exam season? Does this translate to more accepted work because the reviewer can spend more time digesting the content?

bumby 4y ago

>Isn't the whole point of a paper to present a topic in a clear and understandable way, so that your peers are able to cut through the bullshit? Yes, but there is unfortunately a wide variability in the knowledge of reviewers. It's not uncommon to have reviewers directly contradict each other about the shortcomings of the paper. E.g., Reviewer 1: "Section A needs to be condensed, this level of background is unnecessary. Readers will find this as common knowledge." Reviewer 2: "Section A should be expanded as all readers may not be knowledgeable in this area." Particularly for grad students, this can make the process feel arbitrary and the feedback of limited value. There also can be an inordinate amount of stylistic review comments (e.g., "however" should be changed to "nonetheless"). These often come across as the reviewer feels the need to critique something because they don't have sustentative critiques. Good critiques strengthen a paper, bad feedback seems to do nothing but check the process boxes. I actually agree with the OP that one potential solution is to encourage reviewers to decline reviewing papers that they don't have the requisite knowledge to provide good feedback. Unfortunately, this decision is often only offered based on reading the title and abstract.

humanistbot 4y ago
In some publications (mostly CS conferences), there are associate chairs or 'meta-reviewers' whose sole job is to manage the peer review for a paper. They will read the reviews and if there isn't consensus, they'll ask the reviewers to read the other reviews and have a discussion. It is usually 3 reviewers and 1 meta-reviewer per paper, so there are often 2-1 splits. The meta-reviewer doesn't just send the reviews to the authors with no commentary. They write a summary of the reviews with a recommendation about what to do next. If there are contradictions and no consensus is reached, it is their job to decide which concerns the authors need to respond to in order to get it published. They aren't bound by the reviews, they can disregard one if they feel like it is unfair, although sometimes the rules say they must get another reviewer to write a 4th review.