4 ms·

Here are my thoughts based on my experience with peer reviewed publication. There are two high level criteria for publication: novelty and difficulty (this is

Here are my thoughts based on my experience with peer reviewed publication. There are two high level criteria for publication: novelty and difficulty (this is in my field of Programming Languages and Systems so keep that in mind). The novelty requirement is important and I trust that you satisfied it but (as you pointed out in a child comment) you may not have met the difficulty requirement and the reviewer did their "best" to articulate that in a way that isn't the all together ridiculous "not hard enough". Naturally we might wonder why "difficulty" is a requirement at all. Shouldn't the importance or impact of the work be the thing that matters regardless of how difficult it was to achieve? The problem is that it's _extremely_ hard to know what work will be impactful and so reviewers, who have to reject something like 90% of submissions, use the heuristic of "difficulty" to estimate. This is a problem to be sure but I think it would be a problem in other settings as well.

kxyvr 10y ago
I wanted to echo this experience. I agree that reviewers are generally looking for novelty and difficulty. Unfortunately, difficulty can translate into a system where reviewers create small fiefdoms: This isn't difficult enough for my journal, so I'm going to reject it. I say fiefdoms because I don't often see criteria for what difficult means, so it's left to individual reviewers and the editor to decide. Anyway, this creates some perverse incentives where it's now necessary to make a piece of work appear more difficult than it really is. That said, I don't really think this is all that unique to academic publishing. At some point, I think we've all met someone professionally who makes problems sound more difficult than they really are and seen this person rewarded by management because they're the person who can work on really difficult problems.
tincholio 10y ago
I don't know about the details of the OP's paper, but I'm not sure "difficulty" is the right word... Mathematical rigor is needed in many domains. In my field, "not enough math" could well signify that the description of the experimental results is sloppy, or that you can't check whether the results presented are sound.
neel_k 10y ago
I'm more on the theory end of PL, but in my experience program committees generally regard difficulty as a negative for a paper. The main thing PCs want is for papers which make progress on important problems. Now, you can do this either by (1) attacking existing open problems, or (2) by finding new important problems. It's easy to tell that a long-standing open problem is important, but generally these problems are open because they are hard, and consequently the technical sophistication for new attacks on them is generally very high.[] New problems can often be solved with relatively simple techniques, but it's also difficult to tell if a new problem is genuinely important or just some random thing made up for submission. With a low acceptance rate, PCs tend to favor type (1) papers over type (2) papers, because few people feel comfortable arguing for a paper that might* be doing something important over one that is doing something important. This is widely seen as a failure mode of low acceptance-rate venues. So over the last several years many of the top PL conferences (such as POPL, LICS and PLDI) have moved to a multiple track format in order to increase the number of papers they accept. This is precisely to give program committees the freedom to take chances on "risky" papers that might become important, or might become footnotes. [*] Occasionally, someone will solve a difficult open problem with simple techniques, and these papers are immediately accepted with much rejoicing. It's easy to have your cake and eat it, when you have two cakes...
- johnbender 10y ago
  > in my experience program committees generally regard difficulty as a negative for a paper. I don't know if we're talking about the same type of difficulty. I don't think program committees see the difficulty of the problem being solved at the core of the paper as a "negative" necessarily? > The main thing PCs want is for papers which make progress on important problems. Now, you can do this either by (1) attacking existing open problems, or (2) by finding new important problems. My earlier note aside, I think this is an extremely important point. It seems to support the idea that "impact" is hard to gauge but much easier when the community has already arrived at some form of consensus on the importance of a problem.

kxyvr 10y ago

I wanted to echo this experience. I agree that reviewers are generally looking for novelty and difficulty. Unfortunately, difficulty can translate into a system where reviewers create small fiefdoms: This isn't difficult enough for my journal, so I'm going to reject it. I say fiefdoms because I don't often see criteria for what difficult means, so it's left to individual reviewers and the editor to decide. Anyway, this creates some perverse incentives where it's now necessary to make a piece of work appear more difficult than it really is. That said, I don't really think this is all that unique to academic publishing. At some point, I think we've all met someone professionally who makes problems sound more difficult than they really are and seen this person rewarded by management because they're the person who can work on really difficult problems.

tincholio 10y ago

I don't know about the details of the OP's paper, but I'm not sure "difficulty" is the right word... Mathematical rigor is needed in many domains. In my field, "not enough math" could well signify that the description of the experimental results is sloppy, or that you can't check whether the results presented are sound.

neel_k 10y ago

I'm more on the theory end of PL, but in my experience program committees generally regard difficulty as a negative for a paper. The main thing PCs want is for papers which make progress on important problems. Now, you can do this either by (1) attacking existing open problems, or (2) by finding new important problems. It's easy to tell that a long-standing open problem is important, but generally these problems are open because they are hard, and consequently the technical sophistication for new attacks on them is generally very high.[] New problems can often be solved with relatively simple techniques, but it's also difficult to tell if a new problem is genuinely important or just some random thing made up for submission. With a low acceptance rate, PCs tend to favor type (1) papers over type (2) papers, because few people feel comfortable arguing for a paper that might* be doing something important over one that is doing something important. This is widely seen as a failure mode of low acceptance-rate venues. So over the last several years many of the top PL conferences (such as POPL, LICS and PLDI) have moved to a multiple track format in order to increase the number of papers they accept. This is precisely to give program committees the freedom to take chances on "risky" papers that might become important, or might become footnotes. [*] Occasionally, someone will solve a difficult open problem with simple techniques, and these papers are immediately accepted with much rejoicing. It's easy to have your cake and eat it, when you have two cakes...

johnbender 10y ago
> in my experience program committees generally regard difficulty as a negative for a paper. I don't know if we're talking about the same type of difficulty. I don't think program committees see the difficulty of the problem being solved at the core of the paper as a "negative" necessarily? > The main thing PCs want is for papers which make progress on important problems. Now, you can do this either by (1) attacking existing open problems, or (2) by finding new important problems. My earlier note aside, I think this is an extremely important point. It seems to support the idea that "impact" is hard to gauge but much easier when the community has already arrived at some form of consensus on the importance of a problem.