10 ms·

I think the reviewers did a good job; the reviews are pretty reasonable. Reviews are supposed to be about the quality of a paper, not how influential they might

I think the reviewers did a good job; the reviews are pretty reasonable. Reviews are supposed to be about the quality of a paper, not how influential they might be in the future! And not all influential papers are actually very good.

lainga 3y ago
"The eight-legged essay was needed for those candidates in these civil service tests to show their merits for government service... structurally and stylistically, the eight-legged essay was restrictive and rigid. There are rules governing different sections of the essay, including restrictions on the number of sentences in total, the number of words in total, the format and structure of the essay, and rhyming techniques." https://en.wikipedia.org/wiki/Eight-legged_essay#Viewpoints https://en.wikipedia.org/wiki/Eight-legged_essay#Viewpoints
- nybsjytm 3y ago
  I guess your comment is against the restrictive and rigid idea that peer review should be about making research papers more intellectually rigorous?
- hnfong 3y ago
  That's the medieval equivalent of leetcode. The problems that the imperial Chinese government had to solve was pretty much the same as the problem the Big Tech companies are trying to solve with leetcode. In earlier times, it used to be that the exams were more freestyle, but when tens/hundreds of thousands of people compete for a handful of high civil service positions, people are motivated to cheat by memorizing essays pre-written by somebody else. And the open-ended questions had subjective answers that didn't scale. So they basically gamified the whole thing.
  - fl7305 3y ago
    They might not be perfect employees, but at least you know they are smart, are disciplined, and have the capacity to work hard for a long period.
    - lainga 3y ago
      And won't discuss salaries with each other :) https://en.wikipedia.org/wiki/Song_official_headwear https://en.wikipedia.org/wiki/Song_official_headwear
    - wahnfrieden 3y ago
      Sounds like a hazing ritual’s outcome
      - sgift 3y ago
        Exactly what it is. "We had to go through it, so they'll have to too!" plus "Someone who has so little self-respect that they do this will do anything we ask of them." (I know that some people genuinely like Leetcode and that's totally fine. But that's not why company want people to do it)
        lazide 3y ago
        If you think that’s bad, wait until you hear about medschool / nursing.
        smcin 3y ago
        Standardized interviews or panels do not necessarily exist to find the best candidate. They exist as a tradeoff to ensure some measure of objectivity and prevent favoritism/influence/corruption/bribery/forgery/impersonation/unfair cheating by getting advance access to the test material; in such a way that this can then be verified, standardized, audited at higher levels or nationally. Even more important for medschool/nursing than engineering. One of countless examples was the sale of 7600 fake nursing transcripts and diplomas in 2020/1 by three south Florida nursing schools [0]. (This happened in the middle of Covid, and at schools which were already being deaccredited.) Buyers paid $10-15K to obtain fake diplomas and transcripts indicating that they had earned legitimate degrees, like a (two-year) associate degree in nursing; these credentials then allowed the buyers to qualify for the national nursing board exam (NCLEX). About 37% of those who bought the fake documents — or about 2,800 people — passed the exam. (Compare to candidates holding a bachelor's degree in nursing (BSN) reportedly typically pass at 90% compared to 84% for those with an associate degree in nursing (ADN)). Among that 2700, a “significant number” then received nursing licenses and secured jobs in unnamed hospitals and other health care settings in MD, NY, NJ, GA. [0]: "7,600 Fake Nursing Diplomas Were Sold in Scheme, U.S. Says" https://web.archive.org/web/20230928151334/https://www.nytimes.com/2023/01/27/us/florida-nursing-diploma-scam.html https://web.archive.org/web/20230928151334/https://www.nytim...
  - godelski 3y ago
    And it's important to recognize the advantages and disadvantages to ensure that we have proper context. For example, leetcode may be very appropriate for those programming jobs which are fairly standard. At every job you don't need to invent new things. Industrialization was amazing because of this standardization and ability to mass produce (in a way, potentially LLMs can be this for code. Not quite there yet but it seems like a reasonable potential). But on the other hand, there are plenty of jobs where there are high levels of uniqueness and creativity and innovation dominate the skills of repetition and regurgitation. This is even true in research and science, though I think creativity is exceptionally important. The truth is that you need both. Often we actually need more of the former than the latter, but both are needed. They have different jobs. The question is more about the distribution of these skillsets that you need to accomplish your goals. Too much rigidity is stifling and too much flexibility is chaos. But I'd argue that in the centuries we've better learned to wade through chaos and this is one of the unique qualities that makes us human. To embrace the unknown while everything in us fights to find answers, even if they are not truth; because it is better to be ruled by a malicious but rational god than the existential chaos.
    - sevagh 3y ago
      >But on the other hand, there are plenty of jobs where there are high levels of uniqueness and creativity and innovation dominate the skills of repetition and regurgitation. This is even true in research and science, though I think creativity is exceptionally important. Those companies still use leetcode for those positions. It's just a blanket thing at this point.
      - godelski 3y ago
        Yes, and I think it is dumb. I'm personally fed up with how much we as a society rely on metrics for the sake of metrics. I can accept that things are difficult to measure and that there's a lot of chaos. Imperfection is perfectly okay. But I have a hard time accepting willful ignorance, acting like it is objective. I'm sure I am willfully ignorant many times too, but I think my ego should take the hit rather than continue.
gms7777 3y ago
I agree. My own most influential paper received strong rejects the first time we submitted it, and rightfully so, I think. In retrospect, we didn't do a good job motivating it, the contributions weren't clearly presented, and the way we described was super confusing. I'm genuinely grateful for it because the paper that we eventually published is so much better (although the core of the idea barely changed), and it's good because of the harsh reviews we received the first time around. The reviews themselves weren't even particularly "insightful", mostly along the lines of "this is confusing, I don't understand what you're doing or why you're doing it", but sometimes you just really need that outside perspective. I've also reviewed and rejected my share of papers where I could tell there is a seed of a great idea, but the paper as written just isn't good. It always brings me joy to see those papers eventually published because they're usually so much better.
- KittenInABox 3y ago
  > The reviews themselves weren't even particularly "insightful", mostly along the lines of "this is confusing, I don't understand what you're doing or why you're doing it", but sometimes you just really need that outside perspective. IMO maybe scientists should have experience critiquing stuff like poems, short essays, or fiction. Expecting a critiquer to give actually good suggestions matching your original vision, when your original vision's presentation is flawed, is incredibly rare. So the best critiques are usually a "this section right here, wtf is it?" style, with added bonus points to "wtf is this order of information" or other literary technique that is either being misused or unused.
  - gms7777 3y ago
    Oh, I do completely agree and didn't mean to imply otherwise. I have had experiences where reviewers have given me great ideas for new experiments or ways to present things. But the most useful ones usually are the "wtf?" type comments, or comments that suggest the reviewers completely misunderstood or misread the text. While those are initially infuriating, the reviewers are usually among the people in the field that are most familiar with the topic of the paper--if they don't understand it or misread it, 95% of the time it's because it could be written more clearly.
johnfn 3y ago
Don't you think something is missing if we've defined "quality" as an independent and uncorrelated characteristic from importance or influentiality?
- SubiculumCode 3y ago
  When an author refuses to address reasonable questions by the reviewers, what should you expect? There were legitimate questions and concerns about potential alternative explanations for the increase in accuracy raised by the reviewers, and the authors didn't play ball.
- nicklecompte 3y ago
  No, because "quality" means two different things here. I believe the main reason word2vec became important was purely on the software/engineering side, not because it was scientifically novel. Advancements in Python development, especially good higher-level constructs around numerical linear algebra, meant that the fairly shallow and simple tools of word2vec were available to almost every tech company. So I don't think word2vec was (or is) particularly good research, but it became good software for reasons beyond its own control In 2016 or so it was proven that word2vec is equivalent to the pointwise mutual information between the words in its training set, after doing some preprocessing. This means that Claude Shannon had things mostly figured out in the 60s, and some reviewers were quite critical of the word2vec paper for not citing similar developments in the 70s.
- nybsjytm 3y ago
  Quality in the sense I meant it (cogency and intellectual depth/rigor) should certainly be correlated with importance and influence!
- godelski 3y ago
  Yes and no. I think the larger issue is about the ambiguity of what publications mean and should be. There's a lot of ways to optimize this, and none of those has optimal solutions. I don't think you should be down voted for your different framing because I think we just need to be more open about this chaos and consider other values than our own or the road we're on. I think it is very unclear what we are trying to optimize and it is quite apparent that you're going to have many opinions on this and your group of reviewers may all seek to optimize different things. The only solution I can see is to stop pretending as if we know what each other is trying to do and be a bit more explicit about it. Because if we argue based on different assumptions we'll talk past one another if we assume the other is working on the same set of assumptions.
SubiculumCode 3y ago
This is the right take, despite how some might will want to frame it as 'reviewers are dummies'.
godelski 3y ago
> I think the reviewers did a good job I actually disagree, but maybe not for the reasons you're expecting. I actually disagree because the reviews are unreasonably reasonable. They are void of context. It's tough to explain, but I think it's also something every person who has written research papers can understand. How there's a big bias between reading and writing and how our works are not written to communicate our works as best as possible, but in effect how to communicate to reviewers that they should accept our works. The subtle distinction is deceptively large and I think we all could be a bit more honest about the system. After all, we want to optimize it, right? The problem I see is all a matter of context. Good ideas often appear trivial once we see them. Often we fool ourselves into thinking that we already knew this, but do not have meaningful evidence that this is true but may try to reason that x = y + z + epsilon, but almost any idea can be framed that way, even breakthroughs like Evolution, Quantum Mechanics, or Relativity. It is because we look back at giants from a distance but when looking at works now don't see giants, but a bunch of children standing on one another's shoulders standing in a trench coat. That is the reality of it all. That few works are giant leaps and bounds, but rather incremental. The ones that take the biggest leaps are rare, often rely on luck (ambiguous definition), and frequently (but not always, especially considering the former) take a lot of time. Something we certainly don't often have. We're trained as scientists and that means to be trained in critiquing systems and letting questions spiral. Sometimes the spiraling of questions shows how absurd an idea is but other times it can show how ingenious it is. It's easier to recognize the former but often hard to distinguish the latter. It is always easy to ask for more datasets, more experiments, and such, but these are critiques that can apply to any work as no work is complete. This is especially true in cases of larger breakthroughs, because any paradigm shift (even small) will cause a cascade of questions and create a lot of curiosity. But as we've all written papers, we know that this can often be a never ending cycle and often is quite impractical. The complaint about Table 4 is a perfect example. It is quite a difficult situation. The complaint is perfectly reasonable in that the question and concerns are quite valid and do warrant further understanding. But at the same time they are unreasonable because the requisite work required to answer these is not appropriate for the timescale that we work on. Do you have the compute or time to retrain all prior works to your settings? To retrain all your works to their settings? Maybe it doesn't work there which may or may not be evidence that the other works are just as good or not. What it comes down to is asking if these questions being answered could be another work in their own right. I'm not in NLP as deep as I'm in CV, but I suspect that the answer is yes (as in there are works that have been published answering exactly those questions). There are also reasonably unreasonable questions in other respects. Such as the question about cosine distance vs Euclidean. This is one that I see quite often as we rely too deeply on our understanding of lower dimensional geometries to influence our understanding of high dimensional geometries. Such things that seem obvious, like distance, are quite ambiguous there and our example is the specific reason for the curse of dimensionality (it becomes difficult to distinguish the furthest point from the nearest point). But this often leads us in quite the wrong direction. Honestly, it is a bit surprising that the cosine similarity works (as D->inf cos(x,y)-> 0 forall x,y in R^D because any random vector is expected to be orthogonal meaning that to get cos(x,y)=1 means y = x + epsilon with epsilon -> 0 as D->inf. But I digress), but it does. There definitely could be entire works exploring these geometries and determining different geodesics. It is entirely enough for a work to simply have something working, even if it doesn't quite yet make sense. The thing is that science is exceptionally fuzzy. Research is people walking around in the dark and papers are them communicating what we have found. I think it is important for us to remember this framing because we should then characterize the viability/publishability of a work not as illuminating everything but if the things found are useful (which itself is not always known). Because you might uncover a cavern and then it becomes easy to say "report back when you've explored it", but such an effort may be impossible to do alone. It can be a dead end, one that can take decades to explore (we'll always learn something though) or it may lead to riches we've never seen before. We don't know, but that's really what we're all looking for (hopefully more about riches for humanity than one's self, but one should be rewarded for sure). This is why honestly, I advocate for abandoning the journal/conference system and leverage our modern tools like OpenReview to accelerate communication. Because it enables us to be more open about our limitations, to discuss our failures, and write to our peers rather than our critics. Critics are of course important, but they can take over too easily because they are reasonable and correct, but oft missing context. For an example, see the many HN comments about a technology in its infancy where people will complain that it is not yet competitive with existing technologies and thus devalue the potential. Oft missing the context that it takes time to compete, the time and energy put in by so many before us to make what we have now, but also that there are limits and despite only being a demonstration the new thing does not have the same theoretical limits. The question is rather about if such research warrants more eyes and even a negative result can be good because it can communicate that we've found dead ends (which is something we actively discourage, needlessly forcing many researchers to re-explore these dead spaces). There's so much more that I can say and this is woefully incomplete but I can't fit a novel into our comments and I'm afraid the length as it is already results in poor communication to the given context of the platform. Thanks anyone who has taken this time.
geysersam 3y ago
But if that's the case why put so much focus on and effort into the peer review system? If you ask people funding research I'm pretty sure they'd prefer to fund influential ideas than non-influential "high-quality" paper production.
- nybsjytm 3y ago
  Even if you were to take the extreme position that influence or citation counts are all that matter, the problem is that 'future influence' is hard if not impossible to judge well in the present. (Of course it's easy to point to cases where it's possible to make an educated or confident guess.) Also, an emphasis on clarity and intellectual depth/rigor is important for the healthy development of a field. Not for nothing, the lack of this emphasis is a pretty common criticism of the whole AI field!
- Baader-Meinhof 3y ago
  High-quality writing improves information dissemination. A paper like word2vec has probably been skimmed by 10's of thousands, perhaps 100's of thousands people. One extra day of revising is nothing, comparatively.
fanzhang 3y ago
Agree that this is how papers are often judged, but strong disagree on how this is how papers should be judged. This is exactly the problem of reviewers looking for the keys under the lamp post (does the paper check these boxes), versus where they lost the keys (should this paper get more exposure because it advances the field). The fact that the first doesn't lead more to the second is a failure of the system. This is the same sort of value system that leads to accepting job candidates with neat haircuts and says the right shibboleths, versus the ones that make the right bottom line impact. Basically, are "good" papers that are very rigorous but lead to nothing actually "good"? If your model of progress in science is that rigorous papers are a higher probability roll of the dice, and nonrigorous papers are low probability rolls of the dice, then we should just look for rigorous papers. And that a low-rigor paper word2vec actually make progress was "getting really lucky" and we should have not rated the paper well. But I contend that word2vec was also very innovative, and that should be a positive factor for reviewers. In fact, I bet that innovative papers have a hard time being super rigorous because the definition of rigor in that field has yet to be settled yet. I'm basically contending that on the extreme margins, rigor is negatively correlated with innovation.
- nybsjytm 3y ago
  I don't consider clearly stating your model and meaningfully comparing it to prior work and other models (seemingly the main issues here) to be analogous to a proper haircut or a shibboleth. Actually I think it's a strange comparison to make.
- aaronkaplan 3y ago
  Your argument is that if a paper makes a valuable contribution then it should be accepted even if it's not well written. But the definition of "well written" is that it makes it easy for the reader to understand its value. If a paper is not well written, then reviewers won't understand its value and will reject it.
  - seanmcdirmid 3y ago
    Well written and rigor aren’t highly correlated. You can have poorly written papers that are very rigorous, and vic versa. Rigor is often another checkbox (does the paper have some quantitative comparisons), especially if the proper rigor is hard to define by the writer or the reader. My advice to PhD students is to always just focus on subjects where the rigor is straightforward, since that makes writing papers that get in easier. But of course, that is a selfish personal optimization that isn’t really what’s good for society.

lainga 3y ago

"The eight-legged essay was needed for those candidates in these civil service tests to show their merits for government service... structurally and stylistically, the eight-legged essay was restrictive and rigid. There are rules governing different sections of the essay, including restrictions on the number of sentences in total, the number of words in total, the format and structure of the essay, and rhyming techniques." https://en.wikipedia.org/wiki/Eight-legged_essay#Viewpoints https://en.wikipedia.org/wiki/Eight-legged_essay#Viewpoints

nybsjytm 3y ago
I guess your comment is against the restrictive and rigid idea that peer review should be about making research papers more intellectually rigorous?
hnfong 3y ago
That's the medieval equivalent of leetcode. The problems that the imperial Chinese government had to solve was pretty much the same as the problem the Big Tech companies are trying to solve with leetcode. In earlier times, it used to be that the exams were more freestyle, but when tens/hundreds of thousands of people compete for a handful of high civil service positions, people are motivated to cheat by memorizing essays pre-written by somebody else. And the open-ended questions had subjective answers that didn't scale. So they basically gamified the whole thing.
- fl7305 3y ago
  They might not be perfect employees, but at least you know they are smart, are disciplined, and have the capacity to work hard for a long period.
  - lainga 3y ago
    And won't discuss salaries with each other :) https://en.wikipedia.org/wiki/Song_official_headwear https://en.wikipedia.org/wiki/Song_official_headwear
  - wahnfrieden 3y ago
    Sounds like a hazing ritual’s outcome
    - sgift 3y ago
      Exactly what it is. "We had to go through it, so they'll have to too!" plus "Someone who has so little self-respect that they do this will do anything we ask of them." (I know that some people genuinely like Leetcode and that's totally fine. But that's not why company want people to do it)
      - lazide 3y ago
        If you think that’s bad, wait until you hear about medschool / nursing.
        smcin 3y ago
        Standardized interviews or panels do not necessarily exist to find the best candidate. They exist as a tradeoff to ensure some measure of objectivity and prevent favoritism/influence/corruption/bribery/forgery/impersonation/unfair cheating by getting advance access to the test material; in such a way that this can then be verified, standardized, audited at higher levels or nationally. Even more important for medschool/nursing than engineering. One of countless examples was the sale of 7600 fake nursing transcripts and diplomas in 2020/1 by three south Florida nursing schools [0]. (This happened in the middle of Covid, and at schools which were already being deaccredited.) Buyers paid $10-15K to obtain fake diplomas and transcripts indicating that they had earned legitimate degrees, like a (two-year) associate degree in nursing; these credentials then allowed the buyers to qualify for the national nursing board exam (NCLEX). About 37% of those who bought the fake documents — or about 2,800 people — passed the exam. (Compare to candidates holding a bachelor's degree in nursing (BSN) reportedly typically pass at 90% compared to 84% for those with an associate degree in nursing (ADN)). Among that 2700, a “significant number” then received nursing licenses and secured jobs in unnamed hospitals and other health care settings in MD, NY, NJ, GA. [0]: "7,600 Fake Nursing Diplomas Were Sold in Scheme, U.S. Says" https://web.archive.org/web/20230928151334/https://www.nytimes.com/2023/01/27/us/florida-nursing-diploma-scam.html https://web.archive.org/web/20230928151334/https://www.nytim...
- godelski 3y ago
  And it's important to recognize the advantages and disadvantages to ensure that we have proper context. For example, leetcode may be very appropriate for those programming jobs which are fairly standard. At every job you don't need to invent new things. Industrialization was amazing because of this standardization and ability to mass produce (in a way, potentially LLMs can be this for code. Not quite there yet but it seems like a reasonable potential). But on the other hand, there are plenty of jobs where there are high levels of uniqueness and creativity and innovation dominate the skills of repetition and regurgitation. This is even true in research and science, though I think creativity is exceptionally important. The truth is that you need both. Often we actually need more of the former than the latter, but both are needed. They have different jobs. The question is more about the distribution of these skillsets that you need to accomplish your goals. Too much rigidity is stifling and too much flexibility is chaos. But I'd argue that in the centuries we've better learned to wade through chaos and this is one of the unique qualities that makes us human. To embrace the unknown while everything in us fights to find answers, even if they are not truth; because it is better to be ruled by a malicious but rational god than the existential chaos.
  - sevagh 3y ago
    >But on the other hand, there are plenty of jobs where there are high levels of uniqueness and creativity and innovation dominate the skills of repetition and regurgitation. This is even true in research and science, though I think creativity is exceptionally important. Those companies still use leetcode for those positions. It's just a blanket thing at this point.
    - godelski 3y ago
      Yes, and I think it is dumb. I'm personally fed up with how much we as a society rely on metrics for the sake of metrics. I can accept that things are difficult to measure and that there's a lot of chaos. Imperfection is perfectly okay. But I have a hard time accepting willful ignorance, acting like it is objective. I'm sure I am willfully ignorant many times too, but I think my ego should take the hit rather than continue.

gms7777 3y ago

I agree. My own most influential paper received strong rejects the first time we submitted it, and rightfully so, I think. In retrospect, we didn't do a good job motivating it, the contributions weren't clearly presented, and the way we described was super confusing. I'm genuinely grateful for it because the paper that we eventually published is so much better (although the core of the idea barely changed), and it's good because of the harsh reviews we received the first time around. The reviews themselves weren't even particularly "insightful", mostly along the lines of "this is confusing, I don't understand what you're doing or why you're doing it", but sometimes you just really need that outside perspective. I've also reviewed and rejected my share of papers where I could tell there is a seed of a great idea, but the paper as written just isn't good. It always brings me joy to see those papers eventually published because they're usually so much better.

KittenInABox 3y ago
> The reviews themselves weren't even particularly "insightful", mostly along the lines of "this is confusing, I don't understand what you're doing or why you're doing it", but sometimes you just really need that outside perspective. IMO maybe scientists should have experience critiquing stuff like poems, short essays, or fiction. Expecting a critiquer to give actually good suggestions matching your original vision, when your original vision's presentation is flawed, is incredibly rare. So the best critiques are usually a "this section right here, wtf is it?" style, with added bonus points to "wtf is this order of information" or other literary technique that is either being misused or unused.
- gms7777 3y ago
  Oh, I do completely agree and didn't mean to imply otherwise. I have had experiences where reviewers have given me great ideas for new experiments or ways to present things. But the most useful ones usually are the "wtf?" type comments, or comments that suggest the reviewers completely misunderstood or misread the text. While those are initially infuriating, the reviewers are usually among the people in the field that are most familiar with the topic of the paper--if they don't understand it or misread it, 95% of the time it's because it could be written more clearly.

johnfn 3y ago

Don't you think something is missing if we've defined "quality" as an independent and uncorrelated characteristic from importance or influentiality?

SubiculumCode 3y ago
When an author refuses to address reasonable questions by the reviewers, what should you expect? There were legitimate questions and concerns about potential alternative explanations for the increase in accuracy raised by the reviewers, and the authors didn't play ball.
nicklecompte 3y ago
No, because "quality" means two different things here. I believe the main reason word2vec became important was purely on the software/engineering side, not because it was scientifically novel. Advancements in Python development, especially good higher-level constructs around numerical linear algebra, meant that the fairly shallow and simple tools of word2vec were available to almost every tech company. So I don't think word2vec was (or is) particularly good research, but it became good software for reasons beyond its own control In 2016 or so it was proven that word2vec is equivalent to the pointwise mutual information between the words in its training set, after doing some preprocessing. This means that Claude Shannon had things mostly figured out in the 60s, and some reviewers were quite critical of the word2vec paper for not citing similar developments in the 70s.
nybsjytm 3y ago
Quality in the sense I meant it (cogency and intellectual depth/rigor) should certainly be correlated with importance and influence!
godelski 3y ago
Yes and no. I think the larger issue is about the ambiguity of what publications mean and should be. There's a lot of ways to optimize this, and none of those has optimal solutions. I don't think you should be down voted for your different framing because I think we just need to be more open about this chaos and consider other values than our own or the road we're on. I think it is very unclear what we are trying to optimize and it is quite apparent that you're going to have many opinions on this and your group of reviewers may all seek to optimize different things. The only solution I can see is to stop pretending as if we know what each other is trying to do and be a bit more explicit about it. Because if we argue based on different assumptions we'll talk past one another if we assume the other is working on the same set of assumptions.

SubiculumCode 3y ago

This is the right take, despite how some might will want to frame it as 'reviewers are dummies'.

godelski 3y ago

> I think the reviewers did a good job I actually disagree, but maybe not for the reasons you're expecting. I actually disagree because the reviews are unreasonably reasonable. They are void of context. It's tough to explain, but I think it's also something every person who has written research papers can understand. How there's a big bias between reading and writing and how our works are not written to communicate our works as best as possible, but in effect how to communicate to reviewers that they should accept our works. The subtle distinction is deceptively large and I think we all could be a bit more honest about the system. After all, we want to optimize it, right? The problem I see is all a matter of context. Good ideas often appear trivial once we see them. Often we fool ourselves into thinking that we already knew this, but do not have meaningful evidence that this is true but may try to reason that x = y + z + epsilon, but almost any idea can be framed that way, even breakthroughs like Evolution, Quantum Mechanics, or Relativity. It is because we look back at giants from a distance but when looking at works now don't see giants, but a bunch of children standing on one another's shoulders standing in a trench coat. That is the reality of it all. That few works are giant leaps and bounds, but rather incremental. The ones that take the biggest leaps are rare, often rely on luck (ambiguous definition), and frequently (but not always, especially considering the former) take a lot of time. Something we certainly don't often have. We're trained as scientists and that means to be trained in critiquing systems and letting questions spiral. Sometimes the spiraling of questions shows how absurd an idea is but other times it can show how ingenious it is. It's easier to recognize the former but often hard to distinguish the latter. It is always easy to ask for more datasets, more experiments, and such, but these are critiques that can apply to any work as no work is complete. This is especially true in cases of larger breakthroughs, because any paradigm shift (even small) will cause a cascade of questions and create a lot of curiosity. But as we've all written papers, we know that this can often be a never ending cycle and often is quite impractical. The complaint about Table 4 is a perfect example. It is quite a difficult situation. The complaint is perfectly reasonable in that the question and concerns are quite valid and do warrant further understanding. But at the same time they are unreasonable because the requisite work required to answer these is not appropriate for the timescale that we work on. Do you have the compute or time to retrain all prior works to your settings? To retrain all your works to their settings? Maybe it doesn't work there which may or may not be evidence that the other works are just as good or not. What it comes down to is asking if these questions being answered could be another work in their own right. I'm not in NLP as deep as I'm in CV, but I suspect that the answer is yes (as in there are works that have been published answering exactly those questions). There are also reasonably unreasonable questions in other respects. Such as the question about cosine distance vs Euclidean. This is one that I see quite often as we rely too deeply on our understanding of lower dimensional geometries to influence our understanding of high dimensional geometries. Such things that seem obvious, like distance, are quite ambiguous there and our example is the specific reason for the curse of dimensionality (it becomes difficult to distinguish the furthest point from the nearest point). But this often leads us in quite the wrong direction. Honestly, it is a bit surprising that the cosine similarity works (as D->inf cos(x,y)-> 0 forall x,y in R^D because any random vector is expected to be orthogonal meaning that to get cos(x,y)=1 means y = x + epsilon with epsilon -> 0 as D->inf. But I digress), but it does. There definitely could be entire works exploring these geometries and determining different geodesics. It is entirely enough for a work to simply have something working, even if it doesn't quite yet make sense. The thing is that science is exceptionally fuzzy. Research is people walking around in the dark and papers are them communicating what we have found. I think it is important for us to remember this framing because we should then characterize the viability/publishability of a work not as illuminating everything but if the things found are useful (which itself is not always known). Because you might uncover a cavern and then it becomes easy to say "report back when you've explored it", but such an effort may be impossible to do alone. It can be a dead end, one that can take decades to explore (we'll always learn something though) or it may lead to riches we've never seen before. We don't know, but that's really what we're all looking for (hopefully more about riches for humanity than one's self, but one should be rewarded for sure). This is why honestly, I advocate for abandoning the journal/conference system and leverage our modern tools like OpenReview to accelerate communication. Because it enables us to be more open about our limitations, to discuss our failures, and write to our peers rather than our critics. Critics are of course important, but they can take over too easily because they are reasonable and correct, but oft missing context. For an example, see the many HN comments about a technology in its infancy where people will complain that it is not yet competitive with existing technologies and thus devalue the potential. Oft missing the context that it takes time to compete, the time and energy put in by so many before us to make what we have now, but also that there are limits and despite only being a demonstration the new thing does not have the same theoretical limits. The question is rather about if such research warrants more eyes and even a negative result can be good because it can communicate that we've found dead ends (which is something we actively discourage, needlessly forcing many researchers to re-explore these dead spaces). There's so much more that I can say and this is woefully incomplete but I can't fit a novel into our comments and I'm afraid the length as it is already results in poor communication to the given context of the platform. Thanks anyone who has taken this time.

geysersam 3y ago

But if that's the case why put so much focus on and effort into the peer review system? If you ask people funding research I'm pretty sure they'd prefer to fund influential ideas than non-influential "high-quality" paper production.

nybsjytm 3y ago
Even if you were to take the extreme position that influence or citation counts are all that matter, the problem is that 'future influence' is hard if not impossible to judge well in the present. (Of course it's easy to point to cases where it's possible to make an educated or confident guess.) Also, an emphasis on clarity and intellectual depth/rigor is important for the healthy development of a field. Not for nothing, the lack of this emphasis is a pretty common criticism of the whole AI field!
Baader-Meinhof 3y ago
High-quality writing improves information dissemination. A paper like word2vec has probably been skimmed by 10's of thousands, perhaps 100's of thousands people. One extra day of revising is nothing, comparatively.

fanzhang 3y ago

Agree that this is how papers are often judged, but strong disagree on how this is how papers should be judged. This is exactly the problem of reviewers looking for the keys under the lamp post (does the paper check these boxes), versus where they lost the keys (should this paper get more exposure because it advances the field). The fact that the first doesn't lead more to the second is a failure of the system. This is the same sort of value system that leads to accepting job candidates with neat haircuts and says the right shibboleths, versus the ones that make the right bottom line impact. Basically, are "good" papers that are very rigorous but lead to nothing actually "good"? If your model of progress in science is that rigorous papers are a higher probability roll of the dice, and nonrigorous papers are low probability rolls of the dice, then we should just look for rigorous papers. And that a low-rigor paper word2vec actually make progress was "getting really lucky" and we should have not rated the paper well. But I contend that word2vec was also very innovative, and that should be a positive factor for reviewers. In fact, I bet that innovative papers have a hard time being super rigorous because the definition of rigor in that field has yet to be settled yet. I'm basically contending that on the extreme margins, rigor is negatively correlated with innovation.

nybsjytm 3y ago
I don't consider clearly stating your model and meaningfully comparing it to prior work and other models (seemingly the main issues here) to be analogous to a proper haircut or a shibboleth. Actually I think it's a strange comparison to make.
aaronkaplan 3y ago
Your argument is that if a paper makes a valuable contribution then it should be accepted even if it's not well written. But the definition of "well written" is that it makes it easy for the reader to understand its value. If a paper is not well written, then reviewers won't understand its value and will reject it.
- seanmcdirmid 3y ago
  Well written and rigor aren’t highly correlated. You can have poorly written papers that are very rigorous, and vic versa. Rigor is often another checkbox (does the paper have some quantitative comparisons), especially if the proper rigor is hard to define by the writer or the reader. My advice to PhD students is to always just focus on subjects where the rigor is straightforward, since that makes writing papers that get in easier. But of course, that is a selfish personal optimization that isn’t really what’s good for society.