12 ms·

Word2Vec received 'strong reject' four times at ICLR2013

369 points by georgehill 3y ago

tinyhouse 3y ago
I agree that Glove was a fraud.
cs702 3y ago
Surely those seemingly smart anonymous reviewers now feel pretty dumb in hindsight. Peer review does not work for new ideas, because no one ever has the time or bandwidth to spend hours upon hours upon hours trying to understand new things.
- mempko 3y ago
  This is not the takeaway I got. The takeaway I got was the review process improved the paper and made it more rigorous. How is that a bad thing? But yes, sometimes reviewers are focusing on different issues instead of 'is this going to revolutionize A, B, and C'.
  - huijzer 3y ago
    I currently have a paper under review (first round) that was submitted the 2nd of August. This is at the second journal. The first submission was a few months before that. I'm not sure peer review makes things more rigorous, but it surely makes it more slow.
    - deleted 3y ago
      [deleted]
- IKantRead 3y ago
  It's worth pointing out that most of the best science happened before peer review was dominant. There's an article I came across awhile back, that I can't easily find now, that basically mapped out the history of our current peer review system. Peer review as we know it today was largely born in the 70s and a response to several funding crises in academia. Peer review was a strategy to make research appear more credible. The most damning critique of peer-review of course is that it completely failed to stop (and arguably aided) the reproducibility crisis. We have an academic system where the prime motivation is the secure funding through the image of credibility, which from first principles is a recipe for wide spread fraud.
  - HarHarVeryFunny 3y ago
    It seems kind of obvious that peer review is going to reward peer think, peer citation, and academic incremental advance. Obviously that's not how innovation works.
    - fatherzine 3y ago
      the system, as flawed as it is, is very effective for its purpose. see eg "success is 10% inspiration and 90% perspiration". on a darker side, the purpose is not to be fair to any particular individual, or even to be conducive to human flourishing at large.
      - HarHarVeryFunny 3y ago
        yes - maybe a good filter for future academic success, which seems to be a game unto itself
    - pas 3y ago
      academia is not about innovation, it should be trying to tend to the big self-referential kaleidoscope of knowledge. mostly it should try to do it through falsifying things, of course groupthing is seldom effective at that.
  - hnfong 3y ago
    Peer review is basically Github anonymous PRs that has the author pinky swear that the code compiles and 95% of test cases pass. Academic careers are then decided by the Github activity charts.
    - MichaelZuo 3y ago
      The whole 'pinky swear' aspect is far from ideal. But is there an alternative that still allows most academic aspirants to participate?
      - abrichr 3y ago
- sdenton4 3y ago
  I have been deeply unimpressed with the ML conference track this last year... There's too many papers, too few reviewers, leading to an insane number of PhD student-reviewers. We've gotten some real nonsense reviews, with some real sins against the spirit of science baked into them. For example, a reviewer essentially insisting that nothing is worth publishing if it doesn't include a new architecture idea and SOTA results... God forbid we better understand and simplify the tools that already exist!
- mrguyorama 3y ago
  Peer review isn't about the validity of your findings and the reviewers are not tasked with evaluating the findings of the researchers. The point is to be a light filter to make sure a published paper has the necessary information and rigor for someone else to try to replicate your experiment or build off of your findings. Those are the processes for evaluating the correctness of the findings.
- narrator 3y ago
  Do they do anything different in other countries, or is it just a copy of the U.S system?
- andreyk 3y ago
  I have finished a PhD in AI just this past year, and can assure you there exist reviewers who spend hours per review to do it well. It's true that these days it's often the case that you can (and are more likely than not to) get unlucky with lazier reviewers, but that does not appear to have been the case with this paper. For example just see this from the review of f5bf: "The main contribution of the paper comprises two new NLM architectures that facilitate training on massive data sets. The first model, CBOW, is essentially a standard feed-forward NLM without the intermediate projection layer (but with weight sharing + averaging before applying the non-linearity in the hidden layer). The second model, skip-gram, comprises a collection of simple feed-forward nets that predict the presence of a preceding or succeeding word from the current word. The models are trained on a massive Google News corpus, and tested on a semantic and syntactic question-answering task. The results of these experiments look promising. ... (2) The description of the models that are developed is very minimal, making it hard to determine how different they are from, e.g., the models presented in [15]. It would be very helpful if the authors included some graphical representations and/or more mathematical details of their models. Given that the authors still almost have one page left, and that they use a lot of space for the (frankly, somewhat superfluous) equations for the number of parameters of each model, this should not be a problem." These reviews in turn led to significant (though apparently not significant enough) modifications to the paper (https://openreview.net/forum?id=idpCdOWtqXd60&noteId=C8Vn84fqSG8qa https://openreview.net/forum?id=idpCdOWtqXd60&noteId=C8Vn84f...). These were some quality reviews and the paper benefited from going this review process, IMHO.
- canjobear 3y ago
  The issue here wasn't that the reviewers couldn't handle a new idea. They were all very familiar with word embeddings and ways to make them. There weren't a lot a of new concepts in word2vec, what distinguished it was that it was simple, fast, and good quality. The software and pretrained vectors were easy to access and use compared to existing methods.
magnio 3y ago
There are more details in the FB post of Tomas Mikolov (author of word2vec) recently: https://www.facebook.com/share/p/kXYaYaRvRCr5K2Ze https://www.facebook.com/share/p/kXYaYaRvRCr5K2Ze A hilarious and poignant point I see is how experts make mistake too. Quote: > I also received a lot of comments on the word analogies - from "I knew that too but forgot to publish it!" (Geoff Hinton, I believe you :) happens to everyone, and anyways I think everybody knows what the origin of Distributed Representations is) to "it's a total hack and I'm sure it doesn't work!" (random guys who didn't bother to read the papers and try it out themselves - including Ian Goodfellow raging about it on Twitter).
- nybsjytm 3y ago
  I tried asking on another thread what Goodfellow rage he's referring to since all I could find was this: https://twitter.com/goodfellow_ian/status/1133528189651677184 https://twitter.com/goodfellow_ian/status/113352818965167718... If so, frankly I think it makes Mikolov sound pretty insecure.
  - lern_too_spel 3y ago
    Twitter no longer shows threads to people who aren't logged in. https://nitter.net/goodfellow_ian/status/1133528189651677184 https://nitter.net/goodfellow_ian/status/1133528189651677184
  - deleted 3y ago
    [deleted]
- imjonse 3y ago
  That post sounds like a rant TBH with too many stabs at various people. It could have been a lot more graceful. OTOH I can believe most researchers are human and do not put the progress of shared knowledge first but are very much influenced by ego and money *cough* OpenAI *cough*
  - ReactiveJelly 3y ago
    To err is human, to seek profit is common to all lifeforms
lupusreal 3y ago
Boiled down to the core essence, science is about testing ideas to see if they work. Peer review is not part of this process, they rarely if ever attempt replication during the peer review process; and so they inevitably end up rejecting new ideas without even trying them. This isn't science.
- marginalia_nu 3y ago
  Thankfully you can just keep submitting the same paper to different journals until someone is too busy to read it and just approves it blindly. The academic publication shitshow giveth and the academic publication shitshow taketh away.
  - H8crilA 3y ago
    The futile quest for algorithmification of truth, and the loopholes that make the system work again despite having an impossible objective in the first place. Couldn't have been any different - in fact we can take this as evidence that AI has not taken over science yet.
nybsjytm 3y ago
I think the reviewers did a good job; the reviews are pretty reasonable. Reviews are supposed to be about the quality of a paper, not how influential they might be in the future! And not all influential papers are actually very good.
- lainga 3y ago
  "The eight-legged essay was needed for those candidates in these civil service tests to show their merits for government service... structurally and stylistically, the eight-legged essay was restrictive and rigid. There are rules governing different sections of the essay, including restrictions on the number of sentences in total, the number of words in total, the format and structure of the essay, and rhyming techniques." https://en.wikipedia.org/wiki/Eight-legged_essay#Viewpoints https://en.wikipedia.org/wiki/Eight-legged_essay#Viewpoints
  - nybsjytm 3y ago
    I guess your comment is against the restrictive and rigid idea that peer review should be about making research papers more intellectually rigorous?
  - hnfong 3y ago
    That's the medieval equivalent of leetcode. The problems that the imperial Chinese government had to solve was pretty much the same as the problem the Big Tech companies are trying to solve with leetcode. In earlier times, it used to be that the exams were more freestyle, but when tens/hundreds of thousands of people compete for a handful of high civil service positions, people are motivated to cheat by memorizing essays pre-written by somebody else. And the open-ended questions had subjective answers that didn't scale. So they basically gamified the whole thing.
    - fl7305 3y ago
      They might not be perfect employees, but at least you know they are smart, are disciplined, and have the capacity to work hard for a long period.
tbruckner 3y ago
Will keep happening because peer review itself, ironically, has no real feedback mechanism.
- ttpphd 3y ago
  This is exactly correct! It's an asymmetrical accountability mechanism.
- iceIX 3y ago
  The whole reason OpenReview was created was to innovate and improve on the peer review process. If you have ideas, reach out to the program chairs of the conference you're submitting to. Many of them are open to running experiments.
imjonse 3y ago
It seems they have rejected initial versions of the paper, since there had been later updates and clarifications based on the reviews. So it seems this was beneficial in the end and how review process should work? Especially since this was groundbreaking work it makes sense there is more effort put into explaining why it works instead of relying too much on good benchmark results.
Der_Einzige 3y ago
Makes me not feel bad about my own rejections when I see stuff like this or Yann Lecun reacting poorly on twitter to his own papers being rejected.
Hayvok 3y ago
The review thread (start at the bottom & work your way up) reads like a Show HN thread that went negative. The paper initially received some questions/negative feedback, so the authors updated and tweaked the reviewers a bit — > "We welcome discussion... The main contribution (that seems to have been missed by some of the reviews) is that we can use very shallow models to compute good vector representation of words." The response to the authors' update: > Review: The revision and rebuttal failed to address the issues raised by the reviewers. I do not think the paper should be accepted in its current form. > Quality rating: Strong reject > Confidence: Reviewer is knowledgeable
wzdd 3y ago
There are indeed four entries saying "strong reject", but they all appear to be from the same reviewer, at the same time, and saying the same thing. Isn't this just the one rejection? Also, why is only that reviewer's score visible?
pmags 3y ago
I'm curious how many commenters here who are making strong statements about the worth (or not) of peer review have actually participated in it both as author AND reviewer? Or even as an editor who is faced with the challenge of integrating and synthesizing multiple reviews into a recommendation? There are many venues available to share your research or ideas absent formal peer review, arXiv/bioRxiv being among the most popular. If you reject the idea of peer review itself it seems like there are plenty of alternatives.
- ska 3y ago
  It's the internet, therefore a significant percentage of the strong opinions about any topic will come from people who have little to no experience or competence in the area. Being HN, it probably skews a bit better that average. OTOH, it will also skew towards people procrastinating. Factor that in how you will...
mxwsn 3y ago
Flagged for misleading title - the four strong rejects are from a single author. It's listed four times for some unknown reason but likely an openreview quirk. The actual status described by the page is: 2 unknown (with accompanying long text), 1 weak reject, and 1 strong reject.
zaptheimpaler 3y ago
We already have a better mechanism for publishing and peer review.. it's called the internet. Literally the comments section of Reddit would work better. Reviews would be tied to a pseudonymous account instead of anonymous, allowing people to judge the quality of reviewers as well. Hacker News would work just as well too. It's also nearly free to setup a forum and frictionless to use compared to paying academic journals $100k for them to sell your own labour back to you. Cost and ease of use also mean more broadly accessible and hence more widely reviewed.
- vasco 3y ago
  Every once in a while I see a thread on reddit about a subject I know about and if someone shares a factual account that sounds unpopular it'll be downvoted even though it's true. I think reddit would be a terrible way to do this.
  - raverbashing 3y ago
    But academic review is like that, with the worse acktchsually guys in existance
    - ribosometronome 3y ago
      The worst? Are you sure? Reddit's worst acktchsually guys are often spilling out of actual cesspit hate subreddits.
      - raverbashing 3y ago
        There is some meta-knowledge in your comment, but I'm focusing solely on the critique and pedantry levels, no comment on other factors
  - 0xDEAFBEAD 3y ago
    https://www.lesswrong.com/ https://www.lesswrong.com/ lets you vote separately on agreement and quality axes. That seems to help a little bit.
raverbashing 3y ago
And this is why the biggest evolution of AI has happened in companies, not in academic circles Because there's too much nitpicking and grasping at straws amongst people that can't see novelty even when it's dancing in front of them
- layer8 3y ago
  No, the reason is that it required substantial financial investments, and in some cases access to proprietary big-data collections.
  - raverbashing 3y ago
    word2vec does not require a large amount of data mnist might have required a large amount of data at its creation, but it became a staple dataset There was a lot of evolution before ChatGPT
    - L3viathan 3y ago
      And people in academia were all over Word2Vec. Mikolov presented his work in our research group around 2014, and people were very excited. Granted, that was _after_ Word2Vec had been published, and this was a very pro-vectorspaces (although of a different type) crowd.
m3kw9 3y ago
That didn’t age well
picometer 3y ago
In hindsight, reviewer f5bf’s comment is fascinating: > - It would be interesting if the authors could say something about how these models deal with intransitive semantic similarities, e.g., with the similarities between 'river', 'bank', and 'bailout'. People like Tversky have advocated against the use of semantic-space models like NLMs because they cannot appropriately model intransitive similarities. What I’ve noticed in the latest models (GPT, image diffusion models, etc) is an ability to play with words when there’s a double meaning. This struck me as something that used to be very human, but is now in the toolbox of generative models. (Most of which, I assume, use something akin word2vec for deriving embedding vectors from prompts.) Is the word2vec ambiguity contributing to the wordplay ability? I don’t know, but it points to a “feature vs bug” situation where such an ambiguity is a feature for creative purposes, but a bug if you want to model semantic space as a strict vector space. My interpretation here is that the word/prompt embeddings in current models are so huge that they’re overloaded with redundant dimensions, such that it wouldn’t satisfy any mathematical formalism (eg of well-behaved vector spaces) at all.
- intalentive 3y ago
  Even small models (e.g. hidden dims = 32) should be able to handle token ambiguity with attention. The information is not so much in the token itself as in the context.
- PeterisP 3y ago
  The key difference is what I'd call "context-free embeddings" vs "contextual embeddings". Due to its structure, word2vec and similar solutions have to assign every single "bank" in every sentence the exact same vector, but later models (e.g. all the transformer models, BERT, GPT, etc) will assign wildly different vectors to "bank" depending on the context of surrounding words for that particular mention of "bank".
PaulHoule 3y ago
I'd reject it still (speaking of someone who has developed products based on word vectors, document vectors, dimensional reduction, etc. before y'all thought it was cool...) I quit a job because they were insisting on using Word2Vec in an application where it would have doomed the project to failure. The basic problem is that in a real-life application many of the most important words are not in the dictionary and if you throw out words that are not in the dictionary you choose to fail. Let a junk paper like that through and the real danger is that you will get 1000s of other junk papers following it up. For instance, take a look at the illustrations on this page https://nlp.stanford.edu/projects/glove/ https://nlp.stanford.edu/projects/glove/ particularly under "2. Linear Substructures". They make it look like a miracle that they project down from a 50-dimensional subspace down to 2 and get a nice pattern of cities and zip codes, for instance. The thing is you could have a random set of 20 points in a 50-d space and, assuming there are no degeneracy, you can map them to any 20 points you want in the 2-d space with an appropriately chosen projection matrix. Show me a graph like that with 200 points and I might be impressed. (I'd say those graphs on that server damage the Stanford brand for me about as much as SBF and Marc Tessier-Lavign) (It's a constant theme in dimensional reduction literature that people forget that random matrices often work pretty well, fail to consider how much gain they are getting over the random matrix, ...) BERT, FastText and the like were revolutionary for a few reasons, but I saw the use of subword tokens as absolutely critical because... for once, you could capture a medical note and not erase the patient's name! The various conventions of computer science literature prevented explorations that would have put Word2Vec in its place. For instance, it's an obvious idea that you should be able to make a classifier that, given a document vector, can predict "is this a color word?" or "is this a verb?" but if you actually try it, it doesn't work in a particularly maddening way. With a tiny training/eval set (say 10 words) you might convince yourself it is working, but the more data you train on the more you realize the words are scattered mostly randomly and even those those "linear structures" exist in a statistical sense they aren't well defined and not particularly useful. It's the kind of thing that is so weird and inconclusive and fuzzy that I'm not aware of anyone writing a paper about it... Cause you're not going to draw any conclusions out of it except that you found a Jupiter-sized hairball. For all the excitement people had over Word2Vec you didn't see an explosion of interest in vector search engines because... Word2Vec sucked, applying it to documents didn't improve the search engine very much. Some of it is that adding sensitivity to synonyms can hurt performance because many possible synonyms turn out to be red herrings. BERT, on the other hand, is context sensitive and is able to some extent know the different because "my pet jaguar" and "the jaguar dealership in your town" and that really does help find the relevant documents and hide the irrelevant documents.
funnystories 3y ago
when i was on college, i wrote a simple system to make corrections on text based on some heuristics for a class. then, the teacher of the class suggested me to write a paper describing the system for a local conference during the summer, with some results etc I wrote it with his support but it got rejected right away because of poor grammar or something similar. the conference was in Brazil, but required the papers to be in English. I was just a student and thought that indeed my english was pretty bad. the teacher said to me to at least send an email to the reviewers to get some feedback, maybe resubmit with the corrections. i asked specifically which paragraphs were confusing. they sent me some snippets of phrases that were obviously wrong. yes, they were the "before" examples of "before/after" my system applied the corrections. I tried to explain that the grammar should be wrong, but the just replied with "please fix your english mistakes and resubmit". i tried 2 or 3 more times but just gave up.
- adastra22 3y ago
  You remind me of these anecdotes by Feynman of his time in Brazil. Specifically search for "I was invited to give a talk at the Brazilian Academy of Sciences", but the whole thing is worth a read if you haven't seen it: https://southerncrossreview.org/81/feynman-brazil.html https://southerncrossreview.org/81/feynman-brazil.html
- wizzwizz4 3y ago
  *eyeroll* Sounds about right. Want to get that published anyway? You could pop it on the arXiv and let the HN hivemind suggest an appropriate venue. If you don't have arXiv access, find an endorser <https://info.arxiv.org/help/endorsement.html https://info.arxiv.org/help/endorsement.html>, and send them a SHORT polite email (prioritise brevity over politeness) with your paper and the details. Something like: > Hello, > I wrote a paper for college in yyyy (attached) on automatic grammar correction, which got rejected from Venue for grammatical errors in the figures. I still want to publish it. Could you endorse my arXiv account, please? > Also, could you suggest an appropriate venue to submit this work to? > Yours sincerely, > your name, etc Follow the guidance on the arXiv website when asking for endorsement.
  - 3y ago
nsagent 3y ago
Previous discussion: https://news.ycombinator.com/item?id=38654038 https://news.ycombinator.com/item?id=38654038
rahmaniacc 3y ago
This was hilarious! Many very broad and general statements are made without any citations to back them up. - Please be more specific. The number of self-citations seems somewhat excessive. - We added more citations.
ashvardanian 3y ago
That's the most inspiring thing I've learned this year.
jongjong 3y ago
I've found that PhD level academics are usually wrong about practical matters. It's almost as if having a PhD is itself proof that they are not good at identifying optimal solutions to reach practical goals. Also, it may show that they are overly concerned with superficial status markers instead of results. I think it's also the secret to why they can go so deep on certain abstract topics. Their minds are lacking a garbage collector and they can go in literally any direction and accumulate any amount of knowledge and they'll be able to memorize it all easily even if it has 0 utility value.

369 points by georgehill 3y ago

I agree that Glove was a fraud.

Surely those seemingly smart anonymous reviewers now feel pretty dumb in hindsight. Peer review does not work for new ideas, because no one ever has the time or bandwidth to spend hours upon hours upon hours trying to understand new things.

mempko 3y ago
This is not the takeaway I got. The takeaway I got was the review process improved the paper and made it more rigorous. How is that a bad thing? But yes, sometimes reviewers are focusing on different issues instead of 'is this going to revolutionize A, B, and C'.
- huijzer 3y ago
  I currently have a paper under review (first round) that was submitted the 2nd of August. This is at the second journal. The first submission was a few months before that. I'm not sure peer review makes things more rigorous, but it surely makes it more slow.
  - deleted 3y ago
    [deleted]
IKantRead 3y ago
It's worth pointing out that most of the best science happened before peer review was dominant. There's an article I came across awhile back, that I can't easily find now, that basically mapped out the history of our current peer review system. Peer review as we know it today was largely born in the 70s and a response to several funding crises in academia. Peer review was a strategy to make research appear more credible. The most damning critique of peer-review of course is that it completely failed to stop (and arguably aided) the reproducibility crisis. We have an academic system where the prime motivation is the secure funding through the image of credibility, which from first principles is a recipe for wide spread fraud.
- HarHarVeryFunny 3y ago
  It seems kind of obvious that peer review is going to reward peer think, peer citation, and academic incremental advance. Obviously that's not how innovation works.
  - fatherzine 3y ago
    the system, as flawed as it is, is very effective for its purpose. see eg "success is 10% inspiration and 90% perspiration". on a darker side, the purpose is not to be fair to any particular individual, or even to be conducive to human flourishing at large.
    - HarHarVeryFunny 3y ago
      yes - maybe a good filter for future academic success, which seems to be a game unto itself
  - pas 3y ago
    academia is not about innovation, it should be trying to tend to the big self-referential kaleidoscope of knowledge. mostly it should try to do it through falsifying things, of course groupthing is seldom effective at that.
- hnfong 3y ago
  Peer review is basically Github anonymous PRs that has the author pinky swear that the code compiles and 95% of test cases pass. Academic careers are then decided by the Github activity charts.
  - MichaelZuo 3y ago
    The whole 'pinky swear' aspect is far from ideal. But is there an alternative that still allows most academic aspirants to participate?
    - abrichr 3y ago
sdenton4 3y ago
I have been deeply unimpressed with the ML conference track this last year... There's too many papers, too few reviewers, leading to an insane number of PhD student-reviewers. We've gotten some real nonsense reviews, with some real sins against the spirit of science baked into them. For example, a reviewer essentially insisting that nothing is worth publishing if it doesn't include a new architecture idea and SOTA results... God forbid we better understand and simplify the tools that already exist!
mrguyorama 3y ago
Peer review isn't about the validity of your findings and the reviewers are not tasked with evaluating the findings of the researchers. The point is to be a light filter to make sure a published paper has the necessary information and rigor for someone else to try to replicate your experiment or build off of your findings. Those are the processes for evaluating the correctness of the findings.
narrator 3y ago
Do they do anything different in other countries, or is it just a copy of the U.S system?
andreyk 3y ago
I have finished a PhD in AI just this past year, and can assure you there exist reviewers who spend hours per review to do it well. It's true that these days it's often the case that you can (and are more likely than not to) get unlucky with lazier reviewers, but that does not appear to have been the case with this paper. For example just see this from the review of f5bf: "The main contribution of the paper comprises two new NLM architectures that facilitate training on massive data sets. The first model, CBOW, is essentially a standard feed-forward NLM without the intermediate projection layer (but with weight sharing + averaging before applying the non-linearity in the hidden layer). The second model, skip-gram, comprises a collection of simple feed-forward nets that predict the presence of a preceding or succeeding word from the current word. The models are trained on a massive Google News corpus, and tested on a semantic and syntactic question-answering task. The results of these experiments look promising. ... (2) The description of the models that are developed is very minimal, making it hard to determine how different they are from, e.g., the models presented in [15]. It would be very helpful if the authors included some graphical representations and/or more mathematical details of their models. Given that the authors still almost have one page left, and that they use a lot of space for the (frankly, somewhat superfluous) equations for the number of parameters of each model, this should not be a problem." These reviews in turn led to significant (though apparently not significant enough) modifications to the paper (https://openreview.net/forum?id=idpCdOWtqXd60&noteId=C8Vn84fqSG8qa https://openreview.net/forum?id=idpCdOWtqXd60&noteId=C8Vn84f...). These were some quality reviews and the paper benefited from going this review process, IMHO.
canjobear 3y ago
The issue here wasn't that the reviewers couldn't handle a new idea. They were all very familiar with word embeddings and ways to make them. There weren't a lot a of new concepts in word2vec, what distinguished it was that it was simple, fast, and good quality. The software and pretrained vectors were easy to access and use compared to existing methods.

magnio 3y ago

There are more details in the FB post of Tomas Mikolov (author of word2vec) recently: https://www.facebook.com/share/p/kXYaYaRvRCr5K2Ze https://www.facebook.com/share/p/kXYaYaRvRCr5K2Ze A hilarious and poignant point I see is how experts make mistake too. Quote: > I also received a lot of comments on the word analogies - from "I knew that too but forgot to publish it!" (Geoff Hinton, I believe you :) happens to everyone, and anyways I think everybody knows what the origin of Distributed Representations is) to "it's a total hack and I'm sure it doesn't work!" (random guys who didn't bother to read the papers and try it out themselves - including Ian Goodfellow raging about it on Twitter).

nybsjytm 3y ago
I tried asking on another thread what Goodfellow rage he's referring to since all I could find was this: https://twitter.com/goodfellow_ian/status/1133528189651677184 https://twitter.com/goodfellow_ian/status/113352818965167718... If so, frankly I think it makes Mikolov sound pretty insecure.
- lern_too_spel 3y ago
  Twitter no longer shows threads to people who aren't logged in. https://nitter.net/goodfellow_ian/status/1133528189651677184 https://nitter.net/goodfellow_ian/status/1133528189651677184
- deleted 3y ago
  [deleted]
imjonse 3y ago
That post sounds like a rant TBH with too many stabs at various people. It could have been a lot more graceful. OTOH I can believe most researchers are human and do not put the progress of shared knowledge first but are very much influenced by ego and money *cough* OpenAI *cough*
- ReactiveJelly 3y ago
  To err is human, to seek profit is common to all lifeforms

lupusreal 3y ago

Boiled down to the core essence, science is about testing ideas to see if they work. Peer review is not part of this process, they rarely if ever attempt replication during the peer review process; and so they inevitably end up rejecting new ideas without even trying them. This isn't science.

marginalia_nu 3y ago
Thankfully you can just keep submitting the same paper to different journals until someone is too busy to read it and just approves it blindly. The academic publication shitshow giveth and the academic publication shitshow taketh away.
- H8crilA 3y ago
  The futile quest for algorithmification of truth, and the loopholes that make the system work again despite having an impossible objective in the first place. Couldn't have been any different - in fact we can take this as evidence that AI has not taken over science yet.

nybsjytm 3y ago

I think the reviewers did a good job; the reviews are pretty reasonable. Reviews are supposed to be about the quality of a paper, not how influential they might be in the future! And not all influential papers are actually very good.

lainga 3y ago
"The eight-legged essay was needed for those candidates in these civil service tests to show their merits for government service... structurally and stylistically, the eight-legged essay was restrictive and rigid. There are rules governing different sections of the essay, including restrictions on the number of sentences in total, the number of words in total, the format and structure of the essay, and rhyming techniques." https://en.wikipedia.org/wiki/Eight-legged_essay#Viewpoints https://en.wikipedia.org/wiki/Eight-legged_essay#Viewpoints
- nybsjytm 3y ago
  I guess your comment is against the restrictive and rigid idea that peer review should be about making research papers more intellectually rigorous?
- hnfong 3y ago
  That's the medieval equivalent of leetcode. The problems that the imperial Chinese government had to solve was pretty much the same as the problem the Big Tech companies are trying to solve with leetcode. In earlier times, it used to be that the exams were more freestyle, but when tens/hundreds of thousands of people compete for a handful of high civil service positions, people are motivated to cheat by memorizing essays pre-written by somebody else. And the open-ended questions had subjective answers that didn't scale. So they basically gamified the whole thing.
  - fl7305 3y ago
    They might not be perfect employees, but at least you know they are smart, are disciplined, and have the capacity to work hard for a long period.

tbruckner 3y ago

Will keep happening because peer review itself, ironically, has no real feedback mechanism.

ttpphd 3y ago
This is exactly correct! It's an asymmetrical accountability mechanism.
iceIX 3y ago
The whole reason OpenReview was created was to innovate and improve on the peer review process. If you have ideas, reach out to the program chairs of the conference you're submitting to. Many of them are open to running experiments.

imjonse 3y ago

It seems they have rejected initial versions of the paper, since there had been later updates and clarifications based on the reviews. So it seems this was beneficial in the end and how review process should work? Especially since this was groundbreaking work it makes sense there is more effort put into explaining why it works instead of relying too much on good benchmark results.

Der_Einzige 3y ago

Makes me not feel bad about my own rejections when I see stuff like this or Yann Lecun reacting poorly on twitter to his own papers being rejected.

Hayvok 3y ago

The review thread (start at the bottom & work your way up) reads like a Show HN thread that went negative. The paper initially received some questions/negative feedback, so the authors updated and tweaked the reviewers a bit — > "We welcome discussion... The main contribution (that seems to have been missed by some of the reviews) is that we can use very shallow models to compute good vector representation of words." The response to the authors' update: > Review: The revision and rebuttal failed to address the issues raised by the reviewers. I do not think the paper should be accepted in its current form. > Quality rating: Strong reject > Confidence: Reviewer is knowledgeable

wzdd 3y ago

There are indeed four entries saying "strong reject", but they all appear to be from the same reviewer, at the same time, and saying the same thing. Isn't this just the one rejection? Also, why is only that reviewer's score visible?

pmags 3y ago

I'm curious how many commenters here who are making strong statements about the worth (or not) of peer review have actually participated in it both as author AND reviewer? Or even as an editor who is faced with the challenge of integrating and synthesizing multiple reviews into a recommendation? There are many venues available to share your research or ideas absent formal peer review, arXiv/bioRxiv being among the most popular. If you reject the idea of peer review itself it seems like there are plenty of alternatives.

ska 3y ago
It's the internet, therefore a significant percentage of the strong opinions about any topic will come from people who have little to no experience or competence in the area. Being HN, it probably skews a bit better that average. OTOH, it will also skew towards people procrastinating. Factor that in how you will...

mxwsn 3y ago

Flagged for misleading title - the four strong rejects are from a single author. It's listed four times for some unknown reason but likely an openreview quirk. The actual status described by the page is: 2 unknown (with accompanying long text), 1 weak reject, and 1 strong reject.

zaptheimpaler 3y ago

We already have a better mechanism for publishing and peer review.. it's called the internet. Literally the comments section of Reddit would work better. Reviews would be tied to a pseudonymous account instead of anonymous, allowing people to judge the quality of reviewers as well. Hacker News would work just as well too. It's also nearly free to setup a forum and frictionless to use compared to paying academic journals $100k for them to sell your own labour back to you. Cost and ease of use also mean more broadly accessible and hence more widely reviewed.

vasco 3y ago
Every once in a while I see a thread on reddit about a subject I know about and if someone shares a factual account that sounds unpopular it'll be downvoted even though it's true. I think reddit would be a terrible way to do this.
- raverbashing 3y ago
  But academic review is like that, with the worse acktchsually guys in existance
  - ribosometronome 3y ago
    The worst? Are you sure? Reddit's worst acktchsually guys are often spilling out of actual cesspit hate subreddits.
    - raverbashing 3y ago
      There is some meta-knowledge in your comment, but I'm focusing solely on the critique and pedantry levels, no comment on other factors
- 0xDEAFBEAD 3y ago
  https://www.lesswrong.com/ https://www.lesswrong.com/ lets you vote separately on agreement and quality axes. That seems to help a little bit.

raverbashing 3y ago

And this is why the biggest evolution of AI has happened in companies, not in academic circles Because there's too much nitpicking and grasping at straws amongst people that can't see novelty even when it's dancing in front of them

layer8 3y ago
No, the reason is that it required substantial financial investments, and in some cases access to proprietary big-data collections.
- raverbashing 3y ago
  word2vec does not require a large amount of data mnist might have required a large amount of data at its creation, but it became a staple dataset There was a lot of evolution before ChatGPT
  - L3viathan 3y ago
    And people in academia were all over Word2Vec. Mikolov presented his work in our research group around 2014, and people were very excited. Granted, that was _after_ Word2Vec had been published, and this was a very pro-vectorspaces (although of a different type) crowd.

m3kw9 3y ago

That didn’t age well

picometer 3y ago

In hindsight, reviewer f5bf’s comment is fascinating: > - It would be interesting if the authors could say something about how these models deal with intransitive semantic similarities, e.g., with the similarities between 'river', 'bank', and 'bailout'. People like Tversky have advocated against the use of semantic-space models like NLMs because they cannot appropriately model intransitive similarities. What I’ve noticed in the latest models (GPT, image diffusion models, etc) is an ability to play with words when there’s a double meaning. This struck me as something that used to be very human, but is now in the toolbox of generative models. (Most of which, I assume, use something akin word2vec for deriving embedding vectors from prompts.) Is the word2vec ambiguity contributing to the wordplay ability? I don’t know, but it points to a “feature vs bug” situation where such an ambiguity is a feature for creative purposes, but a bug if you want to model semantic space as a strict vector space. My interpretation here is that the word/prompt embeddings in current models are so huge that they’re overloaded with redundant dimensions, such that it wouldn’t satisfy any mathematical formalism (eg of well-behaved vector spaces) at all.

intalentive 3y ago
Even small models (e.g. hidden dims = 32) should be able to handle token ambiguity with attention. The information is not so much in the token itself as in the context.
PeterisP 3y ago
The key difference is what I'd call "context-free embeddings" vs "contextual embeddings". Due to its structure, word2vec and similar solutions have to assign every single "bank" in every sentence the exact same vector, but later models (e.g. all the transformer models, BERT, GPT, etc) will assign wildly different vectors to "bank" depending on the context of surrounding words for that particular mention of "bank".

PaulHoule 3y ago

I'd reject it still (speaking of someone who has developed products based on word vectors, document vectors, dimensional reduction, etc. before y'all thought it was cool...) I quit a job because they were insisting on using Word2Vec in an application where it would have doomed the project to failure. The basic problem is that in a real-life application many of the most important words are not in the dictionary and if you throw out words that are not in the dictionary you choose to fail. Let a junk paper like that through and the real danger is that you will get 1000s of other junk papers following it up. For instance, take a look at the illustrations on this page https://nlp.stanford.edu/projects/glove/ https://nlp.stanford.edu/projects/glove/ particularly under "2. Linear Substructures". They make it look like a miracle that they project down from a 50-dimensional subspace down to 2 and get a nice pattern of cities and zip codes, for instance. The thing is you could have a random set of 20 points in a 50-d space and, assuming there are no degeneracy, you can map them to any 20 points you want in the 2-d space with an appropriately chosen projection matrix. Show me a graph like that with 200 points and I might be impressed. (I'd say those graphs on that server damage the Stanford brand for me about as much as SBF and Marc Tessier-Lavign) (It's a constant theme in dimensional reduction literature that people forget that random matrices often work pretty well, fail to consider how much gain they are getting over the random matrix, ...) BERT, FastText and the like were revolutionary for a few reasons, but I saw the use of subword tokens as absolutely critical because... for once, you could capture a medical note and not erase the patient's name! The various conventions of computer science literature prevented explorations that would have put Word2Vec in its place. For instance, it's an obvious idea that you should be able to make a classifier that, given a document vector, can predict "is this a color word?" or "is this a verb?" but if you actually try it, it doesn't work in a particularly maddening way. With a tiny training/eval set (say 10 words) you might convince yourself it is working, but the more data you train on the more you realize the words are scattered mostly randomly and even those those "linear structures" exist in a statistical sense they aren't well defined and not particularly useful. It's the kind of thing that is so weird and inconclusive and fuzzy that I'm not aware of anyone writing a paper about it... Cause you're not going to draw any conclusions out of it except that you found a Jupiter-sized hairball. For all the excitement people had over Word2Vec you didn't see an explosion of interest in vector search engines because... Word2Vec sucked, applying it to documents didn't improve the search engine very much. Some of it is that adding sensitivity to synonyms can hurt performance because many possible synonyms turn out to be red herrings. BERT, on the other hand, is context sensitive and is able to some extent know the different because "my pet jaguar" and "the jaguar dealership in your town" and that really does help find the relevant documents and hide the irrelevant documents.

funnystories 3y ago

when i was on college, i wrote a simple system to make corrections on text based on some heuristics for a class. then, the teacher of the class suggested me to write a paper describing the system for a local conference during the summer, with some results etc I wrote it with his support but it got rejected right away because of poor grammar or something similar. the conference was in Brazil, but required the papers to be in English. I was just a student and thought that indeed my english was pretty bad. the teacher said to me to at least send an email to the reviewers to get some feedback, maybe resubmit with the corrections. i asked specifically which paragraphs were confusing. they sent me some snippets of phrases that were obviously wrong. yes, they were the "before" examples of "before/after" my system applied the corrections. I tried to explain that the grammar should be wrong, but the just replied with "please fix your english mistakes and resubmit". i tried 2 or 3 more times but just gave up.

adastra22 3y ago
You remind me of these anecdotes by Feynman of his time in Brazil. Specifically search for "I was invited to give a talk at the Brazilian Academy of Sciences", but the whole thing is worth a read if you haven't seen it: https://southerncrossreview.org/81/feynman-brazil.html https://southerncrossreview.org/81/feynman-brazil.html
wizzwizz4 3y ago
*eyeroll* Sounds about right. Want to get that published anyway? You could pop it on the arXiv and let the HN hivemind suggest an appropriate venue. If you don't have arXiv access, find an endorser <https://info.arxiv.org/help/endorsement.html https://info.arxiv.org/help/endorsement.html>, and send them a SHORT polite email (prioritise brevity over politeness) with your paper and the details. Something like: > Hello, > I wrote a paper for college in yyyy (attached) on automatic grammar correction, which got rejected from Venue for grammatical errors in the figures. I still want to publish it. Could you endorse my arXiv account, please? > Also, could you suggest an appropriate venue to submit this work to? > Yours sincerely, > your name, etc Follow the guidance on the arXiv website when asking for endorsement.
- 3y ago

nsagent 3y ago

Previous discussion: https://news.ycombinator.com/item?id=38654038 https://news.ycombinator.com/item?id=38654038

rahmaniacc 3y ago

This was hilarious! Many very broad and general statements are made without any citations to back them up. - Please be more specific. The number of self-citations seems somewhat excessive. - We added more citations.

ashvardanian 3y ago

That's the most inspiring thing I've learned this year.

jongjong 3y ago

I've found that PhD level academics are usually wrong about practical matters. It's almost as if having a PhD is itself proof that they are not good at identifying optimal solutions to reach practical goals. Also, it may show that they are overly concerned with superficial status markers instead of results. I think it's also the secret to why they can go so deep on certain abstract topics. Their minds are lacking a garbage collector and they can go in literally any direction and accumulate any amount of knowledge and they'll be able to memorize it all easily even if it has 0 utility value.