5 ms·

There are more details in the FB post of Tomas Mikolov (author of word2vec) recently: https://www.facebook.com/share/p/kXYaYaRvRCr5K2Ze https://www.facebook.com

by magnio 3y ago

There are more details in the FB post of Tomas Mikolov (author of word2vec) recently: https://www.facebook.com/share/p/kXYaYaRvRCr5K2Ze https://www.facebook.com/share/p/kXYaYaRvRCr5K2Ze A hilarious and poignant point I see is how experts make mistake too. Quote: > I also received a lot of comments on the word analogies - from "I knew that too but forgot to publish it!" (Geoff Hinton, I believe you :) happens to everyone, and anyways I think everybody knows what the origin of Distributed Representations is) to "it's a total hack and I'm sure it doesn't work!" (random guys who didn't bother to read the papers and try it out themselves - including Ian Goodfellow raging about it on Twitter).

nybsjytm 3y ago
I tried asking on another thread what Goodfellow rage he's referring to since all I could find was this: https://twitter.com/goodfellow_ian/status/1133528189651677184 https://twitter.com/goodfellow_ian/status/113352818965167718... If so, frankly I think it makes Mikolov sound pretty insecure.
- lern_too_spel 3y ago
  Twitter no longer shows threads to people who aren't logged in. https://nitter.net/goodfellow_ian/status/1133528189651677184 https://nitter.net/goodfellow_ian/status/1133528189651677184
- deleted 3y ago
  [deleted]
imjonse 3y ago
That post sounds like a rant TBH with too many stabs at various people. It could have been a lot more graceful. OTOH I can believe most researchers are human and do not put the progress of shared knowledge first but are very much influenced by ego and money *cough* OpenAI *cough*
- ReactiveJelly 3y ago
  To err is human, to seek profit is common to all lifeforms
albertzeyer 3y ago
Also, Tomas says he came up with the encoder-decoder (seq-to-seq) idea, and then Ilya and Quoc took over the idea after Tomas moved on to Facebook. However, there is another statement by Quoc, saying this is not true: https://twitter.com/quocleix/status/1736523075943125029 https://twitter.com/quocleix/status/1736523075943125029 > We congratulate Tomas on winning the award. Regarding seq2seq, there are inaccuracies in his account. In particular, we all recall very specifically that he did not suggest the idea to us, and was in fact highly skeptical when we shared the end-to-end translation idea with him. Indeed, we worked very hard to make it work despite his skepticism. So, word against word. I'm not accusing anyone of lying here, one of them probably just misremembers, but this leaves also a somewhat bad taste.
- minwcnt5 3y ago
  I think this must happen all the time. As they say, ideas are cheap. It's likely that ALL of them had the seq-to-seq idea cross their mind at some point before it was acted on, so if credit is assigned to whoever said it out loud first, there's going to be disagreement, since most people don't remember the full details of every conversation. It's also possible for someone to be skeptical of their own idea, so that argument isn't compelling to me either. Ultimately credit usually goes to the people who do the hard work to prove out the idea, so it seems like the system worked as intended in this case.
- oldesthacker 3y ago
  This is what Tomas Mikolov said on Facebook: > I wanted to popularize neural language models by improving Google Translate. I did start collaboration with Franz Och and his team, during which time I proposed a couple of models that could either complement the phrase-based machine translation, or even replace it. I came up (actually even before joining Google) with a really simple idea to do end-to-end translation by training a neural language model on pairs of sentences (say French - English), and then use the generation mode to produce translation after seeing the first sentence. It worked great on short sentences, but not so much on the longer ones. I discussed this project many times with others in Google Brain - mainly Quoc and Ilya - who took over this project after I moved to Facebook AI. I was quite negatively surprised when they ended up publishing my idea under now famous name "sequence to sequence" where not only I was not mentioned as a co-author, but in fact my former friends forgot to mention me also in the long Acknowledgement section, where they thanked personally pretty much every single person in Google Brain except me. This was the time when money started flowing massively into AI and every idea was worth gold. It was sad to see the deep learning community quickly turn into some sort of Game of Thrones. Money and power certainly corrupts people... Reddit post: "Tomas Mikolov is the true father of sequence-to-sequence" https://www.reddit.com/r/MachineLearning/comments/18jzxpf/d_ https://www.reddit.com/r/MachineLearning/comments/18jzxpf/d_...
jncfhnb 3y ago
To be fair I have some memories of the papers and surrounding tech being really bad. The popular implementations didn’t actually do what the papers said and the tech wasn’t great for anything beyond word level comparisons. You got some juice doing tf idf weighting of specific words but then tf idf weighted bag of words was similarly powerful. Cosine similarity of the sum of different word vectors sounds soooo dumb nowadays imo
- derangedHorse 3y ago
  Cosine similarity is like a very primitive version of word2vec. They both work off the idea of context being the key to the derivation of a token's semantic meaning. Cosine similarity is not very useful in itself but it's helpful for understanding the progression in the space
iab 3y ago
I wrote a detailed proof years before Mikolov on Twitter, but the 280 characters were too small to contain it
- WeMoveOn 3y ago
  Did you now?? I'll have you know that I wrote the full word2vec paper on a roll of shabby two-ply tissue paper during my time in a Taco Bell stall. Sadly, it was then used to mop up my dietary regrets and was subsequently lost to foul wretches of the sewage system. Left with nothing but the memories of my groundbreaking thoughts and the lingering aroma of liquid feces, I texted Mikolov an idea I had about using neural nets to map sequences of text tokens from one language to another only for him to reply "lol thx" and ghost me. I was quite negatively surprised when he decided to take this to the public courts of Facebook and failed to mention the "brainest boi alive™" who gave him this idea in the first place.
zniturah 3y ago
The post would have found a more fitting home on Twitter rather than the 'forgotten' realms of FB. The very individuals and entities mentioned (or hinted at) might have had the chance to see the story and share their perspectives. Otherwise it is just sounds like a rant.

by magnio 3y ago

nybsjytm 3y ago

I tried asking on another thread what Goodfellow rage he's referring to since all I could find was this: https://twitter.com/goodfellow_ian/status/1133528189651677184 https://twitter.com/goodfellow_ian/status/113352818965167718... If so, frankly I think it makes Mikolov sound pretty insecure.

lern_too_spel 3y ago
Twitter no longer shows threads to people who aren't logged in. https://nitter.net/goodfellow_ian/status/1133528189651677184 https://nitter.net/goodfellow_ian/status/1133528189651677184
deleted 3y ago
[deleted]

imjonse 3y ago

That post sounds like a rant TBH with too many stabs at various people. It could have been a lot more graceful. OTOH I can believe most researchers are human and do not put the progress of shared knowledge first but are very much influenced by ego and money *cough* OpenAI *cough*

ReactiveJelly 3y ago
To err is human, to seek profit is common to all lifeforms

albertzeyer 3y ago

Also, Tomas says he came up with the encoder-decoder (seq-to-seq) idea, and then Ilya and Quoc took over the idea after Tomas moved on to Facebook. However, there is another statement by Quoc, saying this is not true: https://twitter.com/quocleix/status/1736523075943125029 https://twitter.com/quocleix/status/1736523075943125029 > We congratulate Tomas on winning the award. Regarding seq2seq, there are inaccuracies in his account. In particular, we all recall very specifically that he did not suggest the idea to us, and was in fact highly skeptical when we shared the end-to-end translation idea with him. Indeed, we worked very hard to make it work despite his skepticism. So, word against word. I'm not accusing anyone of lying here, one of them probably just misremembers, but this leaves also a somewhat bad taste.

minwcnt5 3y ago
I think this must happen all the time. As they say, ideas are cheap. It's likely that ALL of them had the seq-to-seq idea cross their mind at some point before it was acted on, so if credit is assigned to whoever said it out loud first, there's going to be disagreement, since most people don't remember the full details of every conversation. It's also possible for someone to be skeptical of their own idea, so that argument isn't compelling to me either. Ultimately credit usually goes to the people who do the hard work to prove out the idea, so it seems like the system worked as intended in this case.
oldesthacker 3y ago
This is what Tomas Mikolov said on Facebook: > I wanted to popularize neural language models by improving Google Translate. I did start collaboration with Franz Och and his team, during which time I proposed a couple of models that could either complement the phrase-based machine translation, or even replace it. I came up (actually even before joining Google) with a really simple idea to do end-to-end translation by training a neural language model on pairs of sentences (say French - English), and then use the generation mode to produce translation after seeing the first sentence. It worked great on short sentences, but not so much on the longer ones. I discussed this project many times with others in Google Brain - mainly Quoc and Ilya - who took over this project after I moved to Facebook AI. I was quite negatively surprised when they ended up publishing my idea under now famous name "sequence to sequence" where not only I was not mentioned as a co-author, but in fact my former friends forgot to mention me also in the long Acknowledgement section, where they thanked personally pretty much every single person in Google Brain except me. This was the time when money started flowing massively into AI and every idea was worth gold. It was sad to see the deep learning community quickly turn into some sort of Game of Thrones. Money and power certainly corrupts people... Reddit post: "Tomas Mikolov is the true father of sequence-to-sequence" https://www.reddit.com/r/MachineLearning/comments/18jzxpf/d_ https://www.reddit.com/r/MachineLearning/comments/18jzxpf/d_...

jncfhnb 3y ago

To be fair I have some memories of the papers and surrounding tech being really bad. The popular implementations didn’t actually do what the papers said and the tech wasn’t great for anything beyond word level comparisons. You got some juice doing tf idf weighting of specific words but then tf idf weighted bag of words was similarly powerful. Cosine similarity of the sum of different word vectors sounds soooo dumb nowadays imo

derangedHorse 3y ago
Cosine similarity is like a very primitive version of word2vec. They both work off the idea of context being the key to the derivation of a token's semantic meaning. Cosine similarity is not very useful in itself but it's helpful for understanding the progression in the space

iab 3y ago

I wrote a detailed proof years before Mikolov on Twitter, but the 280 characters were too small to contain it

WeMoveOn 3y ago
Did you now?? I'll have you know that I wrote the full word2vec paper on a roll of shabby two-ply tissue paper during my time in a Taco Bell stall. Sadly, it was then used to mop up my dietary regrets and was subsequently lost to foul wretches of the sewage system. Left with nothing but the memories of my groundbreaking thoughts and the lingering aroma of liquid feces, I texted Mikolov an idea I had about using neural nets to map sequences of text tokens from one language to another only for him to reply "lol thx" and ghost me. I was quite negatively surprised when he decided to take this to the public courts of Facebook and failed to mention the "brainest boi alive™" who gave him this idea in the first place.

zniturah 3y ago

The post would have found a more fitting home on Twitter rather than the 'forgotten' realms of FB. The very individuals and entities mentioned (or hinted at) might have had the chance to see the story and share their perspectives. Otherwise it is just sounds like a rant.