13 ms·

Differences between the word2vec paper and its implementation

351 points by bollu 7y ago

newen 7y ago
This kind of things happens all the time in academia. The authors are either constrained by space due to paper limitations or they are too lazy to explain all the little details that go into the algorithm. I used to do research in computer vision a few years ago and it used to be that people won't publish their code and they purposely won't put in all of the details of the algorithm in the paper. Many of those algorithms were patent pending and I assume the authors were hoping to make some money from the patents. Compared to that, it's a lot better nowadays where most of the popular papers come with published code.
- deleted 7y ago
  [deleted]
- bollu 7y ago
  Is this really that common? That's disheartening, I want to spend time in academia but experiences like this are sucking the fun out for me...
  - gameswithgo 7y ago
    welcome to earth. we apologize for the mess.
  - nmstoker 7y ago
    Yes, this was a huge disappointment when I read Chemistry & Physics years ago. The naive view of papers was that they were to move human knowledge forward, but it became clear they were an elaborate knowledge withholding device! Fortunately the trend seems to be towards better levels of openness, but it varies by subject. Stories like the BASF team being unable to reproduce vast numbers of established published techniques are way too common.
  - austinjp 7y ago
    Academia isn't flawless, it's no paradise. It's just humans and their factions, with all the good and bad that brings. > The authors are ... constrained by space due to paper limitations This is very real. Different journals have different criteria: word count; formatting; the number of tables you're permitted to include; etc. It's archaic and daft but that's the truth of it. And that's before we even get started on the really bonkers stuff like author order, impact factor, reviewer workload vs lack of pay, publish or perish, and so on.
  - deleted 7y ago
    [deleted]
  - toast0 7y ago
    I tried to make use of some public audio research and it was pretty bad. There was an audio comprehensibility competition a few years ago. Some of the papers submitted are still around, as well as the summary paper describing the results. But many papers are hard to find, and those that claimed to have source code available are hard to find --- i was able to get matlab sources for a few algorithms, but they somehow work on the example files, but mostly crash on my files. It's a shame because I understand the idea of the paper, and have an excellent place to apply it, but I lack the DSP background, so I can't really rebuild the code from scratch -- so the work is not able to be used.
    - DoctorOetker 7y ago
      this sounds interesting, would you care to reference the paper in question?
      - toast0 7y ago
        I'm not sure if I can find the exact paper anymore. This was in response to the Hurricane Challenge, a summary of results is available [1]. I tried to use code for uwSSDRCt available from the legacy page of the conference [2], under the link "Live and recorded speech modifier", direct download here [3]. The basic context is verification code delivery -- I'm playing pre-recorded samples of numbers to users, and can't control or sample the noise (either transmission or environmental), but would like to enhance intelligibility to reduce user effort, improve experience, and reduce costs. [1] https://www.research.ed.ac.uk/portal/files/17887878/Cooke_et_al_2013_Intelligibility_enhancing_speech_modifications.pdf https://www.research.ed.ac.uk/portal/files/17887878/Cooke_et... [2] https://web.archive.org/web/20131012005150/http://listening-talker.org/legacy.html https://web.archive.org/web/20131012005150/http://listening-... [3] http://www.laslab.org/resources/LISTA/code/D4.3.zip http://www.laslab.org/resources/LISTA/code/D4.3.zip
  - gubbrora 7y ago
    Yes it's common that a lot of "little details" are left out. Yes it does make things harder. No it's not a big enough problem that you need to be disheartened. You could even see it as an opportunity to raise the bar and stand out positively.
- chriskanan 7y ago
  Completely agree. The situation today is far better than a decade ago, with code releases for machine learning and computer vision papers being much more common than before. I try to make my students release polished and easy to use code, but it can sometimes fall through the cracks due to deadlines, etc. Many projects are the output of a single PhD student.
  - newen 7y ago
    Yep, it's so much better now in computer vision with early publishing in arxiv and published code. I feel like that is one of the reasons why research in CV is progressing so fast. Also, I think it is something of a fact of life that you can't put all of the details of your algorithm in the paper, not just paper length limitations. I have actually tried to do this in two papers by putting all of the details in the supplements and the work to explain all the details and justify my choice of parameters and decisions for edge cases is almost as hard as writing the main paper. It becomes hard to justify the time spent pretty fast. Also, putting this in the main paper makes your beautiful explanations be tinged with edge cases and digressions haha.
- izacus 7y ago
  Even worse, when actually researching CV, it seems that leaving out the details can even be deliberate in papers - to hide the fact that the authors tuned and cherry picked input datasets so they show the improvements they claim. Actually implementing the code and running on more standard datasets tends to quickly put results of many papers into question.
MichaelStaniek 7y ago
My intuition for that, and you can tell me if its wrong. The normal explanation for Word2Vec is 2 weight matrices, so the formula looks like this: (One_hot_input x W1) x W2, which is then softmaxed. W1 then is the matrix that contain our focus embedding from, but if we only evaluate specific words on the target side, then W2 are actually our context embeddings, and the normal multiplication then is focus_w x context_w. Am I wrong?
- bollu 7y ago
  Now it's `one_hot_focus x W1 x (one_hot_context x W2)^T`. So we still pick one row of the matrix from the focus and context embeddings, but they're separate embeddings.
  - MichaelStaniek 7y ago
    Yes, but thats also what happens in the normal formulation, no? So the second weight matrix actually are our context embeddings?
  - avn2109 7y ago
    New user "MichaelStaniek" has a grammatical, relevant, good-faith sibling comment to this which is (imho) inexplicably banned, almost certainly due to a mistake, and I hope somebody will unban him.
billconan 7y ago
For the past week I have been frustrated by an opensource code of a deep learning paper. This type of things are so common in academia. The particular code I looked at has missing documentation, hardcoded local paths, broken dataset download links and broken pretrained model download links. I have to fix bugs before the code can run. I'm very curious how did the author run that code with the bugs. I call them insincerely opensourced projects.
- GuyPostington 7y ago
  Have you been able to reach out to the author and get clarification?
  - billconan 7y ago
    I tried to file bugs on github, only to find out that people (sometimes 2 years ahead me) was blocked by the same issue and there are no answers. They didn't even bother to merge my pull requests.
    - scott_s 7y ago
      There's probably a disconnect in expectations. The researchers are probably publishing the code more as a document, to say, hey, this is how we did this. It sounds like you have expectations that the code is part of an ongoing project. But ongoing projects require maintenance, and the researchers have likely moved on to new things.
- colechristensen 7y ago
  It is very common. Scientist code isn't software engineering code (and software engineering code is very often worthy of criticism) It's just good news when you can find source code at all instead of just being told vague things about something the author did.
  - raverbashing 7y ago
    > Scientist code isn't software engineering code (and software engineering code is very often worthy of criticism) Absolutely correct, and not in a good way Though to be fair, people doing research have other interests besides maintainable code (though half or more of the annoyances will probably come and bite them later)
DannyBee 7y ago
Speaking as someone who has read about 40 years of papers in compiler optimization, it's very interesting. In the early days, there were fairly exact algorithms that worked as described, and were implemented as described, but were pseudocoded in papers. Where the pseudocode differed from implementation, differences were described in great detail (IE they may say an array can be shared but isn't to make the pseudocode easier to describe). Then, over time, things start to get further away from that. You start to see papers published with algorithms that either don't work as described, or are so inefficient as to be unusable. Like literally cannot work. Where you can get source code, later looking at the source code shows that's not what they did at all. One infamous example of this is SSAPRE - to this day, people have a lot of trouble understanding the paper (and it has significant errors that make the algorithm incorrect as written). The concept sure, but the exact algorithm - less so. Reading the source code to it in Open64 - it is just wildly different than the paper (and often requires a lot of thought for people to convince themselves it is correct). It's not just better engineering/datastructures vs research algorithms. The one shining counterexample is the Rice folks who wrote their massively scalar compiler in nuweb (one of many literate programming environments), so the descriptions/papers and code were in the same place - these are very very readable and useful papers in my experience. Nowadays it's coming back to the earlier daysdue to github/et al. People seem to try to make the code more like the paper algorithm since they now release the code. Word2vec appears to be a counterexample (maybe because they released the code they didn't feel a need to get the paper as right)
- bollu 7y ago
  As someone who works on compiler optimisations, I'd love to get a link the Rice compiler's source code --- do you happen to have a link?
  - DannyBee 7y ago
    Not sure i still have it around ping me at the email in my profile and i'll see what i can do. It used to be on their ftp server, which they took down. I expect if you email ken kennedy over at rice, he may be able to find it as well.
b_tterc_p 7y ago
On a similar note, a long time ago I read the Doc2Vec paper, then looked at popular Doc2Vec implementations. They didn’t seem to do the same thing. The paper said you basically make vectors for words, then append on an additional space that represents the additional information of documents as opposed to single words. All popular implementations I found seemed to put the document vectors into the same space as the word vectors. They also didn’t seem to do any better than a tf-idf weighted average of word vectors... curious if anyone has ever bumped against this.
- gojomo 7y ago
  The only code released by the 'Paragraph Vector' paper authors was a small patch, from Mikolov, that added paragraph-vectors to the original `word2vec.c` implementation in a very simple way: treating the 1st token of each line as a special paragraph-vector, still string-named (and allocated in the same lookup dictionary). Only by convention (a special prefix on those paragraph-vector tokens) could collisions with similarly-named word-vectors avoided. That's a nice minimal way to demo/test the idea, but limited and fragile in other ways. The initial gensim implementation did something similar, then I changed it to use a separate doc-vectors space, to better support a lot of options (including the PV-DM mode with a concatenative input layer – which has never been confirmed to perform as well as the original paper implied).
  - b_tterc_p 7y ago
    Insightful. Thanks
- woliveirajr 7y ago
  This. And my thesis was about how tf-idf and embeddings with deep (deeeeeep) neural networks could be give better results in authorship attribution.
  - visarga 7y ago
    > embeddings with deep (deeeeeep) neural networks Embeddings are shallow, and what comes after them is usually less than 5 layers of LSTM, not a deeeeeep neural net (maybe, deep only horizontally, on the axis of words).
deleted 7y ago
[deleted]
lelf 7y ago
It’s patented BTW — https://patents.google.com/patent/US9037464B1/en https://patents.google.com/patent/US9037464B1/en
- deleted 7y ago
  [deleted]
- rurban 7y ago
  Turns the patent only describes the paper, but not the implementation. Great, and somewhat ironic
utopcell 7y ago
I think it very unfair to the original set of word2vec papers to be talking about 'academic dishonesty'. This is a case of a user that has little to no experience with neural networks. There are a ton of articles describing the need for random initialization [1][2]. In fact, if one spends a few seconds thinking about it, the need is evident. Without it, the NN cannot perform symmetry breaking: If inputs are set to zero, all neurons will perform the same calculations, rendering the network useless. [1] google: "neural networks vector initialization" [2] http://deeplearning.ai/ai-notes/initialization/ http://deeplearning.ai/ai-notes/initialization/
- bollu 7y ago
  That's hardly the point of the article --- the actual paper does not describe the use of two separate vectors for each word. The initialization was an interesting tidbit.
  - exgrv 7y ago
    Except it does? After Equation 2: "v_w and v'_w are the input and output vector representations of w."
  - utopcell 7y ago
    Of course it does. For word w they are v_w and v'_w on Eq.4 for the case of SGNS [1]. [1] https://papers.nips.cc/paper/5021-distributed-representations-of-words-and-phrases-and-their-compositionality.pdf https://papers.nips.cc/paper/5021-distributed-representation...
  - andreasvc 7y ago
    bollu, multiple people have shown that your claim that the paper doesn't match the code is flat out wrong. I think at this point you should issue a retraction of your wildly inappropriate suggestion of academic dishonesty.
RyEgswuCsn 7y ago
I find the title of the article rather exaggerating... As of the first difference pointed out in the article, one of the CS224D lectures on word2vec did addressed it: https://youtu.be/aRqn8t1hLxs?t=2650 https://youtu.be/aRqn8t1hLxs?t=2650 It was also mentioned later in the lecture that having two vectors representing each word is meant to make the optimisation easier (so it's kind of a trick); at the end, the two vectors learnt will have to be averaged over in order to reach a single vector for each word. To be fair, the fact that each word is represented by two vectors was also mentioned in the original paper describing word2vec: https://arxiv.org/pdf/1310.4546.pdf https://arxiv.org/pdf/1310.4546.pdf On page 3, just beneath equation (2). Why so surprised?
xxxpupugo 7y ago
The title reads to me like hyperbole. The implementation can differ, they got time to refactor/optimize it after the publication. But they can't probably revise the paper itself. As long as the code is there and can produce said/better result, then it is probably your responsibility to keep the differences in check. It is actually quite common for deep learning papers overall, the github repo gets updated after the paper is out, and you will find the divergence lying there.
eggie5 7y ago
instead of thinking about what it is in practice: skip-gram negative sampling, I think it's much more intuitive to think about what it is in theory: extreme multi-class classification. word2vec is a multi-class classification problem with a softmax output layer and cross-entropy loss. The novel part of word2vec, in my opinion, is two: 1. dataset (proximal input word & output word) generation from documents eg: skiagram, CBOW, etc 2. engineering speedup for softmax: Approximate Softmax eg Negative Sampling using NCE, hierarchal softmax, etc If you just build word2vec w/o step 2, it's a easier to understand. Then when you get that working, add in the negative sampling speedup trick which isn't core the theoretical algorithm.
- utopcell 7y ago
  Can't really call it a speedup trick, since it actually improves the performance of the embeddings but in terms of qualitative understanding, I see where you're coming from.
skythomas 7y ago
Grossly unfair title
gojomo 7y ago
They're not really that different. There's only a second vector for a word in the (common, default) negative-sampling case, where each predictable word has a distinct "output node" of the neural network, and the second vector is the in-weights to that one node. Still, most implementations don't emphasize this vector – the classic "word-vector" is a word's representation when it's a neural-network input. And in the hierarchical-softmax training mode, there's no clear second vector. I suspect the original word2vec authors left out a clearer description of the initialization as they were following some oft-assumed practices implied by their other descriptions. Another minor difference between the literal descriptions, and original C implementation, was a slightly different looping order in skip-gram training: holding a target-word, and then looping over all context-words, rather than holding a context-word, then looping over all neighboring target-words. One of the authors once mentioned that the shipped approach was slightly more efficient – maybe it was due to CPU cache issues? In any case all the same context->target pairs get trained either way, just in a slightly different order.
slx26 7y ago
Just as a curiosity, complementing what others have already written... I read part of Mikolov's thesis and code in the past (when I was still studying at the university, so I might have got everything wrong (I still don't get half of it :D)). First I found it quite shocking that the code was so bad. The training code was pretty confusing to me, and I found the lack of useful comments discouraging. The test code (which loaded stored embeddings from a file and allowed some basic operations) was even much, much worse. Like, declaring three variables (a, b, c) and reusing them for different things in the main functions without explaining anything, and doing linear searches through the whole embeddings to find a word vector... very ugly and scary things. So, I had a very bad impression of the code. But then, I checked the thesis, and I found it awesome. The amount of tests and implementations the guy made, and how he showed in practice how better results could be achieved in a good number of different setups... I found it really impressive. But such great work paired with such bad code! I was just a CS student, so I found it shocking. Nowadays I realize he was simply focused on a different thing, and the results he obtained were indeed outstanding and talk for themselves. It's easy to look back and criticise the code, but when you look at the work he did in perspective... it's completely unfair to ask more from him (admittedly, they had time to address some of the issues later, but they probably had better things to do too).

351 points by bollu 7y ago

This kind of things happens all the time in academia. The authors are either constrained by space due to paper limitations or they are too lazy to explain all the little details that go into the algorithm. I used to do research in computer vision a few years ago and it used to be that people won't publish their code and they purposely won't put in all of the details of the algorithm in the paper. Many of those algorithms were patent pending and I assume the authors were hoping to make some money from the patents. Compared to that, it's a lot better nowadays where most of the popular papers come with published code.

deleted 7y ago
[deleted]
bollu 7y ago
Is this really that common? That's disheartening, I want to spend time in academia but experiences like this are sucking the fun out for me...
- gameswithgo 7y ago
  welcome to earth. we apologize for the mess.
- nmstoker 7y ago
  Yes, this was a huge disappointment when I read Chemistry & Physics years ago. The naive view of papers was that they were to move human knowledge forward, but it became clear they were an elaborate knowledge withholding device! Fortunately the trend seems to be towards better levels of openness, but it varies by subject. Stories like the BASF team being unable to reproduce vast numbers of established published techniques are way too common.
- austinjp 7y ago
  Academia isn't flawless, it's no paradise. It's just humans and their factions, with all the good and bad that brings. > The authors are ... constrained by space due to paper limitations This is very real. Different journals have different criteria: word count; formatting; the number of tables you're permitted to include; etc. It's archaic and daft but that's the truth of it. And that's before we even get started on the really bonkers stuff like author order, impact factor, reviewer workload vs lack of pay, publish or perish, and so on.
- deleted 7y ago
  [deleted]
- toast0 7y ago
  I tried to make use of some public audio research and it was pretty bad. There was an audio comprehensibility competition a few years ago. Some of the papers submitted are still around, as well as the summary paper describing the results. But many papers are hard to find, and those that claimed to have source code available are hard to find --- i was able to get matlab sources for a few algorithms, but they somehow work on the example files, but mostly crash on my files. It's a shame because I understand the idea of the paper, and have an excellent place to apply it, but I lack the DSP background, so I can't really rebuild the code from scratch -- so the work is not able to be used.
  - DoctorOetker 7y ago
    this sounds interesting, would you care to reference the paper in question?
    - toast0 7y ago
      I'm not sure if I can find the exact paper anymore. This was in response to the Hurricane Challenge, a summary of results is available [1]. I tried to use code for uwSSDRCt available from the legacy page of the conference [2], under the link "Live and recorded speech modifier", direct download here [3]. The basic context is verification code delivery -- I'm playing pre-recorded samples of numbers to users, and can't control or sample the noise (either transmission or environmental), but would like to enhance intelligibility to reduce user effort, improve experience, and reduce costs. [1] https://www.research.ed.ac.uk/portal/files/17887878/Cooke_et_al_2013_Intelligibility_enhancing_speech_modifications.pdf https://www.research.ed.ac.uk/portal/files/17887878/Cooke_et... [2] https://web.archive.org/web/20131012005150/http://listening-talker.org/legacy.html https://web.archive.org/web/20131012005150/http://listening-... [3] http://www.laslab.org/resources/LISTA/code/D4.3.zip http://www.laslab.org/resources/LISTA/code/D4.3.zip
- gubbrora 7y ago
  Yes it's common that a lot of "little details" are left out. Yes it does make things harder. No it's not a big enough problem that you need to be disheartened. You could even see it as an opportunity to raise the bar and stand out positively.
chriskanan 7y ago
Completely agree. The situation today is far better than a decade ago, with code releases for machine learning and computer vision papers being much more common than before. I try to make my students release polished and easy to use code, but it can sometimes fall through the cracks due to deadlines, etc. Many projects are the output of a single PhD student.
- newen 7y ago
  Yep, it's so much better now in computer vision with early publishing in arxiv and published code. I feel like that is one of the reasons why research in CV is progressing so fast. Also, I think it is something of a fact of life that you can't put all of the details of your algorithm in the paper, not just paper length limitations. I have actually tried to do this in two papers by putting all of the details in the supplements and the work to explain all the details and justify my choice of parameters and decisions for edge cases is almost as hard as writing the main paper. It becomes hard to justify the time spent pretty fast. Also, putting this in the main paper makes your beautiful explanations be tinged with edge cases and digressions haha.
izacus 7y ago
Even worse, when actually researching CV, it seems that leaving out the details can even be deliberate in papers - to hide the fact that the authors tuned and cherry picked input datasets so they show the improvements they claim. Actually implementing the code and running on more standard datasets tends to quickly put results of many papers into question.

MichaelStaniek 7y ago

My intuition for that, and you can tell me if its wrong. The normal explanation for Word2Vec is 2 weight matrices, so the formula looks like this: (One_hot_input x W1) x W2, which is then softmaxed. W1 then is the matrix that contain our focus embedding from, but if we only evaluate specific words on the target side, then W2 are actually our context embeddings, and the normal multiplication then is focus_w x context_w. Am I wrong?

bollu 7y ago
Now it's `one_hot_focus x W1 x (one_hot_context x W2)^T`. So we still pick one row of the matrix from the focus and context embeddings, but they're separate embeddings.
- MichaelStaniek 7y ago
  Yes, but thats also what happens in the normal formulation, no? So the second weight matrix actually are our context embeddings?
- avn2109 7y ago
  New user "MichaelStaniek" has a grammatical, relevant, good-faith sibling comment to this which is (imho) inexplicably banned, almost certainly due to a mistake, and I hope somebody will unban him.

billconan 7y ago

For the past week I have been frustrated by an opensource code of a deep learning paper. This type of things are so common in academia. The particular code I looked at has missing documentation, hardcoded local paths, broken dataset download links and broken pretrained model download links. I have to fix bugs before the code can run. I'm very curious how did the author run that code with the bugs. I call them insincerely opensourced projects.

GuyPostington 7y ago
Have you been able to reach out to the author and get clarification?
- billconan 7y ago
  I tried to file bugs on github, only to find out that people (sometimes 2 years ahead me) was blocked by the same issue and there are no answers. They didn't even bother to merge my pull requests.
  - scott_s 7y ago
    There's probably a disconnect in expectations. The researchers are probably publishing the code more as a document, to say, hey, this is how we did this. It sounds like you have expectations that the code is part of an ongoing project. But ongoing projects require maintenance, and the researchers have likely moved on to new things.
colechristensen 7y ago
It is very common. Scientist code isn't software engineering code (and software engineering code is very often worthy of criticism) It's just good news when you can find source code at all instead of just being told vague things about something the author did.
- raverbashing 7y ago
  > Scientist code isn't software engineering code (and software engineering code is very often worthy of criticism) Absolutely correct, and not in a good way Though to be fair, people doing research have other interests besides maintainable code (though half or more of the annoyances will probably come and bite them later)

DannyBee 7y ago

Speaking as someone who has read about 40 years of papers in compiler optimization, it's very interesting. In the early days, there were fairly exact algorithms that worked as described, and were implemented as described, but were pseudocoded in papers. Where the pseudocode differed from implementation, differences were described in great detail (IE they may say an array can be shared but isn't to make the pseudocode easier to describe). Then, over time, things start to get further away from that. You start to see papers published with algorithms that either don't work as described, or are so inefficient as to be unusable. Like literally cannot work. Where you can get source code, later looking at the source code shows that's not what they did at all. One infamous example of this is SSAPRE - to this day, people have a lot of trouble understanding the paper (and it has significant errors that make the algorithm incorrect as written). The concept sure, but the exact algorithm - less so. Reading the source code to it in Open64 - it is just wildly different than the paper (and often requires a lot of thought for people to convince themselves it is correct). It's not just better engineering/datastructures vs research algorithms. The one shining counterexample is the Rice folks who wrote their massively scalar compiler in nuweb (one of many literate programming environments), so the descriptions/papers and code were in the same place - these are very very readable and useful papers in my experience. Nowadays it's coming back to the earlier daysdue to github/et al. People seem to try to make the code more like the paper algorithm since they now release the code. Word2vec appears to be a counterexample (maybe because they released the code they didn't feel a need to get the paper as right)

bollu 7y ago
As someone who works on compiler optimisations, I'd love to get a link the Rice compiler's source code --- do you happen to have a link?
- DannyBee 7y ago
  Not sure i still have it around ping me at the email in my profile and i'll see what i can do. It used to be on their ftp server, which they took down. I expect if you email ken kennedy over at rice, he may be able to find it as well.

b_tterc_p 7y ago

On a similar note, a long time ago I read the Doc2Vec paper, then looked at popular Doc2Vec implementations. They didn’t seem to do the same thing. The paper said you basically make vectors for words, then append on an additional space that represents the additional information of documents as opposed to single words. All popular implementations I found seemed to put the document vectors into the same space as the word vectors. They also didn’t seem to do any better than a tf-idf weighted average of word vectors... curious if anyone has ever bumped against this.

gojomo 7y ago
The only code released by the 'Paragraph Vector' paper authors was a small patch, from Mikolov, that added paragraph-vectors to the original `word2vec.c` implementation in a very simple way: treating the 1st token of each line as a special paragraph-vector, still string-named (and allocated in the same lookup dictionary). Only by convention (a special prefix on those paragraph-vector tokens) could collisions with similarly-named word-vectors avoided. That's a nice minimal way to demo/test the idea, but limited and fragile in other ways. The initial gensim implementation did something similar, then I changed it to use a separate doc-vectors space, to better support a lot of options (including the PV-DM mode with a concatenative input layer – which has never been confirmed to perform as well as the original paper implied).
- b_tterc_p 7y ago
  Insightful. Thanks
woliveirajr 7y ago
This. And my thesis was about how tf-idf and embeddings with deep (deeeeeep) neural networks could be give better results in authorship attribution.
- visarga 7y ago
  > embeddings with deep (deeeeeep) neural networks Embeddings are shallow, and what comes after them is usually less than 5 layers of LSTM, not a deeeeeep neural net (maybe, deep only horizontally, on the axis of words).

deleted 7y ago

[deleted]

lelf 7y ago

It’s patented BTW — https://patents.google.com/patent/US9037464B1/en https://patents.google.com/patent/US9037464B1/en

deleted 7y ago
[deleted]
rurban 7y ago
Turns the patent only describes the paper, but not the implementation. Great, and somewhat ironic

utopcell 7y ago

I think it very unfair to the original set of word2vec papers to be talking about 'academic dishonesty'. This is a case of a user that has little to no experience with neural networks. There are a ton of articles describing the need for random initialization [1][2]. In fact, if one spends a few seconds thinking about it, the need is evident. Without it, the NN cannot perform symmetry breaking: If inputs are set to zero, all neurons will perform the same calculations, rendering the network useless. [1] google: "neural networks vector initialization" [2] http://deeplearning.ai/ai-notes/initialization/ http://deeplearning.ai/ai-notes/initialization/

bollu 7y ago
That's hardly the point of the article --- the actual paper does not describe the use of two separate vectors for each word. The initialization was an interesting tidbit.
- exgrv 7y ago
  Except it does? After Equation 2: "v_w and v'_w are the input and output vector representations of w."
- utopcell 7y ago
  Of course it does. For word w they are v_w and v'_w on Eq.4 for the case of SGNS [1]. [1] https://papers.nips.cc/paper/5021-distributed-representations-of-words-and-phrases-and-their-compositionality.pdf https://papers.nips.cc/paper/5021-distributed-representation...
- andreasvc 7y ago
  bollu, multiple people have shown that your claim that the paper doesn't match the code is flat out wrong. I think at this point you should issue a retraction of your wildly inappropriate suggestion of academic dishonesty.

RyEgswuCsn 7y ago

I find the title of the article rather exaggerating... As of the first difference pointed out in the article, one of the CS224D lectures on word2vec did addressed it: https://youtu.be/aRqn8t1hLxs?t=2650 https://youtu.be/aRqn8t1hLxs?t=2650 It was also mentioned later in the lecture that having two vectors representing each word is meant to make the optimisation easier (so it's kind of a trick); at the end, the two vectors learnt will have to be averaged over in order to reach a single vector for each word. To be fair, the fact that each word is represented by two vectors was also mentioned in the original paper describing word2vec: https://arxiv.org/pdf/1310.4546.pdf https://arxiv.org/pdf/1310.4546.pdf On page 3, just beneath equation (2). Why so surprised?

xxxpupugo 7y ago

The title reads to me like hyperbole. The implementation can differ, they got time to refactor/optimize it after the publication. But they can't probably revise the paper itself. As long as the code is there and can produce said/better result, then it is probably your responsibility to keep the differences in check. It is actually quite common for deep learning papers overall, the github repo gets updated after the paper is out, and you will find the divergence lying there.

eggie5 7y ago

instead of thinking about what it is in practice: skip-gram negative sampling, I think it's much more intuitive to think about what it is in theory: extreme multi-class classification. word2vec is a multi-class classification problem with a softmax output layer and cross-entropy loss. The novel part of word2vec, in my opinion, is two: 1. dataset (proximal input word & output word) generation from documents eg: skiagram, CBOW, etc 2. engineering speedup for softmax: Approximate Softmax eg Negative Sampling using NCE, hierarchal softmax, etc If you just build word2vec w/o step 2, it's a easier to understand. Then when you get that working, add in the negative sampling speedup trick which isn't core the theoretical algorithm.

utopcell 7y ago
Can't really call it a speedup trick, since it actually improves the performance of the embeddings but in terms of qualitative understanding, I see where you're coming from.

skythomas 7y ago

Grossly unfair title

gojomo 7y ago

They're not really that different. There's only a second vector for a word in the (common, default) negative-sampling case, where each predictable word has a distinct "output node" of the neural network, and the second vector is the in-weights to that one node. Still, most implementations don't emphasize this vector – the classic "word-vector" is a word's representation when it's a neural-network input. And in the hierarchical-softmax training mode, there's no clear second vector. I suspect the original word2vec authors left out a clearer description of the initialization as they were following some oft-assumed practices implied by their other descriptions. Another minor difference between the literal descriptions, and original C implementation, was a slightly different looping order in skip-gram training: holding a target-word, and then looping over all context-words, rather than holding a context-word, then looping over all neighboring target-words. One of the authors once mentioned that the shipped approach was slightly more efficient – maybe it was due to CPU cache issues? In any case all the same context->target pairs get trained either way, just in a slightly different order.

slx26 7y ago

Just as a curiosity, complementing what others have already written... I read part of Mikolov's thesis and code in the past (when I was still studying at the university, so I might have got everything wrong (I still don't get half of it :D)). First I found it quite shocking that the code was so bad. The training code was pretty confusing to me, and I found the lack of useful comments discouraging. The test code (which loaded stored embeddings from a file and allowed some basic operations) was even much, much worse. Like, declaring three variables (a, b, c) and reusing them for different things in the main functions without explaining anything, and doing linear searches through the whole embeddings to find a word vector... very ugly and scary things. So, I had a very bad impression of the code. But then, I checked the thesis, and I found it awesome. The amount of tests and implementations the guy made, and how he showed in practice how better results could be achieved in a good number of different setups... I found it really impressive. But such great work paired with such bad code! I was just a CS student, so I found it shocking. Nowadays I realize he was simply focused on a different thing, and the results he obtained were indeed outstanding and talk for themselves. It's easy to look back and criticise the code, but when you look at the work he did in perspective... it's completely unfair to ask more from him (admittedly, they had time to address some of the issues later, but they probably had better things to do too).