4 ms·
This is an interesting surprise about good old word vectors. From the README: > Although continuous bag of word (CBOW) embeddings can be trained more quickly t
by polm23 6y ago
This is an interesting surprise about good old word vectors. From the README:
> Although continuous bag of word (CBOW) embeddings can be trained more quickly than skipgram (SG) embeddings, it is a common belief that SG embeddings tend to perform better in practice. This was observed by the original authors of Word2Vec [1] and also in subsequent work [2]. However, we found that popular implementations of word2vec with negative sampling such as word2vec and gensim do not implement the CBOW update correctly, thus potentially leading to misconceptions about the performance of CBOW embeddings when trained correctly.
The upshot is that they get similar results with CBOW while training three times faster than skipgram.
Given the popularity of Transformers, and that Fasttext exists, I'm curious as to what inspired them to even try this, but it's certainly an interesting result. There's so much word vector research that relies on quirks of the word2vec implementation.
- nmfisher 6y agoI'm not surprised that industry prefers W2V over Transformers, given how heavy-duty the latter can be at inference time. It's been a few years since I looked at it, but IIRC fastText is basically just w2v with subwords, so it's also possible this negative sampling fix applies to w2v and fastText equally.
- riku_iki 6y agoOne can also use shallow transformer models if inference throughput is important.
- steve_g 6y agoTo save a few clicks, here's the paper that describes the fix and gives some comparisons with the supposedly broken implementation. https://arxiv.org/pdf/2012.15332.pdf https://arxiv.org/pdf/2012.15332.pdf
- gojomo 6y agoWhile I'm unsure of this paper/implementation's main claims without a closer reading, the Appendix D 'alias method for negative sampling' looks like it might be a nice standalone performance improvement to Gensim (& others') negative-sampling code.
- utopcell 6y agoIndeed it is a significant performance boost. I believe I was the first to suggest using the alias method for w2v. I had extended it to work in a distributed setting a few years back [1] which allowed me to scale the problem to a dataset of 1 trillion words on a lexicon of 1.4BN words (roughly, the top 40 billion web pages of Yahoo's search engine). [1] https://aaai.org/ocs/index.php/AAAI/AAAI17/paper/view/14956 https://aaai.org/ocs/index.php/AAAI/AAAI17/paper/view/14956
- jarym 6y agoWith all the eyeballs on word2vec and gensim how did this not get picked up before?
- gojomo 6y agoGensim considers the `word2vec.c` code, from the original authors of the Word2Vec paper, as canonical and seeks to match its behavior exactly, even in ways it might deviate from some interpretations of the paper. If there's an actual benefit to be had here, Gensim could add it as an option - but would likely always default to the same CBOW behavior as in `word2vec.c` (& similarly, FastText) - rather than this 'koan' variant.
- logram 6y agoApparently it did: https://github.com/RaRe-Technologies/gensim/issues/1873 https://github.com/RaRe-Technologies/gensim/issues/1873
- gojomo 6y agoWhile I still need to read this paper in detail, I'm not sure their only change is to this scaling of the update. The `koan` CBOW change has mixed effects on benchmarks, and makes their implementation no longer match the choices of the original, canonical `word2vec.c` release from the original Google authors of the word2vec paper. (Or, by my understanding, the CBOW mode of the FastText code.) So all the reasoning in that issue for why Gensim didn't want to make any change stands. Of course, if there's an alternate mode that offers proven benefits, it'd be a welcome suggestion/addition. (At this point, it's possible that simply using the `cbow_mean=0` sum-rather-than-average mode, or a different starting `alpha`, matches any claimed benefits of koan_CBOW.)
- mlthoughts2018 6y agoThe paper itself says the only change is normalizing by the context window size C.
- 6y ago