6 ms·

After reading this I'm left wondering why anyone should stop using word2vec. The article makes the point that you can produce word vectors using other technique

by tensor 9y ago

After reading this I'm left wondering why anyone should stop using word2vec. The article makes the point that you can produce word vectors using other techniques, in this case by computing probabilities of unigrams and skip-grams and running SVD. This is all well and good, but from an industry practitioner standpoint this doesn't explain why one would avoid using, or actually stop using word2vec. 1. Several known good word2vec implementations exist, the complexity of the technique doesn't really matter as you can just pick one of these and use it. 2. Pretrained word vectors produced from word2vec and newer algorithms exist for many languages. Why should someone stop using these and instead spend time implementing a simpler method that produces maybe good enough vectors? Being a simpler method isn't a reason in of itself.

ma2rten 9y ago
Agreed. I actually think the SVD algorithm as it is implemented in most software packages is more complicated than the word2vec algorithm, which is actually quite simple. Also, Glove (which is one of the newer word2vec alternatives you mentioned) works pretty similar to the approach in the blog post, but has some tweaks that make it perform better than this.
efavdb 9y ago
He's arguing that if you want to develop some custom word embeddings (trained on whatever data set you have specific to task at hand), the SVD approach is often much faster and almost as effective. If you're willing to make that trade, go with SVD is the argument. If you're willing to use a pre-built library, that's certainly fastest of all!
- danieldk 9y ago
  He's arguing that if you want to develop some custom word embeddings (trained on whatever data set you have specific to task at hand), the SVD approach is often much faster That depends on your corpus and vocabulary size. word2vec is O(n) where n is the length of the training corpus. Vanilla SVD is O(mn^2) for a co-occurence matrix size of m x n. I have been training word2vec and GloVe embeddings on 'web-scale' corpora and training time is usually not a problem. the SVD approach is often much faster and almost as effective SVD-derived embeddings does bad on some tasks, such as analogy tasks (see Levy & Goldberg, 2014). I agree with your parent poster that the author does not really provide good arguments against the use of word2vec et al. Moreover, But because of advances in our understanding of word2vec, computing word vectors now takes fifteen minutes on a single run-of-the-mill computer Which advances in our understanding? SVD on word-word co-occurrence matrices was proposed by Schütze in 1992 ;). There have been many works since then exploring various co-occurrence measures (including PMI) in combination with SVD.
  - Radim 9y ago
    Chris Moody usually delivers great blog posts; this big-worded rehash of an ancient 2014 technique (see https://rare-technologies.com/making-sense-of-word2vec/ https://rare-technologies.com/making-sense-of-word2vec/ for a more comprehensive eval of SVD vs word2vec vs gloVe) is weird. But on twitter, Chris promised a sequel post, with extra tricks and tips. So I see this as a warm up :) Looking forward to part 2.
    - mino 9y ago
      I thought exactly the same. That was a weird post from Chris, and he has iterated already many times in various talks about SQL-based SVD as a w2v replacement. And your blog post is much better :)
charlysl 9y ago
I have the impression from some comments that the term "complexity" in a machine learning context is not well understood. It doesn't mean "complicated" from a human perspective at all, how difficult it is to grok the SVD algorithm vs NN. It actually refers to the complexity of the hypothesis set, which is, intuitively, the number of independent parameters to be learned. What matters in ML is the relative complexity of the hypothesis set (usually very high in deep learning) and the data complexity (sample size). If this relative complexity is too high, overfitting is likely (ie the noise is learned too), generalization bad, and regularization or a simpler model a must.

by tensor 9y ago

ma2rten 9y ago

Agreed. I actually think the SVD algorithm as it is implemented in most software packages is more complicated than the word2vec algorithm, which is actually quite simple. Also, Glove (which is one of the newer word2vec alternatives you mentioned) works pretty similar to the approach in the blog post, but has some tweaks that make it perform better than this.

efavdb 9y ago

He's arguing that if you want to develop some custom word embeddings (trained on whatever data set you have specific to task at hand), the SVD approach is often much faster and almost as effective. If you're willing to make that trade, go with SVD is the argument. If you're willing to use a pre-built library, that's certainly fastest of all!

danieldk 9y ago
He's arguing that if you want to develop some custom word embeddings (trained on whatever data set you have specific to task at hand), the SVD approach is often much faster That depends on your corpus and vocabulary size. word2vec is O(n) where n is the length of the training corpus. Vanilla SVD is O(mn^2) for a co-occurence matrix size of m x n. I have been training word2vec and GloVe embeddings on 'web-scale' corpora and training time is usually not a problem. the SVD approach is often much faster and almost as effective SVD-derived embeddings does bad on some tasks, such as analogy tasks (see Levy & Goldberg, 2014). I agree with your parent poster that the author does not really provide good arguments against the use of word2vec et al. Moreover, But because of advances in our understanding of word2vec, computing word vectors now takes fifteen minutes on a single run-of-the-mill computer Which advances in our understanding? SVD on word-word co-occurrence matrices was proposed by Schütze in 1992 ;). There have been many works since then exploring various co-occurrence measures (including PMI) in combination with SVD.
- Radim 9y ago
  Chris Moody usually delivers great blog posts; this big-worded rehash of an ancient 2014 technique (see https://rare-technologies.com/making-sense-of-word2vec/ https://rare-technologies.com/making-sense-of-word2vec/ for a more comprehensive eval of SVD vs word2vec vs gloVe) is weird. But on twitter, Chris promised a sequel post, with extra tricks and tips. So I see this as a warm up :) Looking forward to part 2.
  - mino 9y ago
    I thought exactly the same. That was a weird post from Chris, and he has iterated already many times in various talks about SQL-based SVD as a w2v replacement. And your blog post is much better :)

charlysl 9y ago

I have the impression from some comments that the term "complexity" in a machine learning context is not well understood. It doesn't mean "complicated" from a human perspective at all, how difficult it is to grok the SVD algorithm vs NN. It actually refers to the complexity of the hypothesis set, which is, intuitively, the number of independent parameters to be learned. What matters in ML is the relative complexity of the hypothesis set (usually very high in deep learning) and the data complexity (sample size). If this relative complexity is too high, overfitting is likely (ie the noise is learned too), generalization bad, and regularization or a simpler model a must.