4 ms·

He's arguing that if you want to develop some custom word embeddings (trained on whatever data set you have specific to task at hand), the SVD approach is often

by efavdb 9y ago

He's arguing that if you want to develop some custom word embeddings (trained on whatever data set you have specific to task at hand), the SVD approach is often much faster and almost as effective. If you're willing to make that trade, go with SVD is the argument. If you're willing to use a pre-built library, that's certainly fastest of all!

danieldk 9y ago
He's arguing that if you want to develop some custom word embeddings (trained on whatever data set you have specific to task at hand), the SVD approach is often much faster That depends on your corpus and vocabulary size. word2vec is O(n) where n is the length of the training corpus. Vanilla SVD is O(mn^2) for a co-occurence matrix size of m x n. I have been training word2vec and GloVe embeddings on 'web-scale' corpora and training time is usually not a problem. the SVD approach is often much faster and almost as effective SVD-derived embeddings does bad on some tasks, such as analogy tasks (see Levy & Goldberg, 2014). I agree with your parent poster that the author does not really provide good arguments against the use of word2vec et al. Moreover, But because of advances in our understanding of word2vec, computing word vectors now takes fifteen minutes on a single run-of-the-mill computer Which advances in our understanding? SVD on word-word co-occurrence matrices was proposed by Schütze in 1992 ;). There have been many works since then exploring various co-occurrence measures (including PMI) in combination with SVD.
- Radim 9y ago
  Chris Moody usually delivers great blog posts; this big-worded rehash of an ancient 2014 technique (see https://rare-technologies.com/making-sense-of-word2vec/ https://rare-technologies.com/making-sense-of-word2vec/ for a more comprehensive eval of SVD vs word2vec vs gloVe) is weird. But on twitter, Chris promised a sequel post, with extra tricks and tips. So I see this as a warm up :) Looking forward to part 2.
  - mino 9y ago
    I thought exactly the same. That was a weird post from Chris, and he has iterated already many times in various talks about SQL-based SVD as a w2v replacement. And your blog post is much better :)

by efavdb 9y ago

danieldk 9y ago

He's arguing that if you want to develop some custom word embeddings (trained on whatever data set you have specific to task at hand), the SVD approach is often much faster That depends on your corpus and vocabulary size. word2vec is O(n) where n is the length of the training corpus. Vanilla SVD is O(mn^2) for a co-occurence matrix size of m x n. I have been training word2vec and GloVe embeddings on 'web-scale' corpora and training time is usually not a problem. the SVD approach is often much faster and almost as effective SVD-derived embeddings does bad on some tasks, such as analogy tasks (see Levy & Goldberg, 2014). I agree with your parent poster that the author does not really provide good arguments against the use of word2vec et al. Moreover, But because of advances in our understanding of word2vec, computing word vectors now takes fifteen minutes on a single run-of-the-mill computer Which advances in our understanding? SVD on word-word co-occurrence matrices was proposed by Schütze in 1992 ;). There have been many works since then exploring various co-occurrence measures (including PMI) in combination with SVD.

Radim 9y ago
Chris Moody usually delivers great blog posts; this big-worded rehash of an ancient 2014 technique (see https://rare-technologies.com/making-sense-of-word2vec/ https://rare-technologies.com/making-sense-of-word2vec/ for a more comprehensive eval of SVD vs word2vec vs gloVe) is weird. But on twitter, Chris promised a sequel post, with extra tricks and tips. So I see this as a warm up :) Looking forward to part 2.
- mino 9y ago
  I thought exactly the same. That was a weird post from Chris, and he has iterated already many times in various talks about SQL-based SVD as a w2v replacement. And your blog post is much better :)