9 ms·

Understanding word vectors

354 points by jxub 9y ago

ZeroCool2u 9y ago
This is a highly localized problem, but I've wanted to read the past couple articles by Allison and cannot, because my org has to block gists. Allison we know this is a lot of work for a very small group but, if you see this, a couple of us here would be super stoked if you could mirror your articles somewhere else as well!
- wyldfire 9y ago
  Boy, that's a shame for them to do that. But I've been in similar places before. Do they also block reg'lar github.com?
  - ZeroCool2u 9y ago
    Yeah, but we're part of a very small subset of places that genuinely need this level of security. I used to do counter-threat infosec here before I moved to data science, so I know how much work goes into our security. Regular Github is free and clear, so we're good there.
- titanomachy 9y ago
  They should block stack overflow as well to make absolutely certain nobody can do any work. What's it like being a software dev in North Korea?
  - ZeroCool2u 9y ago
    The cabbage is subpar.
wenc 9y ago
The cognoscenti already know this, but word vectors are a game changer in the multi-class text classification game. Finding the correct representation of text makes the classification task so much easier. For my problem (1000+ total classes, 1 class per input), I experimented with Naive Bayes + TFIDF (~50% accuracy, < 1 sec training), then Word2vec + CNN model on GPU (~70% accuracy, 6 hrs training), and finally FastText (99% accuracy, 10 minutes training). FastText [0] in particular is quite impressive. It is essentially a variant of Word2Vec that also supports n-grams, and there is a default implementation in C++ with a built-in classifier that runs on the command-line (no need to setup Tensorflow or anything like that). Despite it only running on plain CPUs and only supporting a linear classifier, it seems to beat GPU-trained Word2Vec CNN models in both accuracy and speed in my use cases. I later discovered this paper from the authors comparing CNNs (and other algorithms) to FastText, and their results track my experiences [1]. This goes to show that while GPU-accelerated models are cool, sometimes using a simpler, more suitable model can have a significantly better pay-off. [0] https://fasttext.cc/docs/en/supervised-tutorial.html https://fasttext.cc/docs/en/supervised-tutorial.html [1] https://arxiv.org/abs/1607.01759 https://arxiv.org/abs/1607.01759
- ppod 9y ago
  If the word embedding is, say, 400 dimensions, are you averaging the vectors of all the words in the document to get a document vector?
  - isoprophlex 9y ago
    Just piping in to say that I'm very curious as well! How do you go from word vectors to a description of the document?
    - useracd 9y ago
      Often the w2v embedding layer will be the first layer of a network, through which a document of word representations will be passed. The output of the embedding layer will be a 2 dimensional tensor with the embedding dimension in one direction and the number of input words in the other. The next step is often to apply zero or more convolutional units over the word direction before some kind of pooling to get a 1 dimensional output. The output of the pooling layer will be a document representation based upon the word vectors. Other approaches like doc2vec are also used.
      - ppod 9y ago
        Thank you for this very helpful information, and I have just one more question: "Often the w2v embedding layer will be the first layer of a network, through which a document of word representations will be passed. The output of the embedding layer will be a 2 dimensional tensor with the embedding dimension in one direction and the number of input words in the other. The next step is often to apply zero or more convolutional units over the word direction before some kind of pooling to get a 1 dimensional output. The output of the pooling layer will be a document representation based upon the word vectors. Other approaches like doc2vec are also used." I thought the approach you described was doc2vec. If not, then does it have a name/citation?
        aglionby 9y ago
melzarei 9y ago
For anyone interested in more explanation, Stanford CS224N course has 2 great lectures on them.
h66p6qb84p4a1 9y ago
b6u2kw15
ovi256 9y ago
A few remarks: In the "Doing bad digital humanities with color vectors", if you consider colors as 3D vectors, which they do, you'll see that summing enough uniformly sampled vectors always gives medium browns, because that's the color in the middle of the colorspace. Instead, you should model colors in a polar space and sum vectors in that space. This will prevent going inside the sphere and losing color saturation. It's explained quite well here in the Interpolation section: http://www.inference.vc/high-dimensional-gaussian-distributions-are-soap-bubble/ http://www.inference.vc/high-dimensional-gaussian-distributi... If you want to understand contemporary use of words embeddings in ML, a nice simple model is explained, with full code, here: https://blog.keras.io/using-pre-trained-word-embeddings-in-a-keras-model.html https://blog.keras.io/using-pre-trained-word-embeddings-in-a... The original model comes from Kim 2014, https://arxiv.org/abs/1408.5882 https://arxiv.org/abs/1408.5882 It's a very neat use of CNNs for language processing, instead of the more popular RNNs/LSTMs. CNNs have the advantage of training much faster.
- Ethcad 9y ago
  Sorry if this is a dumb question, but wouldn’t it be a medium gray in RGB? Or are you talking about HSV?
  - Maybestring 9y ago
    HSV is a polar model. (Holding constant Value)
- wodenokoto 9y ago
  Are rnn that common for wordvectors? I mostly see word2vec and fasttext, neither of which are CNN nor rnn
  - aglionby 9y ago
    Yes, I've not looked at fasttext but word2vec is a simple 1 hidden-layer network to learn word embeddings, which can then be used as pre-trained word embeddings in other tasks. CNNs and RNNs would be used in the next stage of the pipeline for whatever your task is (machine translation etc), probably as some way of combining the word vectors. RNNs are especially nice as they can deal with variable length sentences. Note also that it's possible for systems to learn their own systems as part of training, rather than using pre-trained ones from word2vec etc.
hoerzu 9y ago
Where is T-SNE? ;)
- jxub 9y ago
  For anyone who had to google it like me, T-SNE is T-Distributed Stochastic Neighbor Embedding, and the way I understand it is that it's essentially a statistical way to reduce data dimensionality while still preserving its structure.
- b_tterc_p 9y ago
  Dimension reduction technique. For word vectors, the usual use case is to take the 400 dimension vector and turn it into a 2 dimension vector that you can use to plot like a scatterplot. Similar things will be close together. Dissimilar things will be far apart. It's kind of like principal components analysis but quirkier. For those who like Tsne. Check out the relatively new UMAP, which seems to be faster and better.
radarsat1 9y ago
Gem in here is that I learned about Annoy. https://pypi.python.org/pypi/annoy https://pypi.python.org/pypi/annoy
- Radim 9y ago
  You might be interested to know that Annoy is also integrated into Gensim, which allows you to train, use and query word embeddings on your own data. Gensim implements fast Doc2Vec and FastText too, which are a bit newer embedding techniques [0] :-) [0] https://twitter.com/gensim_py/status/969222857246101504 https://twitter.com/gensim_py/status/969222857246101504
- patelajay285 9y ago
  We've recently created an open source library that can help you get started with pre-trained word vectors and Annoy quickly: https://github.com/plasticityai/magnitude https://github.com/plasticityai/magnitude
  - Radim 9y ago
    Interesting effort :-) Unfortunately, your comparison table there is somewhere between misleading and insulting. Almost all of the "unique" features listed there are in fact a standard part of Gensim. Fast approximative queries (using Annoy), memory mapping with lazy loading, ngrams features, format convertors, Python interface, parallelization, pre-trained models for download… There is a way to promote cool new libraries, but this ain't it.
    - patelajay285 9y ago
      Hey Radim, Shoot me an e-mail (link in HN profile), we just created this a few days ago, so it hasn't been up long, and happy to fix any disagreements in the benchmarks. You're absolutely right about Annoy Indexing, but to be fair, I don't think it was part of Gensim when I started using it :). Gensim's a great library and Magnitude's not meant to be an attack on it (in fact we use it for our own converter) and we provide zero-training which Gensim does handle.
stared 9y ago
For an technical intro for word2vec (what's that exactly, how to train it): https://lilianweng.github.io/lil-log/2017/10/15/learning-word-embedding.html https://lilianweng.github.io/lil-log/2017/10/15/learning-wor... and its readable implementation in PyTorch: https://adoni.github.io/2017/11/08/word2vec-pytorch/ https://adoni.github.io/2017/11/08/word2vec-pytorch/. And just for getting idea why does it work, and play with examples in your browser: http://p.migdal.pl/2017/01/06/king-man-woman-queen-why.html http://p.migdal.pl/2017/01/06/king-man-woman-queen-why.html
visarga 9y ago
Word vectors are at the same time amazing because they contain a huge amount of latent information, and not good enough because they collapse a space of very high dimensionality in ~300 dimensions, so they have a limit to how much they can discriminate between close topics. I have done a lot of experiments on classifying text in thousands of topics, and sometimes they work amazingly well, other times they are really hard to use, depending on how close together are the topics I want to discriminate between. Another problem of word vectors is that any word might actually have multiple senses, while vectors are just point estimates. If we wanted to be correct, we needed first to find the right sense for each word in a phrase and only then assign the vector. There is research in "on-the-fly" word vectors that adapt to context, but it's much harder to use. A third problem of word vectors is out of vocabulary words and words with low frequency. For OOV, the usual solution is to create character or character-ngram embeddings that can be used to compute embeddings for new words. For low frequency words we usually ignore them (apply a cutoff). Then there is the problem of phrases and collocations - some words go together, such as "New York" and "give up". The meaning of the phrase is different from the sum of the meanings of the component words. In these cases we need to have lists of phrases and replace them in the original text before training the vectors, so we have proper vectors for phrases. By the way, one amazing tool that goes with word vectors is the library 'annoy' which can do similarity search in log time. So you can do approx 1000 lookups per second per CPU even if the database contains millions of vectors, pretty good. Annoy can be used to find similar articles, or music recommendations. Another remark - my preferred word vectors are computed with Doc2VecC (a variant of doc2vec with corruption). Doc2VecC seems more apt to discriminate between topics, but the secret is to feed it gigabytes of text. Playing with word vectors has taught me intuitively how it is to navigate a space of high dimensionality. It feels different than 3d-space because each point has a shortcut to other points, each point leads to hundreds of other places which might be far apart. It's like a kaleidoscope where a small change can create a very different perspective. doc2vecC: https://github.com/mchen24/iclr2017 https://github.com/mchen24/iclr2017 annoy: https://github.com/spotify/annoy https://github.com/spotify/annoy
- hiker512 9y ago
  Can you recommend an implementation or paper for handling OOV words via character level embeddings?
debt 9y ago
"Poetry is, at its core, the art of identifying and manipulating linguistic similarity." ah shit this makes me want to stand on a desk.
juanmirocks 9y ago
Love the practicality of this simple practical tutorial. The only pity is that it's in Python 2
- cup-of-tea 9y ago
  Yeah, I really don't get why they would use Python 2 for this kind of thing.
sumitgt 9y ago
My mind was blown when I found out how easy it was to get started using pre-trained Glove embedding in Keras. Took my Kaggle game up a few notches over-night. https://blog.keras.io/using-pre-trained-word-embeddings-in-a-keras-model.html https://blog.keras.io/using-pre-trained-word-embeddings-in-a...

354 points by jxub 9y ago

This is a highly localized problem, but I've wanted to read the past couple articles by Allison and cannot, because my org has to block gists. Allison we know this is a lot of work for a very small group but, if you see this, a couple of us here would be super stoked if you could mirror your articles somewhere else as well!

wyldfire 9y ago
Boy, that's a shame for them to do that. But I've been in similar places before. Do they also block reg'lar github.com?
- ZeroCool2u 9y ago
  Yeah, but we're part of a very small subset of places that genuinely need this level of security. I used to do counter-threat infosec here before I moved to data science, so I know how much work goes into our security. Regular Github is free and clear, so we're good there.
titanomachy 9y ago
They should block stack overflow as well to make absolutely certain nobody can do any work. What's it like being a software dev in North Korea?
- ZeroCool2u 9y ago
  The cabbage is subpar.

wenc 9y ago

The cognoscenti already know this, but word vectors are a game changer in the multi-class text classification game. Finding the correct representation of text makes the classification task so much easier. For my problem (1000+ total classes, 1 class per input), I experimented with Naive Bayes + TFIDF (~50% accuracy, < 1 sec training), then Word2vec + CNN model on GPU (~70% accuracy, 6 hrs training), and finally FastText (99% accuracy, 10 minutes training). FastText [0] in particular is quite impressive. It is essentially a variant of Word2Vec that also supports n-grams, and there is a default implementation in C++ with a built-in classifier that runs on the command-line (no need to setup Tensorflow or anything like that). Despite it only running on plain CPUs and only supporting a linear classifier, it seems to beat GPU-trained Word2Vec CNN models in both accuracy and speed in my use cases. I later discovered this paper from the authors comparing CNNs (and other algorithms) to FastText, and their results track my experiences [1]. This goes to show that while GPU-accelerated models are cool, sometimes using a simpler, more suitable model can have a significantly better pay-off. [0] https://fasttext.cc/docs/en/supervised-tutorial.html https://fasttext.cc/docs/en/supervised-tutorial.html [1] https://arxiv.org/abs/1607.01759 https://arxiv.org/abs/1607.01759

ppod 9y ago
If the word embedding is, say, 400 dimensions, are you averaging the vectors of all the words in the document to get a document vector?
- isoprophlex 9y ago
  Just piping in to say that I'm very curious as well! How do you go from word vectors to a description of the document?
  - useracd 9y ago
    Often the w2v embedding layer will be the first layer of a network, through which a document of word representations will be passed. The output of the embedding layer will be a 2 dimensional tensor with the embedding dimension in one direction and the number of input words in the other. The next step is often to apply zero or more convolutional units over the word direction before some kind of pooling to get a 1 dimensional output. The output of the pooling layer will be a document representation based upon the word vectors. Other approaches like doc2vec are also used.
    - ppod 9y ago
      Thank you for this very helpful information, and I have just one more question: "Often the w2v embedding layer will be the first layer of a network, through which a document of word representations will be passed. The output of the embedding layer will be a 2 dimensional tensor with the embedding dimension in one direction and the number of input words in the other. The next step is often to apply zero or more convolutional units over the word direction before some kind of pooling to get a 1 dimensional output. The output of the pooling layer will be a document representation based upon the word vectors. Other approaches like doc2vec are also used." I thought the approach you described was doc2vec. If not, then does it have a name/citation?
      - aglionby 9y ago

melzarei 9y ago

For anyone interested in more explanation, Stanford CS224N course has 2 great lectures on them.

h66p6qb84p4a1 9y ago

b6u2kw15

ovi256 9y ago

A few remarks: In the "Doing bad digital humanities with color vectors", if you consider colors as 3D vectors, which they do, you'll see that summing enough uniformly sampled vectors always gives medium browns, because that's the color in the middle of the colorspace. Instead, you should model colors in a polar space and sum vectors in that space. This will prevent going inside the sphere and losing color saturation. It's explained quite well here in the Interpolation section: http://www.inference.vc/high-dimensional-gaussian-distributions-are-soap-bubble/ http://www.inference.vc/high-dimensional-gaussian-distributi... If you want to understand contemporary use of words embeddings in ML, a nice simple model is explained, with full code, here: https://blog.keras.io/using-pre-trained-word-embeddings-in-a-keras-model.html https://blog.keras.io/using-pre-trained-word-embeddings-in-a... The original model comes from Kim 2014, https://arxiv.org/abs/1408.5882 https://arxiv.org/abs/1408.5882 It's a very neat use of CNNs for language processing, instead of the more popular RNNs/LSTMs. CNNs have the advantage of training much faster.

Ethcad 9y ago
Sorry if this is a dumb question, but wouldn’t it be a medium gray in RGB? Or are you talking about HSV?
- Maybestring 9y ago
  HSV is a polar model. (Holding constant Value)
wodenokoto 9y ago
Are rnn that common for wordvectors? I mostly see word2vec and fasttext, neither of which are CNN nor rnn
- aglionby 9y ago
  Yes, I've not looked at fasttext but word2vec is a simple 1 hidden-layer network to learn word embeddings, which can then be used as pre-trained word embeddings in other tasks. CNNs and RNNs would be used in the next stage of the pipeline for whatever your task is (machine translation etc), probably as some way of combining the word vectors. RNNs are especially nice as they can deal with variable length sentences. Note also that it's possible for systems to learn their own systems as part of training, rather than using pre-trained ones from word2vec etc.

hoerzu 9y ago

Where is T-SNE? ;)

jxub 9y ago
For anyone who had to google it like me, T-SNE is T-Distributed Stochastic Neighbor Embedding, and the way I understand it is that it's essentially a statistical way to reduce data dimensionality while still preserving its structure.
b_tterc_p 9y ago
Dimension reduction technique. For word vectors, the usual use case is to take the 400 dimension vector and turn it into a 2 dimension vector that you can use to plot like a scatterplot. Similar things will be close together. Dissimilar things will be far apart. It's kind of like principal components analysis but quirkier. For those who like Tsne. Check out the relatively new UMAP, which seems to be faster and better.

radarsat1 9y ago

Gem in here is that I learned about Annoy. https://pypi.python.org/pypi/annoy https://pypi.python.org/pypi/annoy

Radim 9y ago
You might be interested to know that Annoy is also integrated into Gensim, which allows you to train, use and query word embeddings on your own data. Gensim implements fast Doc2Vec and FastText too, which are a bit newer embedding techniques [0] :-) [0] https://twitter.com/gensim_py/status/969222857246101504 https://twitter.com/gensim_py/status/969222857246101504
patelajay285 9y ago
We've recently created an open source library that can help you get started with pre-trained word vectors and Annoy quickly: https://github.com/plasticityai/magnitude https://github.com/plasticityai/magnitude
- Radim 9y ago
  Interesting effort :-) Unfortunately, your comparison table there is somewhere between misleading and insulting. Almost all of the "unique" features listed there are in fact a standard part of Gensim. Fast approximative queries (using Annoy), memory mapping with lazy loading, ngrams features, format convertors, Python interface, parallelization, pre-trained models for download… There is a way to promote cool new libraries, but this ain't it.
  - patelajay285 9y ago
    Hey Radim, Shoot me an e-mail (link in HN profile), we just created this a few days ago, so it hasn't been up long, and happy to fix any disagreements in the benchmarks. You're absolutely right about Annoy Indexing, but to be fair, I don't think it was part of Gensim when I started using it :). Gensim's a great library and Magnitude's not meant to be an attack on it (in fact we use it for our own converter) and we provide zero-training which Gensim does handle.

stared 9y ago

For an technical intro for word2vec (what's that exactly, how to train it): https://lilianweng.github.io/lil-log/2017/10/15/learning-word-embedding.html https://lilianweng.github.io/lil-log/2017/10/15/learning-wor... and its readable implementation in PyTorch: https://adoni.github.io/2017/11/08/word2vec-pytorch/ https://adoni.github.io/2017/11/08/word2vec-pytorch/. And just for getting idea why does it work, and play with examples in your browser: http://p.migdal.pl/2017/01/06/king-man-woman-queen-why.html http://p.migdal.pl/2017/01/06/king-man-woman-queen-why.html

visarga 9y ago

Word vectors are at the same time amazing because they contain a huge amount of latent information, and not good enough because they collapse a space of very high dimensionality in ~300 dimensions, so they have a limit to how much they can discriminate between close topics. I have done a lot of experiments on classifying text in thousands of topics, and sometimes they work amazingly well, other times they are really hard to use, depending on how close together are the topics I want to discriminate between. Another problem of word vectors is that any word might actually have multiple senses, while vectors are just point estimates. If we wanted to be correct, we needed first to find the right sense for each word in a phrase and only then assign the vector. There is research in "on-the-fly" word vectors that adapt to context, but it's much harder to use. A third problem of word vectors is out of vocabulary words and words with low frequency. For OOV, the usual solution is to create character or character-ngram embeddings that can be used to compute embeddings for new words. For low frequency words we usually ignore them (apply a cutoff). Then there is the problem of phrases and collocations - some words go together, such as "New York" and "give up". The meaning of the phrase is different from the sum of the meanings of the component words. In these cases we need to have lists of phrases and replace them in the original text before training the vectors, so we have proper vectors for phrases. By the way, one amazing tool that goes with word vectors is the library 'annoy' which can do similarity search in log time. So you can do approx 1000 lookups per second per CPU even if the database contains millions of vectors, pretty good. Annoy can be used to find similar articles, or music recommendations. Another remark - my preferred word vectors are computed with Doc2VecC (a variant of doc2vec with corruption). Doc2VecC seems more apt to discriminate between topics, but the secret is to feed it gigabytes of text. Playing with word vectors has taught me intuitively how it is to navigate a space of high dimensionality. It feels different than 3d-space because each point has a shortcut to other points, each point leads to hundreds of other places which might be far apart. It's like a kaleidoscope where a small change can create a very different perspective. doc2vecC: https://github.com/mchen24/iclr2017 https://github.com/mchen24/iclr2017 annoy: https://github.com/spotify/annoy https://github.com/spotify/annoy

hiker512 9y ago
Can you recommend an implementation or paper for handling OOV words via character level embeddings?

debt 9y ago

"Poetry is, at its core, the art of identifying and manipulating linguistic similarity." ah shit this makes me want to stand on a desk.

juanmirocks 9y ago

Love the practicality of this simple practical tutorial. The only pity is that it's in Python 2

cup-of-tea 9y ago
Yeah, I really don't get why they would use Python 2 for this kind of thing.

sumitgt 9y ago

My mind was blown when I found out how easy it was to get started using pre-trained Glove embedding in Keras. Took my Kaggle game up a few notches over-night. https://blog.keras.io/using-pre-trained-word-embeddings-in-a-keras-model.html https://blog.keras.io/using-pre-trained-word-embeddings-in-a...