5 ms·

Hey juxtaposicion, fascinating work. I have many questions so I am just going to shoot them rapid fire. What is the dimensionality of each word vector and what

Hey juxtaposicion, fascinating work. I have many questions so I am just going to shoot them rapid fire. What is the dimensionality of each word vector and what does a words position in this space "mean"? What is this dimensionality determined by? Have your tried any dimensionality reduction algorithms like PCA or Isomap? It would be interesting to find the word vectors that contain the most variation across all of wikipedia. Have you tried any other nearest neighbor search methods other than a simple dot product, such as locality sensitive hashing? I guess most of those questions are about the word2vec algorithm, but you are probably in a good place to answer them after working with it. Anyways, cool work, and I am glad you did it in python so I can really dig in and understand it.

dhammack 13y ago
I'm not associated with the OP, but I have played with word2vec. Word2vec can be viewed as dimensionality reduction. In some sense, the dimensionality of words is the same as the vocabulary size, if a one-hot encoding is used (which is most common). For word2vec specifically, the output vector dimension is a parameter specified during training, so you can choose it however you want. I suspect the OP chose something fairly large (>256) because of the performance boost. Nearest neighbor methods work well with word2vec words: https://code.google.com/p/word2vec/#Word_clustering https://code.google.com/p/word2vec/#Word_clustering The paper associated with word2vec is fairly easy to read and understand, even if you don't have a background in NLP or neural networks: http://arxiv.org/pdf/1301.3781.pdf http://arxiv.org/pdf/1301.3781.pdf
juxtaposicion 13y ago
Hey doctoboggan, awesome questions! >> What is the dimensionality of each word vector and what does a words position in this space "mean"? What is this dimensionality determined by? Each dimension roughly is a new way that words can be similar or dissimilar. So I've got 1000-dimensional vectors, so words can be similar or dissimilar in only one thousand 'ways'. So associations like 'luxury', 'thoughtful', 'person', 'place' or 'object' are learned (roughly speaking). Of course, real words are far more diverse, so this is an approximation. The 1000 dimensions is configurable, and in theory more dimensions means more contrast is captured, but you need more training data. In practice, the number 1000 is chosen because that maxes out the size of my large memory machine. That said, the word2vec paper shows good results with 1000D, so it doesn't seem to be a bad choice. >> Have your tried any dimensionality reduction algorithms like PCA or Isomap? Yes! I've tried out PCA, and some spectral biclustering using the off-the-shelf algorithms in SciKits Learn. I only played around with this for an hour or so but got discouraging results. Nevertheless, the word2vec papers actually show that this works really well for projecting France, USA, Paris, DC, London, etc. on a two-dimensional plane where the axis roughly correspond to countried & capitals -- exactly what you'd hope for! I wasn't able to replicate that, but Tomas Mikolov was! >> It would be interesting to find the word vectors that contain the most variation across all of wikipedia. Hmm, interesting indeed! I'm not sure how I'd got about measuring 'variation' -- would this amount to isolating word clusters and finding the most dense ones? Something like finding a cluster with a hundred variations of the word 'snow' (if you're Inuit)? I'd be willing to part with the raw vector database if there's interest. >> Have you tried any other nearest neighbor search methods other than a simple dot product, such as locality sensitive hashing? Only a little bit, although I'm very interested in finding a faster approach than finding the whole damn dot product (see: https://news.ycombinator.com/item?id=6720359 https://news.ycombinator.com/item?id=6720359). I worry that traditional location sensitive hashes, kd-trees, and the like work well for 3D locations, but miserably for 1000D data like I have here. I should reiterate out that most of the hard work revolves around the word2vec algorithm which I used but didn't write. It's awesome, check it and the papers out here: https://code.google.com/p/word2vec/ https://code.google.com/p/word2vec/ Whoa, that was alot. Thanks!
- dhammack 13y ago
  For 2D visualization, t-sne is an excellent tool. I've used it with word2vec and you can see clusters of similar words: https://raw.github.com/dhammack/Word2VecExample/master/visualizations/figure_11.png https://raw.github.com/dhammack/Word2VecExample/master/visua... https://raw.github.com/dhammack/Word2VecExample/master/visualizations/figure_13.png https://raw.github.com/dhammack/Word2VecExample/master/visua... https://raw.github.com/dhammack/Word2VecExample/master/visualizations/figure_5.png https://raw.github.com/dhammack/Word2VecExample/master/visua... And more in https://github.com/dhammack/Word2VecExample/tree/master/visualizations https://github.com/dhammack/Word2VecExample/tree/master/visu...
- murbard2 13y ago
  Build your kd-tree on the vectors expressed in the eigenvector basis. If the eigenvalues decrease fast enough, you can get bounds on the dot product while going only a few levels deep.
- robseed 13y ago
  Don't most search engines use an inverted index to find the similarity between the query vector and the document vectors? (instead of doing the dot product with every document) Maybe something like this could help: http://radimrehurek.com/gensim/similarities/docsim.html http://radimrehurek.com/gensim/similarities/docsim.html Cool stuff, thanks.
- doctoboggan 13y ago
  I'd love access to the raw vector database if you feel like sending it my way.

dhammack 13y ago

I'm not associated with the OP, but I have played with word2vec. Word2vec can be viewed as dimensionality reduction. In some sense, the dimensionality of words is the same as the vocabulary size, if a one-hot encoding is used (which is most common). For word2vec specifically, the output vector dimension is a parameter specified during training, so you can choose it however you want. I suspect the OP chose something fairly large (>256) because of the performance boost. Nearest neighbor methods work well with word2vec words: https://code.google.com/p/word2vec/#Word_clustering https://code.google.com/p/word2vec/#Word_clustering The paper associated with word2vec is fairly easy to read and understand, even if you don't have a background in NLP or neural networks: http://arxiv.org/pdf/1301.3781.pdf http://arxiv.org/pdf/1301.3781.pdf

juxtaposicion 13y ago

Hey doctoboggan, awesome questions! >> What is the dimensionality of each word vector and what does a words position in this space "mean"? What is this dimensionality determined by? Each dimension roughly is a new way that words can be similar or dissimilar. So I've got 1000-dimensional vectors, so words can be similar or dissimilar in only one thousand 'ways'. So associations like 'luxury', 'thoughtful', 'person', 'place' or 'object' are learned (roughly speaking). Of course, real words are far more diverse, so this is an approximation. The 1000 dimensions is configurable, and in theory more dimensions means more contrast is captured, but you need more training data. In practice, the number 1000 is chosen because that maxes out the size of my large memory machine. That said, the word2vec paper shows good results with 1000D, so it doesn't seem to be a bad choice. >> Have your tried any dimensionality reduction algorithms like PCA or Isomap? Yes! I've tried out PCA, and some spectral biclustering using the off-the-shelf algorithms in SciKits Learn. I only played around with this for an hour or so but got discouraging results. Nevertheless, the word2vec papers actually show that this works really well for projecting France, USA, Paris, DC, London, etc. on a two-dimensional plane where the axis roughly correspond to countried & capitals -- exactly what you'd hope for! I wasn't able to replicate that, but Tomas Mikolov was! >> It would be interesting to find the word vectors that contain the most variation across all of wikipedia. Hmm, interesting indeed! I'm not sure how I'd got about measuring 'variation' -- would this amount to isolating word clusters and finding the most dense ones? Something like finding a cluster with a hundred variations of the word 'snow' (if you're Inuit)? I'd be willing to part with the raw vector database if there's interest. >> Have you tried any other nearest neighbor search methods other than a simple dot product, such as locality sensitive hashing? Only a little bit, although I'm very interested in finding a faster approach than finding the whole damn dot product (see: https://news.ycombinator.com/item?id=6720359 https://news.ycombinator.com/item?id=6720359). I worry that traditional location sensitive hashes, kd-trees, and the like work well for 3D locations, but miserably for 1000D data like I have here. I should reiterate out that most of the hard work revolves around the word2vec algorithm which I used but didn't write. It's awesome, check it and the papers out here: https://code.google.com/p/word2vec/ https://code.google.com/p/word2vec/ Whoa, that was alot. Thanks!

dhammack 13y ago
For 2D visualization, t-sne is an excellent tool. I've used it with word2vec and you can see clusters of similar words: https://raw.github.com/dhammack/Word2VecExample/master/visualizations/figure_11.png https://raw.github.com/dhammack/Word2VecExample/master/visua... https://raw.github.com/dhammack/Word2VecExample/master/visualizations/figure_13.png https://raw.github.com/dhammack/Word2VecExample/master/visua... https://raw.github.com/dhammack/Word2VecExample/master/visualizations/figure_5.png https://raw.github.com/dhammack/Word2VecExample/master/visua... And more in https://github.com/dhammack/Word2VecExample/tree/master/visualizations https://github.com/dhammack/Word2VecExample/tree/master/visu...
murbard2 13y ago
Build your kd-tree on the vectors expressed in the eigenvector basis. If the eigenvalues decrease fast enough, you can get bounds on the dot product while going only a few levels deep.
robseed 13y ago
Don't most search engines use an inverted index to find the similarity between the query vector and the document vectors? (instead of doing the dot product with every document) Maybe something like this could help: http://radimrehurek.com/gensim/similarities/docsim.html http://radimrehurek.com/gensim/similarities/docsim.html Cool stuff, thanks.
doctoboggan 13y ago
I'd love access to the raw vector database if you feel like sending it my way.