7 ms·
Learning Word Vectors for 157 Languages
- alexott 9y agoAnd corresponding word vectors: https://fasttext.cc/docs/en/crawl-vectors.html https://fasttext.cc/docs/en/crawl-vectors.html
- rspeer 9y agoAre these the right vectors? The filenames correspond to the fastText vectors I've already tried, which are only in English. EDIT: Indeed, this is old data from a previous publication. It appears they have not actually made the new data public yet.
- rspeer 9y agoUpdate: that link points to the right page now.
- sp332 9y agoI guess I can't argue with the results, but the data doesn't look very clean. I downloaded the Esperanto file because I expected it to be small, but it was a 1.1 GB download that expanded to 4.4 GB. A lot of the most popular "words" are punctuation marks, and the first two real words are "la" and "La". Perhaps naively, I expected those to be the same word.
- alexott 9y agoIn some tasks punctuation marks may play significant role, but I agree that for things like text classification, they may not be necessary...
- posterboy 9y agoyes, me too. I like cooking dogs and my family by he way.
- gojomo 9y agoDoes a word-vector dataset that "looks clean" perform better on real applications, justifying the extra attention required for more preprocessing/cleanup?
- deleted 9y ago[deleted]
- exgrv 9y agoWe decided to keep the casing, as it is useful for some applications such as named entity recognition. Regarding the punctuation, as pointed out in another comment, these tokens might also be useful for some applications (and they are easy to filter out if you don't need them).
- sp332 9y agoIn the Tagalog file, } is near the top but { is over 8,000 lines down. Is there a reason they have such different frequencies? ( and ) are right next to each other. And yes I realize this is a really odd question :)
- jauntbox 9y agoYour comment has twice as many ) as it does ( My first guess would be emojis ;)
- exgrv 9y agoThis is probably due to our preprocessing of Wikipedia that did not get rid of all the '}' from the markup.
- sp332 9y agoOh true. I tried to clean up Wiki markup for ML years ago and it was a huge pain. Next time I think I'll parse the HTML version and pull out the text from the tags explicitly.
- mkl 9y agoThis is a much better way to do it. It's easier, cleaner, and gets the text which is generated by templates, which there is a surprising amount of (you get weird artifacts from that otherwise).
- neves 9y agoNewbie here. Can someone explain what would I use these vectors for?
- godelmachine 9y agoAt one level, it’s simply a vector of weights. In a simple 1-of-N (or ‘one-hot’) encoding every element in the vector is associated with a word in the vocabulary. The encoding of a given word is simply the vector in which the corresponding element is set to one, and all other elements are zero. It's all about Natural Language Processing. If you are interested in more, check out these excellent reviews by Adrian Colyer posted in The Morning Paper. https://blog.acolyer.org/2016/04/21/the-amazing-power-of-word-vectors/ https://blog.acolyer.org/2016/04/21/the-amazing-power-of-wor...
- bllguo 9y agothat's a nice link but the excerpt you quote is kind of misleading; the word vectors in this case are not one-hot encodings. They are learned, continuous representations. But one-hot representations are also a kind of word vector. word vectors are vector representations of each word in the vocabulary. Here they are learned by a neural net. the length of the vector is the # of features. Just for intuition, one feature of a word the NN could learn is the gender of a word, and so on.
- v1n337 9y agoYes, quite correct! The vectors are dense representations. But the features aren't individually interpretable, in practice. For instance, the 'gender' of a word may have it's signal scattered over several features/dimensions of the learned vector.
- Choco31415 9y agoTo help with intuition, here are a few example word vectors that we might encounter: "fruit": {food: 0.99, gender: -0.05, size: 0.2} "king": {food: -0.9, gender: 0.92, size: 0.56} Building off of what v1n337 stated, though, axis can easily be skewed and rotated such that they're still interpretable, just not obviously so.
- gojomo 9y agoOnline-reader-friendly (no 2-column PDF!) paper link: https://www.arxiv-vanity.com/papers/1802.06893/ https://www.arxiv-vanity.com/papers/1802.06893/
- Choco31415 9y agoThere seems to be some interest in learning what word vectors are, so I'll give a basic overview here, then tie it into the research. "One of the challenges for Natural Language Processing (NLP) systems is the question of how to represent input such that the network runs quickly, but also learns well. It's possible to represent each word as a one hot vector, but that's computationally slow. It's also possible to represent each word as some number, but then lots of words look very similar. Instead, why not use a mix? Introducing word embeddings. We'll represent each word as a n-dimensional vector, with each dimension representing a trait about the word. For example, "fruit" might be represented as {food: 0.99, gender: -0.05, size: 0.2}, and "king" might be represented as {food: -0.9, gender: 0.92, size: 0.56}." [Quoted from MuffinTech.org] [See v1n337 for caveats. [0]] Two similar words should have similar word vectors, like "apple" and "peach". If we learn some fact about apples, like "Humans eat apples.", then we can easily generalize that to peaches, pineapples, etc... Let's tie this back to the research. Since we have word vectors for many languages now, that makes it easier for us to build NLP systems in other languages. For example, if we wanted to build an English->French translator. [0] https://news.ycombinator.com/item?id=16448960 https://news.ycombinator.com/item?id=16448960
- itronitron 9y agofor the uninitiated, an 'n-dimensional vector' is an array of numbers
- gigogkggi 9y agoIf Apple and peach had very similar word vectors, an English apple and a French peach would have the same too. And there is a risk of mistranslatuon. How is that situation handled?
- yorwba 9y agoIt is handled by supervised training with paired translations, so that English apple will be associated with French pomme instead of other fruits. If you don't have a parallel corpus, translation gets significantly harder. I'm actually more amazed that it's possible at all.
- visarga 9y agoThis is great for quick jobs but if you have a good dataset that is in your domain you should retrain using fasttext, doc2vecC, word2vec or starspace to name but a few tools that are excellent. The window size is important as well, depending on your downstream task - one size doesn't fit all. It only takes a few hours on a beefy CPU with enough RAM for a corpus with a size of billions of words. Word vectors are fascinating representations. There is a huge amount of information and nuance captured in them. You can use them directly for topic retrieval (using annoy or another optimised vector index), or feed them into a classifier such as those in the Sklearn library. All types of neural nets: fully connected, recurrent and convolutional can be applied on word vectors.
- rspeer 9y agoI argue that you need both domain-general and domain-specific data. Most domains you could apply NLP to don't have billions of words to train on, especially if the language of the data isn't English. Pre-trained data is valuable, and you really aren't going to re-learn it all from your data, so why throw it out?
- laughingman2 9y agoYes, recent work on transfer learning https://arxiv.org/abs/1801.06146 https://arxiv.org/abs/1801.06146 " Fine-tuned Language Models for Text Classification" suggests that training a lstm on a large corpora for language modeling, then using it as sentence encoder after tuning it in specific dataset gives a considerable perfomance. This is covered in fast.ai 's new course 1.
- closed 9y agoI'm always amazed that, although Cantonese is about as commonly spoken as Italian, so little written corpus data exists for it (since most Cantonese write in a form of Mandarin Chinese). For example, here there are all kinds of useful things we can do with these 157 sets of word vectors, but Cantonese escaped the list because most of its transactions happen off the page. https://en.wikipedia.org/wiki/Written_Cantonese https://en.wikipedia.org/wiki/Written_Cantonese
- slx26 9y agoIf you are interested in multi-language text analysis, you might be interested in Freeling [1], a full-fledged, open source library for language analysis written in C++ (which also happens to include a simple interface for working with word vectors). [1] http://nlp.lsi.upc.edu/freeling/node/1 http://nlp.lsi.upc.edu/freeling/node/1