5 ms·

This has the potential to be very very useful and it is great that FB has released them. Some potential caveats. I don't know how well Fasttext vectors perfor

by ninjin 10y ago

This has the potential to be very very useful and it is great that FB has released them. Some potential caveats. I don't know how well Fasttext vectors perform as features for downstream machine learning systems (if anyone know of work along these lines, I would be very happy to know about it), unlike word2vec [1] or GloVe [2] vectors that have been used for a few years at this point. Also, only having trained on Wikipedia gives the vectors less exposure to "real world" text, unlike say word2vec that was trained on the whole of Google News back in the day or GloVe that used Common Crawl. Still, if you need word vectors for a ton of languages this is looking like a great resource and will save you the pre-processing and computational troubles of having to produce them on your own. [1]: https://code.google.com/archive/p/word2vec/ https://code.google.com/archive/p/word2vec/ [2]: http://nlp.stanford.edu/projects/glove/ http://nlp.stanford.edu/projects/glove/

tmalsburg2 10y ago
Not an expert but I would guess that training on Wikipedia gives you broader coverage than training on just news. Wikipedia's whole point is to cover all kinds of topics while only some topics are usually considered news-worthy. I would also think that news language tends to be more formalized than Wikipedia's language simply because Wikipedia contributers are not trained and come from a more varied population. Edit: Unfortunately, it's becoming more and more common that people here express disagreement using downvotes. I guess that's a sign that HN is finally going down the drain like all similar platforms.
- ninjin 10y ago
  I agree, you don't deserve downvotes for what you said. Your concerns are legitimate, it is just that quantifying the impact of changes in data domain is difficult. My experience is that my own vectors trained only on Wikipedia has performed worse for a number of tasks, but your mileage may of course wary. Also agreeing that it is sad to see this forum head this way, we can be better than this.
versteegen 10y ago
This isn't a real downstream task, but one of the researchers at RaRe compared FastText to word2vec/gensim/skipgram word embeddings on the original testsets for the 'semantic' and 'syntactic' analogy tasks from the word2vec papers here: https://rare-technologies.com/fasttext-and-gensim-word-embeddings/ https://rare-technologies.com/fasttext-and-gensim-word-embed... The conclusion: These preliminary results seem to indicate fastText embeddings are significantly better than word2vec at encoding syntactic information. This is expected, since most syntactic analogies are morphology based, and the char n-gram approach of fastText takes such information into account. The original word2vec model seems to perform better on semantic tasks, since words in semantic analogies are unrelated to their char n-grams, and the added information from irrelevant char n-grams worsens the embeddings. Personally I think those analogy testsets are not very good, because they just test all pairs of relations between a very small number of words from very limited domains (like capital and country names). One advantage of FastText should be better learning on small amounts of data like Wikipedia.
- ninjin 10y ago
  Thank you for the link and I agree with your scepticism towards the analogy tasks. I have had a beef with intrinsic evaluation for years, but never found the time to pursue this line of research. However, there is a consensus at this point among many NLP practitioners that they are flaky at best. There is also research indicating that this is very much the case [1]. [1]: http://www.aclweb.org/anthology/W/W16/W16-2501.pdf http://www.aclweb.org/anthology/W/W16/W16-2501.pdf
- dzdt 10y ago
  It takes a certain kind of perspective for wikipedia to be called a "small amount of data." English wikipedia alone would run to about 2500 print volumes. Imagine telling an AI researcher from 1995 that that was "small".
  - versteegen 10y ago
    I admit, I was being funny by being unclear. For word-embeddings English Wikipedia is a moderate-large dataset at 58GB uncompressed (13GB compressed). But most of those other language wikis really are tiny. Welsh is just 67MB compressed, and there are plenty of languages more obscure than that on the list. The point of word2vec was to make use of as much data as possible by being as fast as possible (processing billions of words an hour) rather than clever, so it would be impressive if fastText vectors for those wikis were at all useful.

by ninjin 10y ago

tmalsburg2 10y ago

Not an expert but I would guess that training on Wikipedia gives you broader coverage than training on just news. Wikipedia's whole point is to cover all kinds of topics while only some topics are usually considered news-worthy. I would also think that news language tends to be more formalized than Wikipedia's language simply because Wikipedia contributers are not trained and come from a more varied population. Edit: Unfortunately, it's becoming more and more common that people here express disagreement using downvotes. I guess that's a sign that HN is finally going down the drain like all similar platforms.

ninjin 10y ago
I agree, you don't deserve downvotes for what you said. Your concerns are legitimate, it is just that quantifying the impact of changes in data domain is difficult. My experience is that my own vectors trained only on Wikipedia has performed worse for a number of tasks, but your mileage may of course wary. Also agreeing that it is sad to see this forum head this way, we can be better than this.

versteegen 10y ago

This isn't a real downstream task, but one of the researchers at RaRe compared FastText to word2vec/gensim/skipgram word embeddings on the original testsets for the 'semantic' and 'syntactic' analogy tasks from the word2vec papers here: https://rare-technologies.com/fasttext-and-gensim-word-embeddings/ https://rare-technologies.com/fasttext-and-gensim-word-embed... The conclusion: These preliminary results seem to indicate fastText embeddings are significantly better than word2vec at encoding syntactic information. This is expected, since most syntactic analogies are morphology based, and the char n-gram approach of fastText takes such information into account. The original word2vec model seems to perform better on semantic tasks, since words in semantic analogies are unrelated to their char n-grams, and the added information from irrelevant char n-grams worsens the embeddings. Personally I think those analogy testsets are not very good, because they just test all pairs of relations between a very small number of words from very limited domains (like capital and country names). One advantage of FastText should be better learning on small amounts of data like Wikipedia.

ninjin 10y ago
Thank you for the link and I agree with your scepticism towards the analogy tasks. I have had a beef with intrinsic evaluation for years, but never found the time to pursue this line of research. However, there is a consensus at this point among many NLP practitioners that they are flaky at best. There is also research indicating that this is very much the case [1]. [1]: http://www.aclweb.org/anthology/W/W16/W16-2501.pdf http://www.aclweb.org/anthology/W/W16/W16-2501.pdf
dzdt 10y ago
It takes a certain kind of perspective for wikipedia to be called a "small amount of data." English wikipedia alone would run to about 2500 print volumes. Imagine telling an AI researcher from 1995 that that was "small".
- versteegen 10y ago
  I admit, I was being funny by being unclear. For word-embeddings English Wikipedia is a moderate-large dataset at 58GB uncompressed (13GB compressed). But most of those other language wikis really are tiny. Welsh is just 67MB compressed, and there are plenty of languages more obscure than that on the list. The point of word2vec was to make use of as much data as possible by being as fast as possible (processing billions of words an hour) rather than clever, so it would be impressive if fastText vectors for those wikis were at all useful.