3 ms·

This isn't a real downstream task, but one of the researchers at RaRe compared FastText to word2vec/gensim/skipgram word embeddings on the original testsets for

This isn't a real downstream task, but one of the researchers at RaRe compared FastText to word2vec/gensim/skipgram word embeddings on the original testsets for the 'semantic' and 'syntactic' analogy tasks from the word2vec papers here: https://rare-technologies.com/fasttext-and-gensim-word-embeddings/ https://rare-technologies.com/fasttext-and-gensim-word-embed... The conclusion: These preliminary results seem to indicate fastText embeddings are significantly better than word2vec at encoding syntactic information. This is expected, since most syntactic analogies are morphology based, and the char n-gram approach of fastText takes such information into account. The original word2vec model seems to perform better on semantic tasks, since words in semantic analogies are unrelated to their char n-grams, and the added information from irrelevant char n-grams worsens the embeddings. Personally I think those analogy testsets are not very good, because they just test all pairs of relations between a very small number of words from very limited domains (like capital and country names). One advantage of FastText should be better learning on small amounts of data like Wikipedia.

ninjin 10y ago
Thank you for the link and I agree with your scepticism towards the analogy tasks. I have had a beef with intrinsic evaluation for years, but never found the time to pursue this line of research. However, there is a consensus at this point among many NLP practitioners that they are flaky at best. There is also research indicating that this is very much the case [1]. [1]: http://www.aclweb.org/anthology/W/W16/W16-2501.pdf http://www.aclweb.org/anthology/W/W16/W16-2501.pdf
dzdt 10y ago
It takes a certain kind of perspective for wikipedia to be called a "small amount of data." English wikipedia alone would run to about 2500 print volumes. Imagine telling an AI researcher from 1995 that that was "small".
- versteegen 10y ago
  I admit, I was being funny by being unclear. For word-embeddings English Wikipedia is a moderate-large dataset at 58GB uncompressed (13GB compressed). But most of those other language wikis really are tiny. Welsh is just 67MB compressed, and there are plenty of languages more obscure than that on the list. The point of word2vec was to make use of as much data as possible by being as fast as possible (processing billions of words an hour) rather than clever, so it would be impressive if fastText vectors for those wikis were at all useful.

ninjin 10y ago

Thank you for the link and I agree with your scepticism towards the analogy tasks. I have had a beef with intrinsic evaluation for years, but never found the time to pursue this line of research. However, there is a consensus at this point among many NLP practitioners that they are flaky at best. There is also research indicating that this is very much the case [1]. [1]: http://www.aclweb.org/anthology/W/W16/W16-2501.pdf http://www.aclweb.org/anthology/W/W16/W16-2501.pdf

dzdt 10y ago

It takes a certain kind of perspective for wikipedia to be called a "small amount of data." English wikipedia alone would run to about 2500 print volumes. Imagine telling an AI researcher from 1995 that that was "small".

versteegen 10y ago
I admit, I was being funny by being unclear. For word-embeddings English Wikipedia is a moderate-large dataset at 58GB uncompressed (13GB compressed). But most of those other language wikis really are tiny. Welsh is just 67MB compressed, and there are plenty of languages more obscure than that on the list. The point of word2vec was to make use of as much data as possible by being as fast as possible (processing billions of words an hour) rather than clever, so it would be impressive if fastText vectors for those wikis were at all useful.