3 ms·
Look, let's talk about replacing word2vec because it's old and not that good on its own. Everyone making word vectors, including this article, compares them to
by rspeer 9y ago
Look, let's talk about replacing word2vec because it's old and not that good on its own. Everyone making word vectors, including this article, compares them to word2vec as a baseline because beating word2vec on evaluations is so easy. It's from 2013 and machine learning moves fast these days.
You can replace pre-trained word2vec in 12 languages (with aligned vectors!) with ConceptNet Numberbatch [1]. You can be sure it's better because of the SemEval 2017 results where it came out on top in 4 out of 5 languages and 15 out of 15 aligned language pairs [2]. (You will not find word2vec in this evaluation because it would have done poorly.)
If you want to bring your own corpus, at least update your training method to something like fastText [3], though I recommend looking at how to improve it using ConceptNet anyway, because distributional semantics alone will not get you to the state of the art.
Also: what pre-built word2vec are you using that actually contains valid word associations in many languages? Something trained on just Wikipedia? Al-Rfou's Polyglot? Have you ever actually tested it?
[1] https://github.com/commonsense/conceptnet-numberbatch https://github.com/commonsense/conceptnet-numberbatch
[2] http://nlp.arizona.edu/SemEval-2017/pdf/SemEval002.pdf http://nlp.arizona.edu/SemEval-2017/pdf/SemEval002.pdf
[3] https://fasttext.cc/ https://fasttext.cc/
- rpedela 9y agoFor pre-trained word2vec and fastText vectors, I think kuschku is talking about https://github.com/Kyubyong/wordvectors https://github.com/Kyubyong/wordvectors
- rspeer 9y agoOne thing that pre-trained word2vec and GloVe have going for them (and which is incorporated in ConceptNet Numberbatch) is that they're trained on broader corpora, Google News and the Common Crawl respectively. Of course that's what limits them to English, if you're not willing to use a knowledge graph to align things across languages. Training on just Wikipedia is not a representative model of any language, because not all text is written like an encyclopedia.
- rpedela 9y agoOf course the broader the corpora the better in general (sometimes more narrow is better), but Wikipedia is certainly better than nothing.
- kuschku 9y agoIf you read my comment carefully, I’m referring to "word2vec and co", meaning similar projects from other sources that use the same approach, and for which similarly large pre-trained corpi exist. My general criticism is that it’s easy to invent a new method, or popularize an existing method that’s superior. But that’s not the hard task. Building the algorithm is pretty easy. Getting a corpus of colloquial, professional, and slang texts in dozens of languages, and enough computational power to actually train them all, is usually the hard part for us doing independent research, or working on open source projects without major corporate sponsors. The reason we use word2vec and similar solutions is not because of the approach it uses, but just because it provides a well-working complete package.
- nl 9y agoHey, we tried (your) numberbatch vectors (great work, BTW) for a multilingual task, and we had issues because the "words" are conceptnet node names (eg http://conceptnet.io/c/en/natural_language http://conceptnet.io/c/en/natural_language). I understand why it is like this, but is there an approach others are using if they want to use it more like traditional Word2Vec vectors?
- rspeer 9y agoDon't know if you tried it on 16.09 when this was harder, but now you can just strip off the /c/en/ and the rest has exactly the same normalization as word2vec. Even including the weird thing with changing multiple digits to # signs. There is also a Python function included that does the normalization. But it's pretty simple and if you're already using word2vec you're already doing it.
- nl 9y agoReally? I'll check what version we were using, but the problem I'm referring to was vectors for concepts with the underscore in them. We did work out removing /c/<lang>/
- gargarplex 9y agoI setup fasttext to run in a serverless environment (AWS Lambda) today: https://blog.codefor.cash/2017/10/19/an-aws-lambda-serverless-implementation-of-facebook-fasttext-text-classification/ https://blog.codefor.cash/2017/10/19/an-aws-lambda-serverles...