5 ms·
Word2Vec and bag-of-words/tf-idf are somewhat obsolete in 2018 for modeling. For classification tasks, fasttext (https://github.com/facebookresearch/fastText ht
by minimaxir 9y ago
Word2Vec and bag-of-words/tf-idf are somewhat obsolete in 2018 for modeling. For classification tasks, fasttext (https://github.com/facebookresearch/fastText https://github.com/facebookresearch/fastText) performs better and faster.
Fasttext is also available in the popular NLP Python library gensim, with a good demo notebook: https://radimrehurek.com/gensim/models/fasttext.html https://radimrehurek.com/gensim/models/fasttext.html
And of course, if you have a GPU, recurrent neural networks (or other deep learning architectures) are the endgame for the remaining 10% of problems (a good example is SpaCy's DL implementation: https://spacy.io/ https://spacy.io/). Or use those libraries to incorporate fasttext for text encoding, which has worked well in my use cases.
- taneq 9y agoI'm not an expert but I'd be surprised if you could declare any approach "endgame" at this point.
- sundarurfriend 9y agoMy interpretation is that they meant something like "last resort" there.
- tanilama 9y agoFasttext is not that different from word2vec
- crossroads091 9y agoFasttext's ability to handle OOVs, however naïve, cannot be understated. I'm not sure about the literature on this, but I think generalizing based on character-gram wordvectors is probably better than assigning a dropout based common OOV vector.
- tanilama 9y agoEvolution for sure, heck, they even has the same author. But fasttext is the same idea as word2vec, the objective is identical.
- Xeoncross 9y agoReading the guide for fasttext I am not sure how a non NLP researcher would use it. "This library can also be used to train supervised text classifiers" - https://github.com/facebookresearch/fastText#text-classification https://github.com/facebookresearch/fastText#text-classifica... Looks like you can use it for gensim. https://radimrehurek.com/gensim/models/fasttext.html https://radimrehurek.com/gensim/models/fasttext.html
- sjg007 9y agoYou could probably throw DNA, RNA or protein sequences at it.
- Radim 9y agoA lot of people do. Here are the academic articles that cite Gensim and contain the word "protein" (4 pages of them): https://scholar.google.com/scholar?hl=en&as_sdt=2005&cites=13481712870568031474&scipsc=1&q=protein&btnG= https://scholar.google.com/scholar?hl=en&as_sdt=2005&cites=1...
- losteric 9y agoWhen you say GPU + RNNs address the remaining 10% of problems... what problems fall under that 10%?
- minimaxir 9y agoIn my case, problems where text is not the only input into the model (but the text is a highly-important feature), the text document is only a couple characters/words (or empty), and/or regressions instead of classifications. (although most of those are not specific to text encoding, but are difficult to implement and get good results without tools like Keras regardless)
- ilyaeck 9y agoFirst, FastText is very similar to (a superset of) Word2Vec. Secondly. RNNs are far from being endgame for anything. In fact, recent research has been moving away from them towards more predictable and parallelizable methods (see, e.g., Google's Transformers, aka "Attention is all you need", or FAIR's approach to translation using CNNs).
- minimaxir 9y agoAdded a note about other DL architectures.
- artpar 9y agoI built a docker image exposing spacy.io (with their largest en model) as rest endpoints http://github.com/artpar/languagecrunch http://github.com/artpar/languagecrunch
- e_ameisen 9y agoAuthor here. One of the reasons to start with something even simpler than embeddings such as fastText/Word2Vec, is their ease of explainability. As this blog post alludes to, the majority of ML work is data work, and in order to efficiently work on your data, understanding what your model does right/wrong is critical. This is why we favor starting with approaches that might seem simple or outdated. Once you see how these approaches fail, you can better inform your choice of a complex model.
- KennyCason 9y agoHaving worked on modeling data and sharing results with customers, I agree simpler solutions can be better for ease of explaining alone. Additionally, the loss of x percent accuracy with lower-tech solutions can sometimes be worth it because they are easier to train and reason with. This is particularly the case when you are just looking for loose directional indicators and correlations. If simple solution gets 77% accuracy, and complex solution yields 80%, I would wager that most of the time you should just stick with the simple solution. One specific example that comes to mind is in sentiment analysis where you can achieve "sufficiently" high accuracy with well tailored Bayesian approaches. They are super fast to train and reason with. If a customer/consumer wants to know exactly why a piece of text is positive or negative, the n-gram probability matrix is extremely easy to inspect. Subsequent re-training and fixing is also much easier than re-training a large neural network, svm, etc.
- kk58 9y agoCan you explain how you reduced sentences to 2 dimensional entities using PCA ?
- mooneater 9y agoYou may be painting with too broad a brush. I've worked with a number of NLP people to try to beat our tfidf model using deep methods and various word vectors. The tfidf still performs better. I think the reasons are: many out-of-vocab words, and small amounts of labelled data.
- PeterisP 9y agoComing a bit late to the discussion, but out-of-vocab words and small amounts of labelled data is the exact kind of problem that word embedding vectors are meant to solve, and they do so exceedingly well. Of course, you need a good, large source of (unlabelled) data, preferably approaching the gold standard of "everything that has ever been written about your problem domain" to train the embeddings, and that takes some effort, but then the out-of-vocab words must be exceedingly rare (i.e. only if it's a genuinely new word that has never ever been mentioned before, or a very very rare term that you've discarded), and even that can be solved well with character-based embeddings.
- opportune 9y agoone of fasttext's main features is a pre-trained word2vec implementation, you can hardly call word2vec obsolete. Bag of words has been obsolete for a long time of course. Yeah, fasttext/spacy/gensim are some of the biggest open source NLP libraries these days. However I've found that they aren't very performant - I've done prototyping in them before but wouldn't use them for a finished product. I'm sure most big tech companies use custom implementations to do their heavy lifting
- Radim 9y agoIt's hard to quantify "most", but I can attest that some of the largest companies in the world do in fact use Gensim, in part because of its good performance and robust battle-tested implementations.
- opportune 9y agoI'm honored that you would respond to my comment! I meant that in no way as a dig against Gensim. Really I'm referring more to the pythonic environment itself - the larger an NLP application grows within Python, the more overhead baggage it picks up (due to logic written in Python - plus Cython is far from a silver bullet). That is not to mention all the difficulties involved in parallelization within a Python environment. For what it's worth, when I was at a large company recently, I did use Gensim too - I was originally referring more to the case where you take a prototype for an NLP application and ask "ok, how do we scale this across millions of users"
- make3 9y agoI think you'd be surprised to find out that for a pretty large amount of classification problems, tf idf and Bayes naive classifiers do equally as good or almost as good but with less variance than deep nets. also afaik fastext is just a faster way to build embeddings that are mostly equivalent to word2vec All and all, I'm not sure you're enough of an expert to be commenting on this
- nl 9y ago1) No, they don’t. Eg, on the current Toxic comment classification TF-IDF and Bayes models are about 20% worse than RNN models. 2) FastText is also a classification algorithm, a command line classifier and a set of pretrained embeddings. 3) Yes, he does. I’ve learnt from his notebooks before, and I am confident I’m qualified to judge. I don’t completely agree with his statement here, but he is well qualified to comment.
- make3 9y ago1) I'm saying, depends on the size of the datasets
- nl 9y agoThe Gensim version of FastText doesn’t support subword embeddings without additional support. FastText itself (the command line program and the pretrained embeddings together) is a decent starting point for classification though.
- thomasahle 9y agoThe latest gensim does support fasttext .bin files and subwords (certainly using it with out of vocabulary words right now.) It doesn't support fasttext classification though.
- nl 9y agoOh that's awesome! I've been pre-processing and generating separate embedding files.
- pilooch 9y agoFastext is faster but does not beat vdcnn and other deep architectures. It all depends on the use case really.
- fergie 9y agoBag-of-words/tf-idf is still a pretty good goto for working out "significant terms" to solve "search" tasks. Its simple, old, and reasonably effective, which is what makes it good.