8 ms·
What Kagglers Are Using for Text Classification
- brian_herman__ 8y agoThis was really interesting and insightful thanks!
- PaulHoule 8y agoIt would be nice to see how these methods compare to the classical methods based on word occurrences.
- nerdponx 8y agoKaggle is a pretty serious natural-selection environment for machine learning algorithms. Basically, if bag-of-words worked better, the contest winners would still use it.
- PaulHoule 8y agoOne issue is the kind of problem. I remember getting 95%-ish accuracy with BoW and the SVM circa 2004 when it came to questions like "is this paper about astrophysics or organic chemistry?" In that case you have a distinct vocabulary for different topics and it is hard to beat BoW. Sentiment analysis, on the other hand, is where BoW goes to die since now "not good" means something very different than "good", and even simple heuristics like treating "not X" as a term that is different from "X" give limited gain because negation is expressed with constructions like "i don't believe that is good" and there is no k-word window that you reliably catch negation in since there isn't a limit on how complex sentences are. There is also the question of "is the improvement between method A and method B worth it?" For instance the Netflix prize was much celebrated because some brilliant people busted their ass to go from 92% to 95% accuracy on movie recommendations. In the end the algorithm proved to be too complex for the value it created. (eg. Who would notice that they got 8 bad recommendations instead of 5 out of a hundred? An additional half a bad recommendation out of 10?) The real "Netflix optimization problem" is how to spend as little on acquiring content as possible while motivating people to keep their subscriptions and that is something Netflix will keep closer to their chest and not promote a public competition on. (eg. if it were valuable why would they let competitors know about it?)
- nerdponx 8y agoGood point, I should have written "for machine learning algorithms on problems that the industry is currently interested in".
- radnam 8y agoIt is important to understand "worked better". In practice, I would trade off using simpler methods for few points increase in precision/recall/your choice of metric.
- minimaxir 8y agoYou're assuming that they aren't trying: it's not too hard to try out bag-of-words and see what happens. But things like Attention and LSTMs are pretty good, but not without their costs.
- nerdponx 8y agoI'm assuming the opposite: bag-of-words is the go-to baseline, but as another commenter pointed out, just separating instances based on vocabulary is not sufficient in today's sophisticated text classification problems.
- platz 8y agoThree methods and no idea how I would choose between the three of them aside randomly trying each one and measuring performance. (Not for winning kaggle but for an actual problem)
- nerdponx 8y agoSometimes you just need to rent 3 separate EC2 instances and try precisely that.
- deleted 8y ago[deleted]
- currymj 8y agothis is unfortunately almost always true, you just have to try every possible combination of everything with lots of hyperparameters. nothing makes any sense, it's total chaos, and we are wandering blind in the wastelands.
- platz 8y agominimaxir's other comment was helpful by not trying to guess an outcome but explaining things we do know about, like training/estimation cost and order of complexity, which were topics absent from the article
- platz 8y ago"The information you'd need to choose is included in there. If you're doing this professionally, you should strive to have enough of a high-level understanding of NLP to be able to make these decisions without having a rubric handed to you on a silver platter. In a nutshell, though: Strive to use the simplest model that will get the job done. Less elaborate models are easier to understand and (usually) less prone to things like overfitting, so they'll be more tractable to work with in a business context. To that end: Use a convolutional net when you can get away with a small, fixed-size context window. Use an LSTM when you need long-term memory. Attention can be expensive, so you use it when you have cause to believe you can gain a lot by giving selective attention to features, and have both a lot of training data and a lot of computing resources. It's also worth considering that you might be best off going with none of these options. Cool as deep learning is, I've personally never actually been able to justify using it in a professional setting. Simpler models such as logistic regression and decision trees have characteristics that are near-useless for getting you to the top of a Kaggle leaderboard, but can be indispensable when working on many real-world business problems" - anonymous comment reply
- wenc 8y agoI wonder how FastText (essentially word2vec + word & char n-grams + other stuff) stacks up against these algorithms. In my own tests on my own corpuses, CPU-based FastText is faster to train and produces significantly better results (precision/recall) than the GPU-bound CNN algorithms that I've tried, but have not compared it against RNN techniques.
- deleted 8y ago[deleted]
- linalgmixer 8y agoFastText is essentially a linear classifier and it's not surprising that it would train quickly. As for the prediction metrics, I imagine that will depend on the type of data and problem you're working with. Linear models perform very well on certain types of problems (I've had great success with SVMs on text classification problems personally) but for more complicated tasks, I imagine that the deep learning models would perform very, very well (relatively).
- physicsyogi 8y agoI’ve found that some CNNs consistently beat fasttext in terms of model quality. But I’ve beaten those CNNs and fasttext by doing transfer learning with ULMFit and fasts I. But if we’re talking training speed, fasttext is indeed aptly named.
- biomodel 8y agoNot sure why anyone would use 2D CNNs for processing text when there is no spatial correlation in the embedding features. Recent work such as https://arxiv.org/abs/1803.01271 https://arxiv.org/abs/1803.01271 show that for most tasks, 1D CNNs outperform recurrent architectures while being faster to train
- soraki_soladead 8y agoThis is just a bug in their code. The paper they cite uses 1D convolutions. Though, I suppose having an unused dimension only really hurts efficiency.
- gnulinux 8y ago> Though, I suppose having an unused dimension only really hurts efficiency. That might not be true as it might increase bias and thus might need a more careful hyperparameter tuning to avoid overfitting.
- madavidj 8y agoProbably because the author followed this blog: http://www.wildml.com/2015/12/implementing-a-cnn-for-text-classification-in-tensorflow/ http://www.wildml.com/2015/12/implementing-a-cnn-for-text-cl... That blog used a 2d cnn because tensorflow didn't have a 1d version at the time of writing, so he just created a dummy 2nd dimension of length 1 and called it a day.
- minimaxir 8y agoIt should be noted that CNNs and LSTMs are an order of magnitude slower than things like bag-of-words/fasttext unless you're using an expensive GPU, and the accuracy benefit if any may be marginal in practice. Kaggle prioritizes chasing a metric, but real-world data science has more considerations.
- alexcnwy 8y agoCNNs are usually much faster than LSTMs.
- soraki_soladead 8y agoFor most businesses a GPU is not really expensive, particularly if the results are important to the business. Chasing a single metric is certainly too narrow but getting the best results often does matter in a professional context too. While Kaggle can go overboard on massive ensembles using a state-of-the-art approach to the problem is often warranted outside of Kaggle.
- scottlocklin 8y agoDo you have an example backing up your assertions? In my experience, the worst text classification is usually fine. Labels are usually too inaccurate and subjective for "accuracy" to matter much.
- soraki_soladead 8y ago
- elyase 8y agoWe have done extensive testing in the context of chatbot intent classification and in our particular problem nothing (including CNN, LSTM, fasttext plus LUIS, Watson and other proprietary classifiers) has been able to beat a simple linear model trained on char n-gram features.
- minimaxir 8y agoChatbot intent would be a good use case for a linear model, as a single word/ngram would have a high impact on the result (in contrast to advanced architectures which try to account for ambiguity/contradictions in documents)
- eggie5 8y agoI've seen a rule of thumb that if the ratio of samples to words per sample is less than 1500 you prob don't have enough data for embeddings/cnn
- laichzeit0 8y agoAnecdotally, I found the same thing. N-gram BOW is surprisingly difficult to beat.
- briga 8y agoI've seen the same things in the models I've built. For basic intent classification simpler models seem to be more accurate, not to mention they train faster and require less memory. There seems to be a lot of emphasis on shiny complex neural network architectures, even when simple models work just fine.
- FridgeSeal 8y ago> There seems to be a lot of emphasis on shiny complex neural network architectures, even when simple models work just fine. It's resume-driven-development for data scientists. I've never seen an interviewer impressed with the fact that a job was performed using not-deep learning, but say that you used deep learning (despite how spurious it might be) and they light up like it's Christmas.
- 8y ago
- eggie5 8y agoWhy not just use a 1d conv over the sequence of embeddings?
- rundigen12 8y agoI was really hoping to see a summary comparison of the performance(s) of the different models at the end, e.g. accuracy vs. complexity vs. execution time, etc. Here's a summary from the end of each section... 1. TextCNN: "This kernel scored around 0.661 on the public leaderboard." 2. BiDirectional RNN: 0.671 3. Attention Models: 0.682
- ykevinator 8y agoThank you, that's exactly what I was scrolling through the bickering to find.
- ScoutOrgo 8y agoNo mention of ULMFiT? http://nlp.fast.ai/classification/2018/05/15/introducting-ulmfit.html http://nlp.fast.ai/classification/2018/05/15/introducting-ul...
- nixpulvis 8y agoI already don't like humans parsing my words half the time, I'm confident I'll hate most algorithms. Nothing inherently wrong with these methods, just a lot of possibilities for misuse in my eyes.
- SpaceManNabs 8y agoSeems like the article is mistaken on one part. Attention was first coined in this paper (as far as I know): https://arxiv.org/pdf/1409.0473v7.pdf https://arxiv.org/pdf/1409.0473v7.pdf The second page of the introduction of the "Hierarchical Attention Networks for Document Classification" paper mentioned in the article even cites it.