5 ms·
It should be noted that CNNs and LSTMs are an order of magnitude slower than things like bag-of-words/fasttext unless you're using an expensive GPU, and the acc
by minimaxir 8y ago
It should be noted that CNNs and LSTMs are an order of magnitude slower than things like bag-of-words/fasttext unless you're using an expensive GPU, and the accuracy benefit if any may be marginal in practice.
Kaggle prioritizes chasing a metric, but real-world data science has more considerations.
- alexcnwy 8y agoCNNs are usually much faster than LSTMs.
- soraki_soladead 8y agoFor most businesses a GPU is not really expensive, particularly if the results are important to the business. Chasing a single metric is certainly too narrow but getting the best results often does matter in a professional context too. While Kaggle can go overboard on massive ensembles using a state-of-the-art approach to the problem is often warranted outside of Kaggle.
- scottlocklin 8y agoDo you have an example backing up your assertions? In my experience, the worst text classification is usually fine. Labels are usually too inaccurate and subjective for "accuracy" to matter much.
- soraki_soladead 8y agoElectronic medical records. Anything in healthcare/medical, financial services, legal. There's a long way from the "worst text classification" to "sufficient text classification" in most real world use cases. With a reasonable budget you can relabel data and work around subjectivity.
- 995533 8y agoNobody cares how long it takes to train a model. What matters is prediction speeds, which are comparable (and NLP less likely to require high frequency, where a few more milliseconds matters). Besides that, the accuracy gains are not marginal anymore (BoW can't compete like it used to, especially with pre-trained models).
- darkpuma 8y agoTime to train a model matters for applications where you want to have end users training models on their own computers without spending so much CPU/GPU time that they have to plan their day around it. Consider for instance an RSS reader that classifies articles to determine whether or not to interrupt the user with a notification. This should be fast to train and update the model on the fly every time the user enters a correction (e.g. 'this article actually isn't interesting', or 'interrupt me with articles like this in the future'.)
- 995533 8y agoI would not retrain such a model on all data, just do online updates. Also I still think for that use case training times and latency are negligible (nobody cares or nobody notices any difference between training a BoW and bi-LSTM.) If you are deploying on resource-constrainted devices (IE: low-end PC's without GPU), it is not unusual to take a lot of time training a model on a very powerful computer (which nobody cares about), then distilling or transfering the result for test time.
- Cybiote 8y ago> Nobody cares how long it takes to train a model. This isn't true. It depends on your priorities and goals. Machine learning that spends most of its time unable to learn is not real AI. Some of us are interested in sample and energy efficient learning capable of on-line incremental updates immune to catastrophic forgetting. Not just because this is truer to actual learning but because it moves away from being dependent on a handful of companies to do the actual training. Anticipating some replies: no, transfer learning or meta-learning methods don't really avoid this. In the case of transfer learning, you still have that high coupling between a handful of sources. The down-sides of this is its own discussion. In addition, there are times where the ability to extract local relations can be dulled by the dominant wikipedia and common-crawl representations. Meta-learning gets you fast updates but you still cannot stray too far away from the domains that were met at training time. > What matters is prediction speeds I'm not a fan of bag of words models either but a simple dot product is always going to be faster than many matrix multiplies and or convolutions. The implementor should always try these as a base-line and decide if the performance accuracy trade-off is worth it for them.
- deleted 8y ago[deleted]
- computerex 8y agoAgreed. For many many applications, bag of words/tf-idf/support vector machines are enough but with much better training time.
- autokad 8y agoan active kaggler here. I dont use NN because they simply don't have great accuracy, and most importantly they have a huge amount of variance. this is mostly because the data on kaggle is not very large. the gbm trifecta (xgboost, catboost, lgbm) also does really really well. > "Kaggle prioritizes chasing a metric, but real-world data science has more considerations." this is counter to your point. most real-world considerations need things like model explanability. I notice that things dont make hacker news if they are doing anything other than NN. model with xgboost gets X% accuracy - crickets. model with X-Y% accuracy with DNN - headline news. I also notice that teams in industry tend to throw a DNN at a problem and never try something more simpler like xgboost. I saw a team with an LSTM for text lament they had 80% accuracy on training/evaluation, but when pushed to prod dropped down to 50%. I saw the errors they were getting, I said: maybe its too complicated and not generalizing well, have you tried xgboost? they retorted LSTM's w/ word2vec is very robust. I thought, obviously its not given your results. I tried to offer the idea that word2vec was trained on an entirely different kind of corpus (also a problem when trying to use word2vec in kaggle)
- soraki_soladead 8y agoI agree good models like xgboost get buried and inexperienced practitioners jump to deep models too quickly. Often without understanding how to properly architect and tune them. Always start with a simple baseline (EDIT: and good process). However, what's lacking in the ML practitioner community is nuance. Some applications need deep models some problems need xgboost. There isn't a "best" model in text classification because it depends on your data and problem.
- asah 8y agoThis chart is great for newbies, perhaps give it an update? https://scikit-learn.org/stable/tutorial/machine_learning_map/index.html https://scikit-learn.org/stable/tutorial/machine_learning_ma...
- autokad 8y agoI'm not a huge fan of the chart as it looks pretty 2014 dated. SVMs are rarely used anymore, so perhaps if you replaced all of them (including in the regression part) with gbm. Didn't notice any mention of logistic regression, which I still heavily use as well. dimentionality reduction is part of any step, which depends on number of observations, features, and model. my favorite is l1 or pca, but I am not afraid to use stepwise regression or some tree method.
- wodenokoto 8y agoDo you know of a good way to combine fasttext with none text features? Let's say I know seasonality is a strong feature in classifying my text, how can I add this? With a BOW I can literally just add SEASON_AUTUMN as a word to the text, and I'll get that extra feature as a dummy variable in my feature vector. But for fasttext, if I add such a word, it will just be averaged out in the final document feature vector.