3 ms·
So how, for instance, do I even decide, that Random Forest is not enough for this task and I want to build some specific kind of neural net? The problem here i
by nl 10y ago
So how, for instance, do I even decide, that Random Forest is not enough for this task and I want to build some specific kind of neural net?
The problem here is that it's really hard to give generic advice. As an analogy this is like asking "how do I know if Rails is enough for this task".
The answer is usually "yes", but the specifics matter a lot.
So in this specific case (and I realize you aren't looking for specific advice here, but I think the principles are useful):
Random Forests are very powerful, and work really well for hundreds, maybe thousands of features, on large but not huge amounts of data and are fairly easy to train.
There are a large number of types of neural networks. One of the big advantages of deep neural networks is that that can reduce the need for manual feature engineering. For examples conventional neural networks extract features from images that work better than any human engineered features, and LSTMs (and variations) work well at extracting features from text. The problem with deep neural networks is that they (generally) need a lot of data to train.
So, as usual the answer is "it depends".
In industry though, 90% of the time the question isn't "what classifier should I use". It's "how do I get the data"/"how do I extract features" and then "lets try all the classifiers and see what works best".
- mattkrause 10y ago"Try 'em all" is not just an answer, but the only answer. The No Free Lunch Theorem says that averaged across all possible problems, no single classifier is the best; in fact, they're all equivalent. However, you probably don't care about all possible problems, but a specific one. Over the last decade or so, we've discovered that deep learning works really well on certain classes of problems, particularly those that may have some kind of nested structure, as in object or speech recognition. If your problem resembles one of those, a deep neural network might be a good place to start.
- nl 10y agoRight - this is good advice. To paraphrase the learnings of thousands of data scientists on years of Kaggle competitions: A quick and dirty model for a baseline: Random Forest Structured data: Use a boosted tree algorithm (specifically the XGBoost implementation of gradient boosting), ensembled with maybe Extra Trees, Random Forests and MLPs Some kind of time component on large datasets: FTL regression, XGB Binary data (images or sound): Deep neural nets Text: Try LSTMs, but this will often be beaten by manual feature engineering and Word2Vec derived features put into XGB.
- minimaxir 10y agoLightGBM (https://github.com/Microsoft/LightGBM https://github.com/Microsoft/LightGBM) is shaping up to beat XGBoost; it has mostly API parity and it won in benchmarks before a v2 with a new algorithm.
- nl 10y agoI tried LightGBM for a Kaggle. I couldn't get anywhere near XGB. I was using the LambdaRank stuff. Given the boasting the LightGBM team had done I had assumed it would be close to XGB out-of-the-box for a ranking problem (since XGB only does pairwise ranking). It was far enough away that I had to ask if I was misinterpreting the output[1]. That was 6 months ago now, so maybe it has improved. I know they made big claims. [1] https://github.com/Microsoft/LightGBM/issues/37 https://github.com/Microsoft/LightGBM/issues/37
- minimaxir 10y agoDevelopment was rapid when I was working on a blog post in January using the tool. Things have likely improved if you want to give it another shot.
- nl 10y agoYeah, I might, thanks. Did you manage to replicate their results vs XGB? I don't think anyone has successfully used it for a high result in a Kaggle yet, which - for all its faults - is a good way to see what the maximum performance of a software package seems to be. LibFFM is the other thing I should have mentioned previously as being worth trying.
- soVeryTired 10y agoAnother less-recognised point is that in industry, you also need to ask "how can I maintain this?" and "what can go wrong with my algorithm?". In one use case, a "blip" in your algorithm might mean showing the wrong kind of advertisement to a user. Not great, but ultimately no big deal. In another, it might mean automatically buying billions of dollars' worth of pumpkin futures (cf. Knight capital). In the latter case you need a much greater penalty on model complexity, and much more emphasis on interpretability.
- nl 10y agoWhile I agree with your point (and often use this in interview questions) that wasn't what caused the Knight Capital problem. That was bad software engineering and deployment practices, and had nothing to do with interprability of the model (actually it had little to do with the model at all.) They repurposed a feature toggle, then misdeployed the code: http://pythonsweetness.tumblr.com/post/64740079543/how-to-lose-172222-a-second-for-45-minutes http://pythonsweetness.tumblr.com/post/64740079543/how-to-lo... I understand that this was an example, but I'm sure someone will misread it as what happened in that case.
- soVeryTired 10y agoYep - meant it as an example of a general catastrophic software glitch rather than a ML algorithm gone haywire.