3 ms·
"Try 'em all" is not just an answer, but the only answer. The No Free Lunch Theorem says that averaged across all possible problems, no single classifier is th
by mattkrause 10y ago
"Try 'em all" is not just an answer, but the only answer.
The No Free Lunch Theorem says that averaged across all possible problems, no single classifier is the best; in fact, they're all equivalent.
However, you probably don't care about all possible problems, but a specific one. Over the last decade or so, we've discovered that deep learning works really well on certain classes of problems, particularly those that may have some kind of nested structure, as in object or speech recognition. If your problem resembles one of those, a deep neural network might be a good place to start.
- nl 10y agoRight - this is good advice. To paraphrase the learnings of thousands of data scientists on years of Kaggle competitions: A quick and dirty model for a baseline: Random Forest Structured data: Use a boosted tree algorithm (specifically the XGBoost implementation of gradient boosting), ensembled with maybe Extra Trees, Random Forests and MLPs Some kind of time component on large datasets: FTL regression, XGB Binary data (images or sound): Deep neural nets Text: Try LSTMs, but this will often be beaten by manual feature engineering and Word2Vec derived features put into XGB.
- minimaxir 10y agoLightGBM (https://github.com/Microsoft/LightGBM https://github.com/Microsoft/LightGBM) is shaping up to beat XGBoost; it has mostly API parity and it won in benchmarks before a v2 with a new algorithm.
- nl 10y agoI tried LightGBM for a Kaggle. I couldn't get anywhere near XGB. I was using the LambdaRank stuff. Given the boasting the LightGBM team had done I had assumed it would be close to XGB out-of-the-box for a ranking problem (since XGB only does pairwise ranking). It was far enough away that I had to ask if I was misinterpreting the output[1]. That was 6 months ago now, so maybe it has improved. I know they made big claims. [1] https://github.com/Microsoft/LightGBM/issues/37 https://github.com/Microsoft/LightGBM/issues/37
- minimaxir 10y agoDevelopment was rapid when I was working on a blog post in January using the tool. Things have likely improved if you want to give it another shot.
- nl 10y agoYeah, I might, thanks. Did you manage to replicate their results vs XGB? I don't think anyone has successfully used it for a high result in a Kaggle yet, which - for all its faults - is a good way to see what the maximum performance of a software package seems to be. LibFFM is the other thing I should have mentioned previously as being worth trying.