4 ms·
Well iirc a convenient trait of a random forest classifier is that it cannot overfit onto learning data. Something that's not exactly true for deep learning.
by moffkalast 4y ago
Well iirc a convenient trait of a random forest classifier is that it cannot overfit onto learning data. Something that's not exactly true for deep learning.
- taeric 4y agoI'm curious on this claim. Feels like any model can over fit.
- HelloNurse 4y agoWith algorithms that, for each additional input entry, either add a predictable amount of complexity to improve fitness (e.g. another tree split) or decide that the model is already fit enough, one needs to do something deliberate or very stupid to overfit data compare to neural networks where overfitting is hard to detect, let alone prevent quantitatively, and that are sometimes designed to overfit in order to interpolate "intelligently" between samples.
- marcosdumay 4y agoA model can only overfit if it has enough parameters. But indeed, any model with enough parameters can overfit. The thing about deep learning is that the number of parameters is astronomical.
- disgruntledphd2 4y agoHence, the interesting question is why deep learning models don't overfit all the time.
- DougBTX 4y agoThis is from 2019 so I suspect there’s a more recent paper now, but this is interesting: https://openai.com/blog/deep-double-descent/ https://openai.com/blog/deep-double-descent/
- disgruntledphd2 4y agoAgreed, but it doesn't really explain it in any real sense.
- mr_toad 4y ago“However, in the over-parameterized regime, there are many models that fit the train set and there exist such good models.” One explanation is that a very large model effectively fits several models and penalises over-fit models itself.
- skinner_ 4y agoWork is ongoing, this is a whole field of research now, a good keyword to search for is "implicit regularization".
- marcosdumay 4y agoI really like how AI research is all about empirical analysis of mathematics.
- gnramires 4y agoThe answer is really very simple: because we use training and test sets, and generally use iterative training methods. As soon as test set error plateaus or starts to degrade, you stop training. This is called early stopping and can be considered a form of regularization. In theory it would seem a neural network (with early stopping "correctly" implemented) can never overfit, although I suppose in practice it can vary a little (because of test variance, too large step sizes, etc.). It's possible to train huge overparametrized networks this way. One big problem I can see with this approach (maybe compared to over forms of regularization) is that it might be very space/memory inefficient. But if you've got a big budget I think it's a surefire way to make sure your model performs as well as possible (without having to tune hyperparameters too much or deal with local minima that probably happens at near-capacity networks).
- bbstats 4y agoRandom Forests can overfit, but with sufficient data (and no data leakage) it is difficult to do so. The final output of a random forest regression for example is just the average of all the final prediction nodes of the individual trees, which thanks to bootstrapping are decorrelated. So - adding more trees does not tend to alter predictions very much.
- melony 4y agoIf you train your ensemble long enough, won't it overfit too?
- empyrrhicist 4y agoNo, it's not sequential. For a given number of bootstrap replicates you can over fit by using trees that are too complex, but you won't overfit in the number of bootstrap replicates like you would with Gradient Boosted Trees.
- melony 4y agoCan you elaborate? I am unfamiliar with gradient boosted trees.
- leto_ii 4y agoI think the idea is that gbt's are trained sequentially on the "residuals" left from the previous tree, while rf's are uncorrelated (can be trained in parallel, don't depend on one another). This means that in rf's there's no "compounding" effect that can lead to overfitting. The main way to overfit rf's would be to train many large trees.
- omegalulw 4y agoAny reference for this claim? In my opinion this is most certainly not the case - random forests are hard to overfit compared to gradient boosted trees but you can overfit with them too if you don't tune your parameters right. Overfitting is generally a function of the the size of your data and the complexity expressible in your model.
- moffkalast 4y agoMy reference would be my machine learning prof mentioning it at the lectures a few years ago, something about decorrelation as the other guy in the replies mentions. It's possible he meant hard to overfit in comparison to something, not completely impossible.
- taylorius 4y agoA random forest is an ensemble model - i.e. lots of instances of a model, all separately fitted to the data, with some randomness so they're all different (e.g. bootstrapping, random subset of the data used to train etc). These models are then all used to classify, and their results averaged. This averaging will work to mitigate any overfitting that a single model might suffer from.
- sweezyjeezy 4y agoIf you keep adding more and more trees to a random forest, variance will go down (they stop being 'random' and become more like the average over all possible trees). In this respect they can't overfit, but in practice there are other tweaks you can make such as tree depth and number of features) that you often want to tweak to increase model performance. These can certainly cause orverfitting.
- ellisv 4y agoRFs can overfit but in practice usually won't. I think Hastie & Tibshirani have an article on this.
- rsfern 4y agoI think the origin of the claim is from Leo Breiman himself [0]. I’m not sure if he discusses this in any papers, but probably. Sibling comments have some nuanced discussion of overfitting in tree-based models 0: https://www.stat.berkeley.edu/~breiman/RandomForests/cc_home.htm#remarks https://www.stat.berkeley.edu/~breiman/RandomForests/cc_home...
- lupire 4y agoRandom forests can overfit, but they naturally have fewer parameters than nets, so don't overfit as a function of training time. Neural nets have far more parameters and so are susceptible to overfitting with more training time.