3 ms·

Any reference for this claim? In my opinion this is most certainly not the case - random forests are hard to overfit compared to gradient boosted trees but you

Any reference for this claim? In my opinion this is most certainly not the case - random forests are hard to overfit compared to gradient boosted trees but you can overfit with them too if you don't tune your parameters right. Overfitting is generally a function of the the size of your data and the complexity expressible in your model.

moffkalast 4y ago
My reference would be my machine learning prof mentioning it at the lectures a few years ago, something about decorrelation as the other guy in the replies mentions. It's possible he meant hard to overfit in comparison to something, not completely impossible.
taylorius 4y ago
A random forest is an ensemble model - i.e. lots of instances of a model, all separately fitted to the data, with some randomness so they're all different (e.g. bootstrapping, random subset of the data used to train etc). These models are then all used to classify, and their results averaged. This averaging will work to mitigate any overfitting that a single model might suffer from.
sweezyjeezy 4y ago
If you keep adding more and more trees to a random forest, variance will go down (they stop being 'random' and become more like the average over all possible trees). In this respect they can't overfit, but in practice there are other tweaks you can make such as tree depth and number of features) that you often want to tweak to increase model performance. These can certainly cause orverfitting.
ellisv 4y ago
RFs can overfit but in practice usually won't. I think Hastie & Tibshirani have an article on this.
rsfern 4y ago
I think the origin of the claim is from Leo Breiman himself [0]. I’m not sure if he discusses this in any papers, but probably. Sibling comments have some nuanced discussion of overfitting in tree-based models 0: https://www.stat.berkeley.edu/~breiman/RandomForests/cc_home.htm#remarks https://www.stat.berkeley.edu/~breiman/RandomForests/cc_home...

moffkalast 4y ago

My reference would be my machine learning prof mentioning it at the lectures a few years ago, something about decorrelation as the other guy in the replies mentions. It's possible he meant hard to overfit in comparison to something, not completely impossible.

taylorius 4y ago

A random forest is an ensemble model - i.e. lots of instances of a model, all separately fitted to the data, with some randomness so they're all different (e.g. bootstrapping, random subset of the data used to train etc). These models are then all used to classify, and their results averaged. This averaging will work to mitigate any overfitting that a single model might suffer from.

sweezyjeezy 4y ago

If you keep adding more and more trees to a random forest, variance will go down (they stop being 'random' and become more like the average over all possible trees). In this respect they can't overfit, but in practice there are other tweaks you can make such as tree depth and number of features) that you often want to tweak to increase model performance. These can certainly cause orverfitting.

ellisv 4y ago

RFs can overfit but in practice usually won't. I think Hastie & Tibshirani have an article on this.

rsfern 4y ago

I think the origin of the claim is from Leo Breiman himself [0]. I’m not sure if he discusses this in any papers, but probably. Sibling comments have some nuanced discussion of overfitting in tree-based models 0: https://www.stat.berkeley.edu/~breiman/RandomForests/cc_home.htm#remarks https://www.stat.berkeley.edu/~breiman/RandomForests/cc_home...