3 ms·
This is good intuition for why ensembling overparametrized is a good idea. Doesn’t speak to why ensembles of tree-structured estimators in particular perform so
by math_dandy 2y ago
This is good intuition for why ensembling overparametrized is a good idea. Doesn’t speak to why ensembles of tree-structured estimators in particular perform so well compared to ensembles of other nonparametric estimators.
- youoy 2y agoIf you look at what makes it work well in the example, I would say it is being able to easily approximate a function with whatever degree of precision that you want, which translates to being able to isolate spikes in the approximation. For example, one could ask, what if instead of an approximation by step functions, we use a piecewise linear approximation (which is as good)? You can do that with a fully connected artificial neural network with ReLU nonlinearity, and if you check it experimentally, you will see that the results are equivalent. Why do people often use ensembles of tree structures? The ensembling part is included in the programming packages and that is not the case for ANN, so it is quicker to experiment with. Appart from that, if you have some features that behave like categorical variables, trees also behave better in training.