3 ms·
How would you even apply layer normalization or SWA to XGB? These methods are neural net specific.
by ipsum2 5y ago
How would you even apply layer normalization or SWA to XGB? These methods are neural net specific.
- nightcracker 5y agoBatch normalization is nothing neural network specific to it if you use it on the input layer. I don't think it matters for a tree algorithm like XGBoost either way though. SWA is pretty NN specific. So leave it out for XGB. There's a bunch that are relevant, and they could be very important.
- jackylupino 5y agoBatch norm has an advantage for iterative methods on mini-batches, while XGB uses the full training set. Using batch norm on the full training set is equivalent to Z-normalizing the features, which has no effect at all for XGB as the scale of features plays no role at the split decisions of the tree nodes. Apart few non-parametric data augmentation methods (notice adversarial augmentation is also nn specific), I do not think any other regularization used in that paper can be directly/intuitively applied to XGB.
- mandelken 5y agoGBDT have their own set of hyperparameters such as learning rate, number of trees, min samples per bin, l0, l1, etc. So you could definitely also create an appropriate cocktail to optimize on, although GBDT are typically more robust wrt huperparameters.
- nightcracker 5y agoThe authors do claim to do a hyperparameter sweep but only for vanilla XGB hyperparams.
- civilized 5y agoThe old "my method (with as much optimization as I could get away with) beats the other method (with as little optimization as I could get away with)"
- disgruntledphd2 5y agoYup. The Least Publishable Unit strikes again.