3 ms·
I don't think every single regularization method in the cocktail can be applied to non-neural-network methods, but I'm pretty sure some of them can, like data a
by blt 5y ago
I don't think every single regularization method in the cocktail can be applied to non-neural-network methods, but I'm pretty sure some of them can, like data augmentation. The authors could have figured out which methods can be applied to non-NN models or considered if equivalent/analogous methods exist. I agree it would make a fairer comparison.
- jackylupino 5y agoI agree with your point, e.g. data augmentation can be added, but thats pretty much it. All the other regularization techniques they use are neural network specific and cannot be applied to gradient-boosted trees. What I find particularly striking at this paper is that their method trains a single neural network which outperforms an ensemble of decision trees (XGBoost). Asking for perfect apple-to-apple comparisons means also comparing an ensemble of the MLPs vs. XGBoost. In this context, at least the message here is that XGBoost and/or other gradient-boosted methods are not anymore a silver bullet for tabular datasets. Boosting for trees was great in reducing both bias and variance, but apparently neural networks can achieve the same effect with a high capacity (low bias) and a mix of modern regularization techniques (low variance).