2 ms·
It's actually quite obvious why gradient boosted trees don't work on the synthetic dataset, if you know how they're trained. I guess it's a testimonial for thei
by yorwba 8y ago
It's actually quite obvious why gradient boosted trees don't work on the synthetic dataset, if you know how they're trained. I guess it's a testimonial for their usefulness as black-box predictors that the author didn't realize that.
For gradient boosted trees, you first need to grow a single tree. That tree starts with a single leaf and then needs to be split to try and improve performance. But because the data is perfectly antisymmetric, no suitable split can be found. So the growing process terminates. Gradient boosting can't help you, because the residuals to train the next tree on are identical to the original data.
If you add even the slightest amount of imbalance to the data, e.g. by sampling random positions instead of using a grid, the problem disappears.
- mariofilho 8y agoHi, thanks for your comment: I opened an issue on Github. This didn't seem obvious to many people that read the article, so it's nice that now we can keep this in mind while using the model: https://github.com/dmlc/xgboost/issues/4069 https://github.com/dmlc/xgboost/issues/4069