3 ms·
What are the principled ways that tree based models handle categorical data? If you end up having to do one-hot encoding it feels like you need very wide forest
by thom 4y ago
What are the principled ways that tree based models handle categorical data? If you end up having to do one-hot encoding it feels like you need very wide forests or very deep trees. If your categorical data is actually vaguely continuous then splits can be quite efficient but that’s rare.
I assume some day someone will be able to explain all this in information theoretic terms. I’m never sure if we’re comparing like with like (are the deep learning models we’re comparing against actually that deep, for example?) but clearly there’s something to the intuition that many small overfit models are more efficient than one big general model.
- coffee_am 4y agoOne way is to use categorical set splits [1] (proposed for categorical set inputs, but works for categorical features as well), used in TF-DF [1]. Greedy, and expensive to train (cheap inference though), but it gives great results. [1] https://arxiv.org/pdf/2009.09991.pdf https://arxiv.org/pdf/2009.09991.pdf [2] https://www.tensorflow.org/decision_forests/text_features https://www.tensorflow.org/decision_forests/text_features
- CapmCrackaWaka 4y agoSome implementations will group the categories into two groups, right and left. An easy way to do this is to create the groups based on whatever grouping creates the biggest decrease in the loss. There are more complicated implementations, lightgbm groups by the sum(gradient) / (sum(hessian) + smooth) where smooth is some smoothing parameter which prioritizes class size over loss decrease. They link to this paper in their docs: https://www.tandfonline.com/doi/abs/10.1080/01621459.1958.10501479 https://www.tandfonline.com/doi/abs/10.1080/01621459.1958.10...