8 ms·
Why do tree-based models still outperform deep learning on tabular data? (2022)
- deleted 3y ago[deleted]
- frgtpsswrdlame 3y agoI still don't get the impetus or desire to make NNs work better for tabular data. Regression works pretty well and is easy to interpret/diagnose/work with. GBMs work really well (given a few considerations) and is trickier to work with but nothing crazy. When I see all the fancy hijinks people get up to when applying NNs to audio/text/pictures I think it's really cool but also not something I'd want to have to do if I didn't absolutely need to when working with data out of a relational db. And anyways, how much of a benefit could it actually bring? GBMs are already capable of fitting and dramatically overfitting most datasets.
- mrfox321 3y agoWhen you need the best possible model, full stop. E.g. finance In a sufficiently competitive space, good enough doesn't cut it.
- dchftcs 3y agoDo you know of any shop that is running deep learning profitably?
- mjhay 3y agoPlenty of places use DL models, even if it's just a component of their stack. I would guess that that gradient-boosted trees are more common in applications, though.
- hackerlight 3y agoDo you know what kind of strategies it's seeing use in?
- foobar20k 3y agoReal-time parsing of incoming news events and live scanning of internet news sites - coupled with sentiment analysis. Latency is an interesting challenge in that space.
- mjhay 3y agoStill mostly NLP and image stuff. Most actual data in the wild is tabular - which GBTs are usually some combination of better and easier. In some circumstances, NN can still work well in tabular problems with the right feature engineering or model stacking. They are also more attractive for streaming data. Tree-based models can't learn incrementally. They have to be retrained from scratch each time.
- dist-epoch 3y agoML is very good at figuring out stuff like every day at 22:00 this asset goes up if this another asset is not at a daily maximum and the volatility of the market is low. You might call this overfitting/noise/.... but if you do it carefully it's profitable.
- TimPC 3y agoMultiple parts of the iPhone stack run DL models locally on your phone. They even added hardware acceleration to the camera because most of the picture quality upgrades is software rather than hardware.
- Thrymr 3y agoThere is no such thing as "best possible model, full stop". Models are always context dependent, have implicit or explicit assumptions about what is signal and what is noise, have different performance characteristics in training or execution. Choosing the "best" model for your task is a form of hyperparameter optimization in itself.
- naijaboiler 3y agoI can’t upvote this enough. Whether in life, or with models, some people really do believe in the myth of absolutely meritocracy
- asdff 3y agoThese models usually have poorer fit though
- dawnofdusk 3y agoThe paper offers a reason why NNs working for tabular data would be good: >Creating tabular-specific deep learning architectures is a very active area of research (see section 2) given that tree-based models are not differentiable, and thus cannot be easily composed and jointly trained with other deep learning blocks. Here is a second reason, from the paper >Impressed by the superiority of tree-based models on tabular data, we strive to understand which inductive biases make them well-suited for these data. which is a great reason, because understanding the inductive biases of different learning/regression techniques gets us closer to a more general understanding of how to encode inductive biases in a generic learning algorithm.
- hackerlight 3y agoMy hypothesis is decision trees are more robust to nonstationary distributions. If the variance and means of the features shift dramatically, the model isn't going to blow up, because it's not additive. In the domains where NNs work well (image processing and language), you're dealing with a predictable and stable distribution of values. Elephants might look a bit different in the train and test set, but you're not randomly getting 100x the variance of the input data. The decision tree just isn't going to care as much, because splits around the mean will lead to the same outcome. Another hypothesis is that zooming into bivariable relationships is more important in tabular data. Neural nets are better at local and global context. But they struggle if all that matters is the relationship between two columns of data because of the additive nature. Large networks can figure it out due to model capacity, but then you'll run into overfitting.
- hansvm 3y agoIn case anyone's sufficiently motivated (no promises, but I might test it out eventually), a couple deep architectures that might address those concerns are: 1. Something like a deep support vector machine. Instead of (linear) -> (any activation), you want to create a bunch of features that look like testing the vector against a splitting hyperplane. One option is (bias) -> (matmul) -> (1-bit sigmoid). Applying a bias term _for each row_ let's you choose the branch location, the matmul's result will be positive or negative at each output feature depending on which side of the hyperplane normal to the vector described by the corresponding row you happen to fall on. Then just bring that down to -1 or 1 so you can't sneak much nonstationary drift variance into the output (perhaps train with a normal sigmoid annealed to behave more like this one, and a suitable regularizing term to keep the network from sneaking in values near 0 to thwart your annealing). 2. Use an attention-like mechanism, but across features (this would likely require an additional tensor channel, so that each "feature" carries information in a high enough dimensional space for this to do something meaningful). You apply the inductive bias that sparse feature interactions are important and need to be discovered. Those two ideas also compose easily.
- melondonkey 3y agoAt this point I wish every junior DS could read this paper and not come in to every problem with the new bright idea that they’re going to beat XGBoost with their DL architecture. Free promotion if they never say the words “latent subspace”
- barrenko 3y agoOne of those juniors is going to do it once!
- bbstats 3y agobecause smooth is better than jagged :)
- mikkom 3y agoIt's very important to note that this is from 2022. I'm not saying it's not true today but neural models have gotten much better in 2 years. (I'm personally using NN models for predicting certain values for tabularly structured data and at least for my case, the NN works better than state-of-the art tree models.)
- mjhay 3y agoDo you have any intuition you could share of why NNs work better in this case?
- queuebert 3y agoNot the parent, but NNs typically work better when you can't linearize your data. For classification, that means a space in which hyperplanes separate classes, and for regression a space in which a linear approximation is good. For example, take the circle dataset here: https://playground.tensorflow.org https://playground.tensorflow.org That doesn't look immediately linearly separable, but since it is 2D we have the insight that parameterizing by radius would do the trick. Now try doing that in 1000 dimensions. Sometimes you can, sometimes you can't or don't want to bother.
- mjhay 3y agoThat's an advantage over linear models, but GBTs handle non linearly-separated data just fine. Each individual tree can represent an arbitrary piecewise-constant function given enough depth, and then each tree in turn tries to minimize the loss on the residual of the previous trees. As such, they're effectively like a neural network with two hidden layers in terms of expressiveness.
- melondonkey 3y agoThis explanation doesn’t make sense to me. What do you mean by “linearize your data”—tree methods assume no linear form and are not even monotonically constrained. Classification is not done by plane-drawing but by probability estimation + cost function
- dawnofdusk 3y agoPaper seems interesting but I don't like the question title. I think the answer to the question would just be that tabular data is not fully in the "big data" regime yet so there is no reason a priori to expect deep NNs to do better. Factor in computational simplicity of tree-based models and I think the deck is stacked against deep learning from the start.
- math_dandy 3y agoDo you know of any (families of) examples of tabular datasets of any size (you can choose what "big" means) where deep learning convincingly outperforms traditional methods? I would love some quality examples of this nature to use in my teaching.
- Scene_Cast2 3y agoRecommendation engines: search, feeds (tiktok / youtube shorts / etc), ads, netflix suggestions, doordash suggestions, etc etc. Also happens to be my specialty.
- usgroup 3y agoI'm not sure that is true. I think inference speed is often the bottleneck for the use cases stated, as is the need for frequent re-training. As a result algorithms like catboost are very popular in those domains. I think catboost was actually invented by Yandex. PS: Its weird that you are being down-voted. I think your opinion is reasonable.
- Scene_Cast2 3y agoInference speed: more sophisticated stacks use multiple stages. Early stage might be a sublinear vector search, and the heavy hitting neural nets only rerank the remainder. Bytedance has a paper on their fairly fancy sublinear approach. Retraining - online training solves this for the most part. Frameworks - the only battle-tested batteries-included one I've seen is Vespa. Noone else publishes any of interesting bits. KDD is the most relevant conference if you're interested in the field. IIRC Xiaohongshu has some papers that can only really be done with NNs.
- Scene_Cast2 3y agoThe team behind Yggdrasil tree library at Google was doing some interesting research into tree differentiability (and thus unlocking SGD & end-to-end learning for hybrid architectures).
- Rexxar 3y agoIs it this library https://github.com/google/yggdrasil-decision-forests https://github.com/google/yggdrasil-decision-forests ?
- Scene_Cast2 3y agoYep.
- Rexxar 3y agoThanks
- mistrial9 3y ago[flagged]
- melondonkey 3y agoThis is interesting. Are BART models differentiable? I haven’t looked closely at them but I would have thought for posterior sampling they’d have to be. BART has been around for a while, too
- doubtfuluser 3y agoSince this is from 2022, I’m wondering how “tabular foundation models” could change this. The incredible success of DL we see at the moment comes partially from foundation models learning on a lot of “semi-related” data an “understanding” of the behavior. Something similar has been explored in tabular data as well iirc. So I would be curious to see latest DL results. On the other hand it is also the case that in most cases where DL based on foundation models is used, specific heavily tuned models outperform the generalistic models. And for tabular data there is a lot of experience how to make it great with tree based models.
- dweinus 3y agoWhat would these tabular foundation models look like? LLMs work as foundation models because the input is fixed in format (a sequence of text). Would the model be for a specific fixed tabular format?
- scottyak 3y agoOne promising approach is to encode each feature key and feature value as embedding vectors, concatenate them into "feature tokens", then feed them into a Transformer (without positional encodings). This takes advantage of column-order invariance. See: https://arxiv.org/abs/2403.01841 https://arxiv.org/abs/2403.01841 (ICLR 2024 spotlight)
- moelf 3y agoThere seems to be differentiable tree models now that perfor somewhat better than e.g. XGBoost https://github.com/Evovest/MLBenchmarks.jl?tab=readme-ov-file#higgs https://github.com/Evovest/MLBenchmarks.jl?tab=readme-ov-fil...
- candiodari 3y agoTLDR: Because tree-based models don't just outperform deep learning, they totally outclass deep learning on simple data. But they don't scale with larger and more complex data. You cannot (realistically) make an LLM with XGBoost. Kind of surprised how well Resnet and FT Transformer do though.
- nickpsecurity 3y agoI want to see more work combining them. Here’s an example I saw in one of the links in this thread: https://arxiv.org/abs/1806.06988 https://arxiv.org/abs/1806.06988 It combines NN’s with decision trees.
- huac 3y agothe famous FB ads paper (from 10 years ago!) combines decision trees with a logistic regression and shows a significant improvement: https://research.facebook.com/publications/practical-lessons-from-predicting-clicks-on-ads-at-facebook/ https://research.facebook.com/publications/practical-lessons... feel free to extend logistic regression to an MLP :)
- MAXPOOL 3y ago> deep learning architectures have been crafted to create inductive biases matching invariances and spatial dependencies of the data. Finding corresponding invariances is hard in tabular data, made of heterogeneous features, small sample sizes, extreme values Transformers with positional encoding have embeddings are invariant to the input order. CNN's have translation invariance and can have little rotational invariance. It's harder to find similar invariances to tabular data. Maybe applying methods from GNN's would help?
- queuebert 3y agoThis is tangential, but that paper has some amazingly good plots for an ML paper.
- melondonkey 3y ago[dead]
- usgroup 3y agoI'm not sure this is surprising. Say you were to glue together 10 datasets with the same 10 explanatory features and 1 response feature, but distributed very differently to each other. This would be no problem for tree based model because they'll conditionalise indefinitely to get a good fit. If the number of records is relatively small (say 10k) the dataset will be much too scarce for an NN to learn these discontinuities -- its like it has 1000 records per segment. Similarly, tabular data is often of this nature. Its not i.i.d, it tends to cluster.
- deleted 3y ago[deleted]
- 3abiton 3y agoI wonder if that would be the case for graph based models too
- skadamat 3y agoWhen working with tabular data, there are very few situations where absolute model performance is the only criteria that's important. In practice, the following are equally as important: - Explainability / debug-ability of models - Effort to train, deploy, and manage NN models in production - Capturing, collating, and organizing new & better datasets - Local developer experience and human-model-iteration time Building all of your software in C or Assembly will be faster and higher performant. But at what cost and with what tradeoffs? Building a website has a different set of tradeoffs than building a program for the Mars rover.
- derefr 3y agoIt's funny; as a regular non-ML programmer, the optimum for every one of those factors for "tabular data" would seem to me, to be to "throw the tabular data into a relational data warehouse, and ask your questions in the form of SQL queries." Or, if the "tabular data" is heavily relationship-based, then possibly replace "relational data warehouse" with "graph database", and "SQL queries" with whatever querying language that graph DB is natively / most expressively queried in. Of course, this is the most important implicit "equally important" factor, one that an ML dev would think goes without mentioning: the generality or "power" of the model in what questions it can answer. You can only make these trade-offs in the context of knowing what kinds of questions you want your model to solve for! If all your questions are quantitative ones, maybe the right "model" for you is an RDBMS! --- Though, that being said... why can't a deep-learning model emulate the thing that an RDBMS does, "at runtime", as part of its "mental toolkit" for approaching problems? That would be the best of both worlds, no? I know that LLMs in particular have been observed to have "emergent numeracy" above a certain training-set size. There is a step function in how they approach such problems, going from their only being able to answer arithmetic questions on numbers of bounded size, and sometimes getting the answers wrong (probably this is due to a memorization-based approach); to being able to answer arbitrary arithmetic questions on operands of unbounded size, and always getting the answer correct. I would guess that that what's happening, is that they are developing a functional component of their network that works akin to an Arithmetic Logic Unit, operating not on tokens, but on tokens transformed into a "numeric register" representation that is amenable to having math done to it with stable, quantized, position-independent results. (Just like the functional component that human brains develop after seeing enough math problems... probably.) Do you, as an ML dev, think it would ever be possible for any of the model architectures we're familiar with today, to be trained such that they would develop an analogous emergent functional component for handling tabular-data questions, by transforming its internal working state into relational-DB/graph-DB data structures — e.g. page-heaps of binary-packed row-tuples; B-tree indices; etc — and then manipulating the working state in that form, using learned algorithms applicable to that type of data? It seems to me (possibly just because I don't know any better) that just as with numeracy, "being able to put the data into a different and better internal representation" is what would be needed for deep-learning models to become truly good at dealing with tabular-data problems. But, unlike with numeracy, "thinking as if you were a relational database" is not something a single human would ever intuit how to do without being taught. Relational algebra — and the data-structures and algorithms to make it practical to have a Turing machine do said relational algebra — wasn't even a single intuition, but a conscious effort, of multiple humans, working together over years. I strongly doubt that there's any number of "tabular-data problems" that you could show a human being, that would result in them developing an intuitional ability to do what a relational database does with its memory to efficiently answer queries. (I suppose we could give an ML model an RDBMS, and hardwire it to interact with it. I know there are hybrid ML + formal-logic systems. Are there hybrid ML + data-warehouse systems? Not where the model queries an external DB — while that can be done, it'd be only in the same "stop and do this" way that ChatGPT runs Python code, which wouldn't make it a thinking tool the way that the formal-logic proof engines are for hybrid ML systems. Rather, I mean that some data-warehouse execution engine could be embedded into the ML execution framework itself, deployed as part of the GPU shader-program to each tensor core, such that data-warehouse operations can be done as a native part of the network's per-node instruction-set. Anyone ever tried this?)
- martindbp 3y agoDeep learning really shines when the input is raw and at a very low abstraction level: pixels, byte pair encodings etc. Using deeply learning for classification on tabular data is just needless complexity, as the variables are often at a very high abstraction level already. Also with tabular data there are generally not many spatial or temporal relationship between the variables, which CNNs and transformers excel at.
- jobigoud 3y agoWe have things that can describe and explain why an image they have never seen is funny. That's pretty high level.
- martindbp 3y agoYes, what I meant was deep learning is great at deriving those higher level abstractions from low level raw data. Words can be seen as something in between, bag of words can be fairly effective at simpler tasks, but LLMs embed words into higher and higher abstractions.
- andy99 3y agoAlso images and text have tons of recurring patterns that can be exploited to train big models with lots of data. There is an internet worth of each modality that at least generally can all contribute helping a model build up a better overall understanding. There is no analog for tabular data, it's all different.
- hresvelgr 3y agoI wonder if tree-based models would outperform in situations concerning source code as well; given it's already quite structured. Going a step further and supplying and AST may be even more beneficial.
- sgt101 3y agoIsn't this just that trees are a natural compression of tables? smthin smthin inductive bias?
- deleted 3y ago[deleted]
- hashta 3y agoI have a lot of experience working with both families of models. If you use an ensemble of 10 NNs, they outperform well-optimized tree-based models such as XGBoost & RFs.
- padthai 3y agoWhich kind of ensemble? Because it cannot be as easy as a voting meta model of nn with same architecture/hyperparametres right?
- knightoffaith 3y agoIs this true? I hear that XGBoost tends to win Kaggle competitions for tabular data - how come you don't see NN ensembles winning instead? Or do they?
- malshe 3y agoI would like to know this too. From what I understand, it is XGBoost that rules Kaggle.
- hashta 3y agoTo both questions above, just simple averaging of the logits (classification) or raw outputs (regressions) usually works well. If I had to guess why people don't use this approach often in kaggle competitions is the relative difficulty of training an ensemble of NNs. Also, NNs are a bit more sensitive to the type of features used and their distribution relative to decision trees (DTs). Ensemble models work well because they reduce both bias & variance errors. Like DTs, NNs have low bias errors and high variance errors when used individually. The variance error drops as you use more learners (DTs/NNs) in the ensemble. Also, the more diverse the learners, the lower the overall error. Simple ways to promote the diversity of the NNs in the ensemble is to start their weights from different random seeds and train each one of them on a random sample from the overall training set (say 70-80% w/o replacement).
- jhoho 3y agoCan you recommend a resource on this for the curious learner?
- up2isomorphism 3y agoBecause human-being heavily compress picture, sounds, but much less with tabular data.
- GVaroquaux 3y agoAs an author, I'm happy to see this work here. For those curious about what we have been up to on the topic of tabular learning, we have found a setting where deep learning does seem to bring sizable benefits (spoiler alert, it's about being able to pre-train, and transferring to new data works best when there are some strings to be recognized): https://arxiv.org/abs/2402.16785 https://arxiv.org/abs/2402.16785 In the above work, pre-trained tabular models markedly outperform tree-based models (including catboost, which is a very strong baseline). As someone who has been banging on tabular data for years, I'm really excited about this development.
- afiodorov 3y agoA calculator outperforms deep learning on basic arithmetic tasks. Tree-based models are extremely good at finding clustering patterns; they outperform trained humans at that, thus we have commercial applications such as fraud detection. Deep Learning is most promising way of getting us to general intelligence. So far only known general intelligence, human intelligence, has many quirks at specific tasks and I think Deep Learning won't be any different. However Deep Learning models can recognise their own weakness and call tree-based model if they think that's appropriate.