14 ms·

Why do tree-based models still outperform deep learning on tabular data?

315 points by isolli 4y ago

kriro 4y ago
I think one of the issues is that there's no pretrained universal spreadsheet models (that I'm aware of, granted I don't do much work with tabular data) equivalent to ImageNet based models that you can use as a base and then transfer learn on top of that.
LittlePeter 4y ago
What is tabular data precisely? I can represent an image as a table of RGB values. I can represent hierarchical data as a table of unnested values.
coffee_am 4y ago
I can't explain it, but I help maintain TensorFlow Decision Forests [1] and Yggdrasil Decision Forests [2], and in an AutoML system at work that trains models on lots of various users data, decision forest models gets selected as best (after AutoML tries various model types and hyperparameters) somewhere between 20% to 40% of the times, systematically. It's pretty interesting. Other ML types considered are NN, linear models (with auto feature crossings generation), and a couple of other variations. [1] https://github.com/tensorflow/decision-forests https://github.com/tensorflow/decision-forests [2] https://github.com/google/yggdrasil-decision-forests https://github.com/google/yggdrasil-decision-forests
- onasta 4y ago
  Super interesting! Do you know the kind of data that it's usually used for? And in the remaining 80% to 60%, do NNs acccount for a large portion of the best models? Bonus question: are the stats you're mentioning publically available?
  - coffee_am 4y ago
    Sadly (but correctly) nothing is public, no one ever sees any data, it's a service. Pure FNN (feedforward NN) models, if I recall correctly is also ~30 to 40%. Since the server doesn't work for all types of data, and probably folks that are experts in ML would do their own hyperparameter tuning, and custom models, this leads to the bias on the type of datasets that are compete. But this share have been consistent over many months of various unrelated datasets, I believe.
fennecfoxy 4y ago
Are the tree based models being provided with data that's "kind of like" the data they were designed for/at training time for the deep learning model? Or "exactly like"? I think that's perhaps the difference?
throwaway0asd 4y ago
The answer is extremely straightforward: less computation. That’s less computation when traversing nodes in a set which can result in either an exponential or logarithmic difference depending upon the data and means of traversal. This holds true almost universally.
rdedev 4y ago
I've mostly come to the conclusion that DL is not going to beat tree based models especially when it comes to training speed. What I am now looking at is if it's possible to feed in world knowledge from models like beer as features to XGBoost but without resorting to creating templates for table rows
muyuu 4y ago
why would anyone assume deep learning to beat everything at everything, even applied relatively naively?
marcodiego 4y ago
Wait until we see deep learning AI creating tree-based models. /s
- PartiallyTyped 4y ago
  Technically you can transform any finite / not runtime defined computation graph into a "tree" model.
  - optimalsolver 4y ago
    See symbolic regression and genetic programming.
- uoaei 4y ago
  Not actually that complicated of an approach! Hard to implement correctly, which is why it's not more popular right now.
- karmasimida 4y ago
  triggering co-pilot to use xgboost might count as one today lol?
deleted 4y ago
[deleted]
isolli 4y ago
Related, an article from 2019 [0] on how neural network finally beat statistical models (e.g. ARIMA) in time-series forecasting. [0] https://towardsdatascience.com/n-beats-beating-statistical-models-with-neural-nets-28a4ba4a4de8 https://towardsdatascience.com/n-beats-beating-statistical-m...
- mr_toad 4y ago
  I had an econometrics professor who would have objected to calling ARIMA models statistics. He though they were no better than data mining.
- nomel 4y ago
  I can only assume this was achieved years ago, as a black project in the financial market.
spywaregorilla 4y ago
Tree based models seem like obvious approaches that should just work. It's more or less how a human would parse a problem. Throw some X and O's on a 2d plot. Draw lines to partition it into X sections and O sections. That's a tree based model.
- havercosine 4y ago
  I actually agree with this take! Deep learning models shine in problem domains where the gap between the data representation to the way how humans parse it is large. In images, for example, we just have RGB matrix but what human vision does is very complex. So representation learning done by deep learning models tries to close the gap. For lot of tabular data, human's don't go from raw representation to a mental model quickly. We in fact follow a model similar to tree based classifier: look at how a variable is splitting the output, compare averages across groups etc.
- melenaboija 4y ago
  Now you can contact the authors and tell them to replace the publication by your 4 lines. And tree based models is not what you described, at most that could be a tree classifier. EDIT: First of all sorry for my comment, I think you did not like it. My point is that what you state as obvious on how tree based methods work is most probably not what makes them powerful but the fact that you have a bunch of them. To me if there is some intuition, statistics is more prevalent than separating spaces in hyperplanes. Here you can see the boundaries of a random forest compared to a single tree, the more dimensions you add the more blurry and unintuitive in terms of hyperplanes it gets: https://scikit-learn.org/stable/_images/sphx_glr_plot_classifier_comparison_001.png https://scikit-learn.org/stable/_images/sphx_glr_plot_classi...
  - spywaregorilla 4y ago
    - abduhl 4y ago
      So it sounds like you’d need your 4 lines of pseudo-code from your first post and then another few lines of comments to describe how your 4 lines replace the authors’ publication? Something like “All tree models are partitions of the space based on training data, and the extension to any particular tree model is an obvious exercise left to the reader”?
jwilber 4y ago
If you’re interested in how tree-based models work, I wrote an interactive explanation on decision trees here: https://mlu-explain.github.io/decision-tree/ https://mlu-explain.github.io/decision-tree/ and random forests here: https://mlu-explain.github.io/random-forest/ https://mlu-explain.github.io/random-forest/ It’s also worth noting that a recentish paper shows neural networks can perform well on tabular data if well-regularized: https://arxiv.org/abs/2106.11189v1?utm_source=jesper&utm_medium=email&utm_campaign=coming-up-with-weather-puns-is-a-breeze https://arxiv.org/abs/2106.11189v1?utm_source=jesper&utm_med...
- isabellat 4y ago
  Really nice interactive explanations!
- lnenad 4y ago
  That was super easy to digest, thank you!
moffkalast 4y ago
Well iirc a convenient trait of a random forest classifier is that it cannot overfit onto learning data. Something that's not exactly true for deep learning.
- taeric 4y ago
  I'm curious on this claim. Feels like any model can over fit.
  - HelloNurse 4y ago
    With algorithms that, for each additional input entry, either add a predictable amount of complexity to improve fitness (e.g. another tree split) or decide that the model is already fit enough, one needs to do something deliberate or very stupid to overfit data compare to neural networks where overfitting is hard to detect, let alone prevent quantitatively, and that are sometimes designed to overfit in order to interpolate "intelligently" between samples.
  - marcosdumay 4y ago
    A model can only overfit if it has enough parameters. But indeed, any model with enough parameters can overfit. The thing about deep learning is that the number of parameters is astronomical.
    - disgruntledphd2 4y ago
      Hence, the interesting question is why deep learning models don't overfit all the time.
      - DougBTX 4y ago
        This is from 2019 so I suspect there’s a more recent paper now, but this is interesting: https://openai.com/blog/deep-double-descent/ https://openai.com/blog/deep-double-descent/
        4y ago
VoVAllen 4y ago
It's all about features and data scale. Recommendation system itself is actually a large table, DL method already proved effectiveness there. Let's say if you have text in your tabular data. Tree model(with traditional method such as tfidf) will do much worse than the transformer-based model. DL always suffers from inadequate data, so if there's no enough data or inductive biases, tree model can be a better choice in that way.
Permit 4y ago
> Results show that tree-based models remain state-of-the-art on medium-sized data (∼10K samples) even without accounting for their superior speed. Is that really "medium"? That seems very small to me. MNIST has 60,000 samples and ImageNet has millions. I think the title overstates the findings. I'd be interested to hear how these methods compare on much larger datasets. Is there a threshold at which deep learning outperforms tree-based models? Edit: They touch on this in the appendix: > A.2.2 Large-sized datasets > We extend our benchmark to large-scale datasets: in Figures 9, 10, 11 and 12, we compare the results of our models on the same set of datasets, in large-size (train set truncated to 50,000 samples) and medium-size (train set truncated to 10,000 samples) settings. > We only keep datasets with more than 50,000 samples and restrict the train set size to 50,000 samples (vs 10,000 samples for the medium-sized benchmark). Unfortunately, this excludes a lot of datasets, which makes the comparison less clear. However, it seems that, in most cases, increasing the train set size reduces the gap between neural networks and tree-based models. We leave a rigorous study of this trend to future work.
- mochomocha 4y ago
  I've put in production numerous models with millions of tabular data points and a 10^5-10^6 feature space where tree-based models (or FF nets) outperform more complex DL approaches.
  - whymauri 4y ago
    I'll chime in with billions of data points and 100-300 feature space with some smart feature engineering outperforming DL in runtime/compute (by orders of magnitude) and performance. But the domain was very specific and everything prior to the tree was doable with matrix operations, with the tree model summarizing a mixture of experts that chose optimal features.
    - bloudermilk 4y ago
      Damn, this sounds fascinating! Have you shared more anywhere eelse?
  - 4y ago
CapmCrackaWaka 4y ago
I have a theory - tree based models require minimal feature engineering. They are capable of handling categorical data in principled ways, they can handle the most skewed/multimodal/heteroskedastic continuous numeric data just as easily as a 0-1 scaled normal distribution, and they are easy to regularize compared to a DL model (which could have untold millions of possible parameter combinations, let alone getting the thing to train to a global optimum). I think if you spent months getting your data and model structure to a good place, you could certainly get a DL model to out-perform a gradient boosted tree. But why do that, when the GBT will be done today?
- jb_s 4y ago
  do you reckon it's possible to somehow transfer learn from a GBT to a NN ?
- a-dub 4y ago
  this is along the lines of my thinking. people organize and summarize data before throwing it into spreadsheets, where deep learning models do their thing by generating new representations from raw data. in a sense, most data in spreadsheets is compressed and deep learning models prefer to find their own compression that best suits the task at hand. or in human terms: "these spreadsheets are garbage. i can't work with this. can you bring me the raw data please?" :)
- deleted 4y ago
  [deleted]
- username_exists 4y ago
  occam's razor
- drzoltar 4y ago
  I think another aspect is that most modern GBT models prefer the entire dataset to be in memory, thereby doing a full scan of the data for each iteration to calculate the optimal split point. That’s hard to compete with if your batch size is small in a NN model.
fdgsdfogijq 4y ago
Because high tabular data doesnt have enough complexity compared to language or images.
jmmcd 4y ago
A great paper and an important result. However, it omits to cite the highly relevant SRBench paper from 2021, which also carefully curates a suitable set of regression benchmarks and shows that Genetic Programming approaches also tend to be better than deep learning. https://github.com/cavalab/srbench https://github.com/cavalab/srbench cc u/optimalsolver
ersiees 4y ago
Our lab works on changing this. I think it might still take some years for a full solution, but so far we are successful with NNs for small datasets with meta-learning (https://arxiv.org/abs/2207.01848 https://arxiv.org/abs/2207.01848) and large datasets with regularisation (https://arxiv.org/abs/2106.11189 https://arxiv.org/abs/2106.11189). The first is very new, but the second is also cited in this paper, but they didn’t run it as a baseline.
- freemint 4y ago
  Tree based algorithms have the advantage that global or robust optimization of them is possible. Do your approaches offer similar guarantees?
  - ersiees 4y ago
    The first work, I linked, arguably gets around this issue completely. There we train a neural network to approximate Bayesian prediction directly. It accepts the dataset as input. Thus, there is no optimisation procedure per dataset, just a forward pass. So far, this seems to be pretty stable.
    - freemint 4y ago
      Just because you optimize network parameters to fit a surrogate model of the data doesn't mean you are not doing local optimization without guarantees on robustness or global optimality w.r.t. network parameters to fit the surrogate. Am i missing something?
- civilized 4y ago
  Why not run your own method on their benchmark? If your method is general-purpose, it should be very easy. And it would be very impressive to see your method succeed on a task that was not chosen by you to write your paper.
  - ersiees 4y ago
    Yes, good point! The paper I am responsible for is targeted at smaller datasets, but I will propose this to the authors of the second paper :)
bilsbie 4y ago
I think K nearest neighbor is an underrated learning model. Assuming you can define a good distance measurement how cool is it to use every piece of data. And with no training. I wonder how they compare against tree based models on tabular data just in accuracy.
- teruakohatu 4y ago
  I like using KNN as much as the next person but tabular data can be quite wide, and KNN suffers from the curse of dimensionality.
  - CornCobs 4y ago
    The next course of action would be to split your distance calculation into multiple steps - a few dimensions at a time. Then we need to decide on an order to these comparisons - what about using entropy derived from the training data? Then as a further improvement, what if the order need not be constant, depending on which group the data point is closer to? And what if not all dimensions need to be considered? Before you know it, woah! You've reinvented trees! This comment is meant as a joke
    - bilsbie 4y ago
      That’s a great way to explain it though.
BenoitEssiambre 4y ago
I like decision trees and this helps support my case for using them. I often go even further and don't use an algorithm to build the trees but build trees myself along intuitive causal lines and use the data to train its parameters. I sometimes build a few models manually and see which fits better with the data. Prior knowledge can prevent the pitfalls of automatically built models. Trees may be better than NNs because they overfit less but you can overfit even less with a bespoke model. For example, I've seen an automatically generated tree made to tune the efficiency of a factory end up using as a main feature "be after a specific date" because a machine was upgraded on that date and so the learning algorithm latched on to that unactionable piece of data as a main predictor for the model. This was an easy fix, to not feed timestamp data to the model but there are lots of more subtle cases like this and I've seen people spend much time cleaning and "tweaking" the input data to get the answers they want out of their ML models. If you have to try to make your ML model behave by manually selecting what data to feed it, you might as well go all the way and build a clean causal model yourself that reflect your priors and domain knowledge about the subject. I have an ML background but I often get more performance out of my models by doing something along the lines of what a Bayesian statistician would do. Of course with highly dimensional data like pixels in images you almost have no choice to use NNs. There's no way to hand-build these models.
- hackerlight 4y ago
  I want to know more about your process. How do you optimise the split point. How do you choose which feature to split on first
  - BenoitEssiambre 4y ago
    I start with common sense priors or "intuition" but also sometimes write code to try different combinations and assess accuracy. For example, not too long ago I was trying to estimate and predict ATM usage and considered using days of the week as a feature. The simplest case is a single node tree that assumes every day of the week has similar usage. You just calculate mean and variance. But then, a lot of ATMs have different usage patterns on weekends than weekdays. So I considered a split into two groups, calculating different mean/variance for weekend days and week days. I could have tried different groupings of days but those two were easy to understand for users, covered most cases I was interested in with less risk of overfitting given the amount of data that I was working with (with more data I might have considered more splits even take into account special days like holidays). The more splits the more data you need to train the model, especially if you want not just a point estimate but an idea of variance (more parameters to estimate require more data). The decision whether to split or not can be done on a per ATM basis cross validating the splitting strategy by assessing a fleet of ATMs. People optimize different things in these kinds of problem. Here https://www.saedsayad.com/decision_tree_reg.htm https://www.saedsayad.com/decision_tree_reg.htm for example they try to reduce prediction variance. Oftentimes decision trees try to optimize information gain or gain ratio (https://en.wikipedia.org/wiki/Information_gain_(decision_tree) https://en.wikipedia.org/wiki/Information_gain_(decision_tre...) Gain ratio seems like a pretty ad-hoc technique to me so to prevent over-fitting and select the correct model (to split or not), I sometimes use a relative entropy penalty on the trained parameters kinda like described here: http://www.inference.org.uk/mackay/itprnn/ps/343.355.pdf http://www.inference.org.uk/mackay/itprnn/ps/343.355.pdf. I've never used it but apparently if you have problems where you can't calculate that parameter relative entropy, there are approximation techniques such as https://en.wikipedia.org/wiki/Bayesian_information_criterion https://en.wikipedia.org/wiki/Bayesian_information_criterion. Then there's things I sometimes do with hyperparameters to improve performance like mixing into individual ATM data fake data points that represent the prior for a wider group or class of ATMs. Again using common sense to adjust those priors (and always cross validating my tweaks against real data).
_pastel 4y ago
It's baffling to me how little research attention there has been to improving tree-based methods, considering their effectiveness. For example, LightGBM and XGBoost allow some regularization terms, but the variance/bias is still mostly controlled by globally setting the max depth and max node count (and then parameter searching to find good settings). Surely there must be more powerful and sophisticated ways of deciding when to stop building each tree than counting the number of nodes? If this was neural nets there would be a hundred competing papers proposing different methods and arguing over their strengths and weaknesses. I'm not sure whether the problem is that neural nets are just fundamentally more sexy, or that in order to make SOTA improvements in GBMs you need to dive into some gnarly C++. Probably both.
- micro_cam 4y ago
  There are a fair number of papers (start with Dart/dropout, Bart (bayesian sampling of the whole gbm) but they start to look like global optimization problems and part of the reasons trees work so well is that the local greedy optimization can be made super fast on modern cpu caches. So even if you can fit a more compact forest that performs well through clever regularization its usually better/faster in practice to grow more simple trees with more randomization and let overfitting average out.
- mistrial9 4y ago
  > LightGBM not improving, meanwhile 8-figure budgets builds GPU and auto-logins.. My take? management agenda to build plug-and-play researchers (humans on jobs), rather than domain specialists. DeepLearning fits that description with all-plumbing, all-the-time.. domain specialists want graduate school, weekends and health benefits..
  - nl 4y ago
    Is this supposed to be an ironic take on the "plug-and-play HN comment blaming cost cutting management for everything"? Because it's not this. Deep learning researchers are specialists in their specific technologies. For example recent advances in transformers not withstanding, it takes quite a long time for say someone who knows how to build deep learning models on images using CNNs to come up to speed on how build good natural language processing models.
orasis 4y ago
We use XGBoost as the core learner for reinforcement learning at https://improve.ai https://improve.ai despite the popularity of neural networks in academia. With tabular or nested data a human has already done a lot of work to organize that data in a machine friendly form - much of the feature engineering is performed by the data schema itself.
- ramraj07 4y ago
  Your app sounds amazing, as is the ethics model of it. Can’t wait to test it out!
  - orasis 4y ago
    Thank you. I’m surprised that anyone read that part. Thank you for noticing.
- Bedon292 4y ago
  Your quick start link just links to the iOS SDK on GitHub. Perhaps things would make more sense if that were available, but right now no idea how to use your tool. And it appears that there are some html docs in the python repo, but they don't seem to be hosted anywhere. So its not clear how to use it either.
  - orasis 4y ago
    Thanks so much for pointing this out. I’ll get that fixed!
wills_forward 4y ago
Does anyone see explainability as another good reason to trees on tabular data, for which I think users would expect more digestable outputs?
- oofbey 4y ago
  The kinds of trees that come out of these algorithms are so huge they really aren’t any more interpretable than a NN.
  - CornCobs 4y ago
    Not exactly. These tree models are ensemble methods, meaning they comprise several trees. Each individual tree may be small, but it is difficult to pinpoint explanations when that tree is but one amongst a forest
- aimor 4y ago
  Yes, I've been looking at using decision trees for explaining models that are difficult to understand. Currently seeing useful results on real data sets. If you're interested, I've implemented parts of TREPAN [1] and it's very approachable. However it's also important to have interpretable features which is a whole other thing. [1] https://research.cs.wisc.edu/machine-learning/shavlik-group/craven.thesis.pdf https://research.cs.wisc.edu/machine-learning/shavlik-group/...
hatmatrix 4y ago
I didn't get to what extent they compared capability of these methods for extrapolation, which tree-based methods are not really suited for.
westurner 4y ago
That should only be the case when the variables are conditionally independent?
bigbillheck 4y ago
It seems to me that one crucial difference between tabular data and images or text is that in the latter there is a huge amount of structure available. In text, all the words depend on all their neighbor words, and images tend to be well-approximated by low-rank things in one form or another. Tabular data doesn't have any of that.
- officehero 4y ago
  Exactly. I get why people like to compare things like NN's and trees because it's a good way to learn. But it doesn't take too much understanding to see that they both have strengths/weaknesses and are suitable for different problems.
savant_penguin 4y ago
A important point is that it's an absolute pain in the ass to preprocess tabular data for neural networks. Categorical > one hot encoding > deal with new categories in test time (sklearn does this, but it's really slow and clunky) Numerical > either figure it out the data distribution for each column and normalize by that or normalize everything by z score. Found an outlier?? Oops, every feature collapsed to 0 Can you that for 10 features? Sure, now try it again with 500, it's not fun Ok, now that you've done all that you can begin training and possibly get some reasonable result. Compare that with tree models: data>model>results
- johnthewise 4y ago
  You could encode categorical variables into embeddings, they are more natural to deal that way. https://tech.instacart.com/deep-learning-with-emojis-not-math-660ba1ad6cdc https://tech.instacart.com/deep-learning-with-emojis-not-mat...
nicodjimenez 4y ago
Deep learning doesn't actually work unless you're doing vision or NLP.
- eutectic 4y ago
  What an inane comment.
- visarga 4y ago
  And RL, graphs, audio/speech, approximating simulations (weather), etc.
  - nicodjimenez 4y ago
    Ok true I forgot speech recognition

315 points by isolli 4y ago

I think one of the issues is that there's no pretrained universal spreadsheet models (that I'm aware of, granted I don't do much work with tabular data) equivalent to ImageNet based models that you can use as a base and then transfer learn on top of that.

LittlePeter 4y ago

What is tabular data precisely? I can represent an image as a table of RGB values. I can represent hierarchical data as a table of unnested values.

coffee_am 4y ago

I can't explain it, but I help maintain TensorFlow Decision Forests [1] and Yggdrasil Decision Forests [2], and in an AutoML system at work that trains models on lots of various users data, decision forest models gets selected as best (after AutoML tries various model types and hyperparameters) somewhere between 20% to 40% of the times, systematically. It's pretty interesting. Other ML types considered are NN, linear models (with auto feature crossings generation), and a couple of other variations. [1] https://github.com/tensorflow/decision-forests https://github.com/tensorflow/decision-forests [2] https://github.com/google/yggdrasil-decision-forests https://github.com/google/yggdrasil-decision-forests

onasta 4y ago
Super interesting! Do you know the kind of data that it's usually used for? And in the remaining 80% to 60%, do NNs acccount for a large portion of the best models? Bonus question: are the stats you're mentioning publically available?
- coffee_am 4y ago
  Sadly (but correctly) nothing is public, no one ever sees any data, it's a service. Pure FNN (feedforward NN) models, if I recall correctly is also ~30 to 40%. Since the server doesn't work for all types of data, and probably folks that are experts in ML would do their own hyperparameter tuning, and custom models, this leads to the bias on the type of datasets that are compete. But this share have been consistent over many months of various unrelated datasets, I believe.

fennecfoxy 4y ago

Are the tree based models being provided with data that's "kind of like" the data they were designed for/at training time for the deep learning model? Or "exactly like"? I think that's perhaps the difference?

throwaway0asd 4y ago

The answer is extremely straightforward: less computation. That’s less computation when traversing nodes in a set which can result in either an exponential or logarithmic difference depending upon the data and means of traversal. This holds true almost universally.

rdedev 4y ago

I've mostly come to the conclusion that DL is not going to beat tree based models especially when it comes to training speed. What I am now looking at is if it's possible to feed in world knowledge from models like beer as features to XGBoost but without resorting to creating templates for table rows

muyuu 4y ago

why would anyone assume deep learning to beat everything at everything, even applied relatively naively?

marcodiego 4y ago

Wait until we see deep learning AI creating tree-based models. /s

PartiallyTyped 4y ago
Technically you can transform any finite / not runtime defined computation graph into a "tree" model.
- optimalsolver 4y ago
  See symbolic regression and genetic programming.
uoaei 4y ago
Not actually that complicated of an approach! Hard to implement correctly, which is why it's not more popular right now.
karmasimida 4y ago
triggering co-pilot to use xgboost might count as one today lol?

deleted 4y ago

[deleted]

isolli 4y ago

Related, an article from 2019 [0] on how neural network finally beat statistical models (e.g. ARIMA) in time-series forecasting. [0] https://towardsdatascience.com/n-beats-beating-statistical-models-with-neural-nets-28a4ba4a4de8 https://towardsdatascience.com/n-beats-beating-statistical-m...

mr_toad 4y ago
I had an econometrics professor who would have objected to calling ARIMA models statistics. He though they were no better than data mining.
nomel 4y ago
I can only assume this was achieved years ago, as a black project in the financial market.

spywaregorilla 4y ago

Tree based models seem like obvious approaches that should just work. It's more or less how a human would parse a problem. Throw some X and O's on a 2d plot. Draw lines to partition it into X sections and O sections. That's a tree based model.

havercosine 4y ago
I actually agree with this take! Deep learning models shine in problem domains where the gap between the data representation to the way how humans parse it is large. In images, for example, we just have RGB matrix but what human vision does is very complex. So representation learning done by deep learning models tries to close the gap. For lot of tabular data, human's don't go from raw representation to a mental model quickly. We in fact follow a model similar to tree based classifier: look at how a variable is splitting the output, compare averages across groups etc.
melenaboija 4y ago
Now you can contact the authors and tell them to replace the publication by your 4 lines. And tree based models is not what you described, at most that could be a tree classifier. EDIT: First of all sorry for my comment, I think you did not like it. My point is that what you state as obvious on how tree based methods work is most probably not what makes them powerful but the fact that you have a bunch of them. To me if there is some intuition, statistics is more prevalent than separating spaces in hyperplanes. Here you can see the boundaries of a random forest compared to a single tree, the more dimensions you add the more blurry and unintuitive in terms of hyperplanes it gets: https://scikit-learn.org/stable/_images/sphx_glr_plot_classifier_comparison_001.png https://scikit-learn.org/stable/_images/sphx_glr_plot_classi...
- spywaregorilla 4y ago
  - abduhl 4y ago
    So it sounds like you’d need your 4 lines of pseudo-code from your first post and then another few lines of comments to describe how your 4 lines replace the authors’ publication? Something like “All tree models are partitions of the space based on training data, and the extension to any particular tree model is an obvious exercise left to the reader”?

jwilber 4y ago

If you’re interested in how tree-based models work, I wrote an interactive explanation on decision trees here: https://mlu-explain.github.io/decision-tree/ https://mlu-explain.github.io/decision-tree/ and random forests here: https://mlu-explain.github.io/random-forest/ https://mlu-explain.github.io/random-forest/ It’s also worth noting that a recentish paper shows neural networks can perform well on tabular data if well-regularized: https://arxiv.org/abs/2106.11189v1?utm_source=jesper&utm_medium=email&utm_campaign=coming-up-with-weather-puns-is-a-breeze https://arxiv.org/abs/2106.11189v1?utm_source=jesper&utm_med...

isabellat 4y ago
Really nice interactive explanations!
lnenad 4y ago
That was super easy to digest, thank you!

moffkalast 4y ago

Well iirc a convenient trait of a random forest classifier is that it cannot overfit onto learning data. Something that's not exactly true for deep learning.

taeric 4y ago
I'm curious on this claim. Feels like any model can over fit.
- HelloNurse 4y ago
  With algorithms that, for each additional input entry, either add a predictable amount of complexity to improve fitness (e.g. another tree split) or decide that the model is already fit enough, one needs to do something deliberate or very stupid to overfit data compare to neural networks where overfitting is hard to detect, let alone prevent quantitatively, and that are sometimes designed to overfit in order to interpolate "intelligently" between samples.
- marcosdumay 4y ago
  A model can only overfit if it has enough parameters. But indeed, any model with enough parameters can overfit. The thing about deep learning is that the number of parameters is astronomical.
  - disgruntledphd2 4y ago
    Hence, the interesting question is why deep learning models don't overfit all the time.
    - DougBTX 4y ago
      This is from 2019 so I suspect there’s a more recent paper now, but this is interesting: https://openai.com/blog/deep-double-descent/ https://openai.com/blog/deep-double-descent/
      - 4y ago

VoVAllen 4y ago

It's all about features and data scale. Recommendation system itself is actually a large table, DL method already proved effectiveness there. Let's say if you have text in your tabular data. Tree model(with traditional method such as tfidf) will do much worse than the transformer-based model. DL always suffers from inadequate data, so if there's no enough data or inductive biases, tree model can be a better choice in that way.

Permit 4y ago

> Results show that tree-based models remain state-of-the-art on medium-sized data (∼10K samples) even without accounting for their superior speed. Is that really "medium"? That seems very small to me. MNIST has 60,000 samples and ImageNet has millions. I think the title overstates the findings. I'd be interested to hear how these methods compare on much larger datasets. Is there a threshold at which deep learning outperforms tree-based models? Edit: They touch on this in the appendix: > A.2.2 Large-sized datasets > We extend our benchmark to large-scale datasets: in Figures 9, 10, 11 and 12, we compare the results of our models on the same set of datasets, in large-size (train set truncated to 50,000 samples) and medium-size (train set truncated to 10,000 samples) settings. > We only keep datasets with more than 50,000 samples and restrict the train set size to 50,000 samples (vs 10,000 samples for the medium-sized benchmark). Unfortunately, this excludes a lot of datasets, which makes the comparison less clear. However, it seems that, in most cases, increasing the train set size reduces the gap between neural networks and tree-based models. We leave a rigorous study of this trend to future work.

mochomocha 4y ago
I've put in production numerous models with millions of tabular data points and a 10^5-10^6 feature space where tree-based models (or FF nets) outperform more complex DL approaches.
- whymauri 4y ago
  I'll chime in with billions of data points and 100-300 feature space with some smart feature engineering outperforming DL in runtime/compute (by orders of magnitude) and performance. But the domain was very specific and everything prior to the tree was doable with matrix operations, with the tree model summarizing a mixture of experts that chose optimal features.
  - bloudermilk 4y ago
    Damn, this sounds fascinating! Have you shared more anywhere eelse?
- 4y ago

CapmCrackaWaka 4y ago

I have a theory - tree based models require minimal feature engineering. They are capable of handling categorical data in principled ways, they can handle the most skewed/multimodal/heteroskedastic continuous numeric data just as easily as a 0-1 scaled normal distribution, and they are easy to regularize compared to a DL model (which could have untold millions of possible parameter combinations, let alone getting the thing to train to a global optimum). I think if you spent months getting your data and model structure to a good place, you could certainly get a DL model to out-perform a gradient boosted tree. But why do that, when the GBT will be done today?

jb_s 4y ago
do you reckon it's possible to somehow transfer learn from a GBT to a NN ?
a-dub 4y ago
this is along the lines of my thinking. people organize and summarize data before throwing it into spreadsheets, where deep learning models do their thing by generating new representations from raw data. in a sense, most data in spreadsheets is compressed and deep learning models prefer to find their own compression that best suits the task at hand. or in human terms: "these spreadsheets are garbage. i can't work with this. can you bring me the raw data please?" :)
deleted 4y ago
[deleted]
username_exists 4y ago
occam's razor
drzoltar 4y ago
I think another aspect is that most modern GBT models prefer the entire dataset to be in memory, thereby doing a full scan of the data for each iteration to calculate the optimal split point. That’s hard to compete with if your batch size is small in a NN model.

fdgsdfogijq 4y ago

Because high tabular data doesnt have enough complexity compared to language or images.

jmmcd 4y ago

A great paper and an important result. However, it omits to cite the highly relevant SRBench paper from 2021, which also carefully curates a suitable set of regression benchmarks and shows that Genetic Programming approaches also tend to be better than deep learning. https://github.com/cavalab/srbench https://github.com/cavalab/srbench cc u/optimalsolver

ersiees 4y ago

Our lab works on changing this. I think it might still take some years for a full solution, but so far we are successful with NNs for small datasets with meta-learning (https://arxiv.org/abs/2207.01848 https://arxiv.org/abs/2207.01848) and large datasets with regularisation (https://arxiv.org/abs/2106.11189 https://arxiv.org/abs/2106.11189). The first is very new, but the second is also cited in this paper, but they didn’t run it as a baseline.

freemint 4y ago
Tree based algorithms have the advantage that global or robust optimization of them is possible. Do your approaches offer similar guarantees?
- ersiees 4y ago
  The first work, I linked, arguably gets around this issue completely. There we train a neural network to approximate Bayesian prediction directly. It accepts the dataset as input. Thus, there is no optimisation procedure per dataset, just a forward pass. So far, this seems to be pretty stable.
  - freemint 4y ago
    Just because you optimize network parameters to fit a surrogate model of the data doesn't mean you are not doing local optimization without guarantees on robustness or global optimality w.r.t. network parameters to fit the surrogate. Am i missing something?
civilized 4y ago
Why not run your own method on their benchmark? If your method is general-purpose, it should be very easy. And it would be very impressive to see your method succeed on a task that was not chosen by you to write your paper.
- ersiees 4y ago
  Yes, good point! The paper I am responsible for is targeted at smaller datasets, but I will propose this to the authors of the second paper :)

bilsbie 4y ago

I think K nearest neighbor is an underrated learning model. Assuming you can define a good distance measurement how cool is it to use every piece of data. And with no training. I wonder how they compare against tree based models on tabular data just in accuracy.

teruakohatu 4y ago
I like using KNN as much as the next person but tabular data can be quite wide, and KNN suffers from the curse of dimensionality.
- CornCobs 4y ago
  The next course of action would be to split your distance calculation into multiple steps - a few dimensions at a time. Then we need to decide on an order to these comparisons - what about using entropy derived from the training data? Then as a further improvement, what if the order need not be constant, depending on which group the data point is closer to? And what if not all dimensions need to be considered? Before you know it, woah! You've reinvented trees! This comment is meant as a joke
  - bilsbie 4y ago
    That’s a great way to explain it though.

BenoitEssiambre 4y ago

I like decision trees and this helps support my case for using them. I often go even further and don't use an algorithm to build the trees but build trees myself along intuitive causal lines and use the data to train its parameters. I sometimes build a few models manually and see which fits better with the data. Prior knowledge can prevent the pitfalls of automatically built models. Trees may be better than NNs because they overfit less but you can overfit even less with a bespoke model. For example, I've seen an automatically generated tree made to tune the efficiency of a factory end up using as a main feature "be after a specific date" because a machine was upgraded on that date and so the learning algorithm latched on to that unactionable piece of data as a main predictor for the model. This was an easy fix, to not feed timestamp data to the model but there are lots of more subtle cases like this and I've seen people spend much time cleaning and "tweaking" the input data to get the answers they want out of their ML models. If you have to try to make your ML model behave by manually selecting what data to feed it, you might as well go all the way and build a clean causal model yourself that reflect your priors and domain knowledge about the subject. I have an ML background but I often get more performance out of my models by doing something along the lines of what a Bayesian statistician would do. Of course with highly dimensional data like pixels in images you almost have no choice to use NNs. There's no way to hand-build these models.

hackerlight 4y ago
I want to know more about your process. How do you optimise the split point. How do you choose which feature to split on first
- BenoitEssiambre 4y ago
  I start with common sense priors or "intuition" but also sometimes write code to try different combinations and assess accuracy. For example, not too long ago I was trying to estimate and predict ATM usage and considered using days of the week as a feature. The simplest case is a single node tree that assumes every day of the week has similar usage. You just calculate mean and variance. But then, a lot of ATMs have different usage patterns on weekends than weekdays. So I considered a split into two groups, calculating different mean/variance for weekend days and week days. I could have tried different groupings of days but those two were easy to understand for users, covered most cases I was interested in with less risk of overfitting given the amount of data that I was working with (with more data I might have considered more splits even take into account special days like holidays). The more splits the more data you need to train the model, especially if you want not just a point estimate but an idea of variance (more parameters to estimate require more data). The decision whether to split or not can be done on a per ATM basis cross validating the splitting strategy by assessing a fleet of ATMs. People optimize different things in these kinds of problem. Here https://www.saedsayad.com/decision_tree_reg.htm https://www.saedsayad.com/decision_tree_reg.htm for example they try to reduce prediction variance. Oftentimes decision trees try to optimize information gain or gain ratio (https://en.wikipedia.org/wiki/Information_gain_(decision_tree) https://en.wikipedia.org/wiki/Information_gain_(decision_tre...) Gain ratio seems like a pretty ad-hoc technique to me so to prevent over-fitting and select the correct model (to split or not), I sometimes use a relative entropy penalty on the trained parameters kinda like described here: http://www.inference.org.uk/mackay/itprnn/ps/343.355.pdf http://www.inference.org.uk/mackay/itprnn/ps/343.355.pdf. I've never used it but apparently if you have problems where you can't calculate that parameter relative entropy, there are approximation techniques such as https://en.wikipedia.org/wiki/Bayesian_information_criterion https://en.wikipedia.org/wiki/Bayesian_information_criterion. Then there's things I sometimes do with hyperparameters to improve performance like mixing into individual ATM data fake data points that represent the prior for a wider group or class of ATMs. Again using common sense to adjust those priors (and always cross validating my tweaks against real data).

_pastel 4y ago

It's baffling to me how little research attention there has been to improving tree-based methods, considering their effectiveness. For example, LightGBM and XGBoost allow some regularization terms, but the variance/bias is still mostly controlled by globally setting the max depth and max node count (and then parameter searching to find good settings). Surely there must be more powerful and sophisticated ways of deciding when to stop building each tree than counting the number of nodes? If this was neural nets there would be a hundred competing papers proposing different methods and arguing over their strengths and weaknesses. I'm not sure whether the problem is that neural nets are just fundamentally more sexy, or that in order to make SOTA improvements in GBMs you need to dive into some gnarly C++. Probably both.

micro_cam 4y ago
There are a fair number of papers (start with Dart/dropout, Bart (bayesian sampling of the whole gbm) but they start to look like global optimization problems and part of the reasons trees work so well is that the local greedy optimization can be made super fast on modern cpu caches. So even if you can fit a more compact forest that performs well through clever regularization its usually better/faster in practice to grow more simple trees with more randomization and let overfitting average out.
mistrial9 4y ago
> LightGBM not improving, meanwhile 8-figure budgets builds GPU and auto-logins.. My take? management agenda to build plug-and-play researchers (humans on jobs), rather than domain specialists. DeepLearning fits that description with all-plumbing, all-the-time.. domain specialists want graduate school, weekends and health benefits..
- nl 4y ago
  Is this supposed to be an ironic take on the "plug-and-play HN comment blaming cost cutting management for everything"? Because it's not this. Deep learning researchers are specialists in their specific technologies. For example recent advances in transformers not withstanding, it takes quite a long time for say someone who knows how to build deep learning models on images using CNNs to come up to speed on how build good natural language processing models.

orasis 4y ago

We use XGBoost as the core learner for reinforcement learning at https://improve.ai https://improve.ai despite the popularity of neural networks in academia. With tabular or nested data a human has already done a lot of work to organize that data in a machine friendly form - much of the feature engineering is performed by the data schema itself.

ramraj07 4y ago
Your app sounds amazing, as is the ethics model of it. Can’t wait to test it out!
- orasis 4y ago
  Thank you. I’m surprised that anyone read that part. Thank you for noticing.
Bedon292 4y ago
Your quick start link just links to the iOS SDK on GitHub. Perhaps things would make more sense if that were available, but right now no idea how to use your tool. And it appears that there are some html docs in the python repo, but they don't seem to be hosted anywhere. So its not clear how to use it either.
- orasis 4y ago
  Thanks so much for pointing this out. I’ll get that fixed!

wills_forward 4y ago

Does anyone see explainability as another good reason to trees on tabular data, for which I think users would expect more digestable outputs?

oofbey 4y ago
The kinds of trees that come out of these algorithms are so huge they really aren’t any more interpretable than a NN.
- CornCobs 4y ago
  Not exactly. These tree models are ensemble methods, meaning they comprise several trees. Each individual tree may be small, but it is difficult to pinpoint explanations when that tree is but one amongst a forest
aimor 4y ago
Yes, I've been looking at using decision trees for explaining models that are difficult to understand. Currently seeing useful results on real data sets. If you're interested, I've implemented parts of TREPAN [1] and it's very approachable. However it's also important to have interpretable features which is a whole other thing. [1] https://research.cs.wisc.edu/machine-learning/shavlik-group/craven.thesis.pdf https://research.cs.wisc.edu/machine-learning/shavlik-group/...

hatmatrix 4y ago

I didn't get to what extent they compared capability of these methods for extrapolation, which tree-based methods are not really suited for.

westurner 4y ago

That should only be the case when the variables are conditionally independent?

bigbillheck 4y ago

It seems to me that one crucial difference between tabular data and images or text is that in the latter there is a huge amount of structure available. In text, all the words depend on all their neighbor words, and images tend to be well-approximated by low-rank things in one form or another. Tabular data doesn't have any of that.

officehero 4y ago
Exactly. I get why people like to compare things like NN's and trees because it's a good way to learn. But it doesn't take too much understanding to see that they both have strengths/weaknesses and are suitable for different problems.

savant_penguin 4y ago

A important point is that it's an absolute pain in the ass to preprocess tabular data for neural networks. Categorical > one hot encoding > deal with new categories in test time (sklearn does this, but it's really slow and clunky) Numerical > either figure it out the data distribution for each column and normalize by that or normalize everything by z score. Found an outlier?? Oops, every feature collapsed to 0 Can you that for 10 features? Sure, now try it again with 500, it's not fun Ok, now that you've done all that you can begin training and possibly get some reasonable result. Compare that with tree models: data>model>results

johnthewise 4y ago
You could encode categorical variables into embeddings, they are more natural to deal that way. https://tech.instacart.com/deep-learning-with-emojis-not-math-660ba1ad6cdc https://tech.instacart.com/deep-learning-with-emojis-not-mat...

nicodjimenez 4y ago

Deep learning doesn't actually work unless you're doing vision or NLP.

eutectic 4y ago
What an inane comment.
visarga 4y ago
And RL, graphs, audio/speech, approximating simulations (weather), etc.
- nicodjimenez 4y ago
  Ok true I forgot speech recognition