27 ms·
Regularization is all you need: simple neural nets can excel on tabular data
- toxik 5y agoWould you please stop it with the "$whatever is all you need"?
- The_rationalist 5y agoSometimes it feels like papers authors consider their audience as children.
- toxik 5y agoIt's such a cheap trick too, basically clickbait.
- bonoboTP 5y agoBut it works, for example it got to the top of HN.
- ausbah 5y agoto be fair many of the tasks of machine learning are things children do like telling if something is a dog, coloring in a a picture, solving a maze, etc.
- edmundsauto 5y agoCommunication is based on the lowest common denominator. The best communication can be understood by a wide range of audiences. Have you considered that you weren't the intended audience?
- jgalt212 5y agoI don't know about that. It's probably driven by the need to for a short attention* gathering title. * I see what I did there.
- bonoboTP 5y agoThe field is certainly moving in a direction of infantilization due to the social media attention economy. Your paper competes with cute flashy cat gifs etc. on Twitter. And Twitter is ridiculously important in AI/ML, except perhaps above 50-60 years of age.
- touisteur 5y agoA little whimsy doesn't hurt... Nobody said science and research had to be dry and witless.
- jasonzemos 5y agoWhat if the paper which finally gives us The Singularity is titled "All you need is all you need?"
- prionassembly 5y ago"All you need considered harmful"
- joe_the_user 5y ago"The Unreasonable Effectiveness of All You Need Is Considered Harmful" -- All interpretations worth considering...
- ayyy 5y ago"The Unreasonable Effectiveness of All You Need Is Considered Harmful for Fun and Profit"
- deleted 5y ago[deleted]
- philipswood 5y agoFANG doesn't want you to know how this one weird trick about the unreasonable effectiveness of all you need considered harmful for fun and profit and you will never guess what happened next.
- qolop 5y agoWe live in a society where FANG doesn't want you to know how this one weird trick about the unreasonable effectiveness of all you need considered harmful for fun and profit and you will never guess what happened next in my social experiment gone wrong.
- prionassembly 5y agoWe live in a society. https://asemic-horizon.com/2019/12/18/we-live-in-a-society/ https://asemic-horizon.com/2019/12/18/we-live-in-a-society/
- melling 5y agoIsn’t it a reference to the “Attention is All You Need” machine learning paper? https://arxiv.org/abs/1706.03762 https://arxiv.org/abs/1706.03762
- toxik 5y agoYes, and this is the fourth or fifth paper I see copying that format. The attention paper was pretty damn significant. This won’t be, because it just shows that hyperparameters are important. We know that.
- lux 5y agoI often interpret that pattern as a reference to the lyric "love is all you need" by the Beatles (e.g., an attempt at being playful and relatable), but that may be my musical bias. Either way, totally agree. Overused, almost always incorrect, and easily misconstrued especially by people who don't speak English as their first language.
- jackylupino 5y agoIf the paper would not have used this clickbait title, it would probably go unnoticed and noone would pay attention? Sad truth about sensation-oriented modern lives :(
- nightcracker 5y agoPaper is missing the control: how good is this 'cocktail of regularization' when applied to traditional methods like XGBoost? At best you can claim the result here that neural networks with regularization methods can beat traditional methods without it, but to be apples to apples both methods must have access to the same 'cocktail of regularization'.
- audi0slave 5y agoThis paper compares xgboost vs nueural nets and an ensemble [Tabular Data: Deep Learning is Not All You Need] https://arxiv.org/abs/2106.03253 https://arxiv.org/abs/2106.03253
- ipsum2 5y agoFrom the paper: > This paper is the first to provide compelling evidence that wellregularized neural networks (even simple MLPs!) indeed surpass the current state-of-the-art models in tabular datasets, including recent neural network architectures and GBDT (Section 6). > Next, we analyze the empirical significance of our well-regularized MLPs against the GBDT implementations in Figure 2b. The results show that our MLPs outperform both GBDT variants (XGBoost and auto-sklearn) with a statistically significant margin. They test against XGBoost, GBDT Auto-sklearn, and others. Did you read the paper?
- nightcracker 5y ago> They test against XGBoost, GBDT Auto-sklearn, and others. Did you read the paper? Yes. Did you read my comment? They compare NN + Cocktail vs. vanilla XGB. They don't compare NN + Cocktail vs. XGB + Cocktail. To make it crystal clear, if I wrote a paper "existing medicine A enhanced with novel method B is more effective than existing medicine C" and I did not include the control "C + B" (assuming if relevant, which is the case here), that'd be bad science. It's very much possible that novel method B is doing the heavy lifting and A isn't all that relevant. s/A/NN, s/B/Cocktail, s/C/XGBoost.
- ipsum2 5y ago
- amilios 5y agoThis is interesting, but the paper still notes that in most "real-life" applications people will likely still prefer gradient-boosted trees, just because you need to allocate significant computation to hyperparameter tuning even in the case of the MLP-with-regularization-cocktail. For just getting something off the ground quickly based on tabular data, GBDTs are still unbeatable.
- quantum_mcts 5y agoAn overlooked advantage of the MLP is that it is differentiable. Essentially, one trades the extra CPU for a classifier that one can pipe gradients through. That can be extremely useful in larger NN architectures.
- deleted 5y ago[deleted]
- not_jd_salinger 5y ago> GBDTs are still unbeatable. You'd be surprised how many times I've replaced a GBDT with logistic regression and had negligible drop off in model performance with a dramatic improvement in both training time as well as debugging and fixing production models. I've had plenty of cases where a bit of reasonable feature transformation can get a logistic model to outperform a gbdt. Any non-linearity your picking up with a GBDT can often easily be captured with some very simple feature tweaking. My experience has been that GBDTs are only particularly useful in Kaggle contests, where minuscule improvements in an arbitrary metric are valuable and training time and model debugging are completely unimportant. There are absolutely cases where NNs can go places that logistic regression can't touch (CV and NLP), but I have yet to see a real world production pipeline where GBDT provides enough improvement over Logistic Regression, to throw out all of the performance and engineering benefits of linear models.
- kimukasetsu 5y agoI strongly agree with this. Not to mention parameter interpretability and, in the case of Bayesian models, uncertainty estimates and convergence diagnostics. Such things are very important when making decision under uncertainty. Kaggle competitions and empirical benchmarks are very biased samples of model performance in real life. I feel these two things often influence too much the course of Machine Learning research and communities, and this is not good. Most ML researchers and pratictioners are barely aware of the latest advances in parametric modelling, which is a shame. Multilevel models allow you to model response variables with explicit dependent structures. This is done through random (sometimes hierarchical) effects constrained by variance parameters. These parameters regularize the effects themselves and converge really well when fitting factors with high cardinality. Also, multilevel models are very interesting when it comes to the bias-variance tradeoff. Having more levels in a distribution of random effects actually DECREASES [1] overfitting, which is fascinating. [1] https://m-clark.github.io/posts/2019-05-14-shrinkage-in-mixed-models/ https://m-clark.github.io/posts/2019-05-14-shrinkage-in-mixe...
- lern_too_spel 5y agoA lot of these papers can be titled "Tuning hyperparameters on the evaluation dataset is all you need." I see a few cases in this paper.
- joe_the_user 5y agoFrom paper: Tabular datasets are the last "unconquered castle" for deep learning, with traditional ML methods like Gradient-Boosted Decision Trees still performing strongly even against recent specialized neural architectures. My but this statement seems more than a little grandiose. Never mind that XGBoost still does well on a substantial portion of ML challenges (supposedly). The bigger problematic is that there's a confusion of maps and territories in this way of talking of machine learning. The field of ML has made a certain level of palpable progress by having created a number of challenges and benchmarks and then doing well on them. But success on a benchmark isn't necessarily the same as a success the "task" broadly. An NLP test doesn't imply mastering real language, a driving benchmark doesn't imply master over the road driving. etc. Notably, success on a benchmark also "isn't nothing". In a situation like the game of go, the possibilities can be fully captured "in the lab" and success at tests indeed became success against humans. But with driving or language, things are much more complicated. What I would say is that benchmark success seems to produce at least a situation where the machine can achieve human-like performance for some neighborhood (or tile or etc) limited in time, space and subject. Of course, driving is the poster-child for the limitations of "works most of the time" but lot of "intelligent" activities require an ongoing thread of "reasonableness" aside from having an immediate logic. Anyway, it would be nice if our ML folks looked at this stuff more as a beginning than as a situation where they're poised on success.
- whatshisface 5y agoIf you want to use NNs on tabular data look up the work that's being done on point clouds. They both share the same major symmetry over permutations.
- eachro 5y agoOh that's interesting. Can you say more? Haven't seen much relating the two topics before.
- whatshisface 5y agoA row in a table of data is an point in R^n. I'm not sure how much there is to write about it other than to say, that's a point cloud.
- asdf3331 5y agoNo, that's a point.
- whatshisface 5y agoThe table's the point cloud. I guess if you want to be really pedantic about grammar then I should point out that a set of one item is still a set. :-)
- asdf3331 5y agoTrue - I misread your post. Your first post was intriguing but the second was dismissive. Surely there's more to be said than data sets are point clouds? Images are points in R^N too, right?
- whatshisface 5y agoIf your unit of recognition was a set of images, and not an image alone, then you would have permutation symmetry and want to use point cloud techniques to design your first layer. So yes images are points in R^N.
- zozbot234 5y agoRidiculous clickbait title. How the heck did this neural network "can" Excel wrt. this tabular dataset? What's the underlying objective and the evaluation benchmark? And what feature of Excel was this tested against in the first place? I suppose the follow up will be titled "One Weird Trick is All You Need To Destroy SOTA On This Dataset!"
- TFortunato 5y agoNot Excel: the spreadsheet software, Excel: the verb. They aren't saying that they "canned" Excel (the software), they are saying that neural nets have the potential to perform well on tasks involving tabular data that are traditionally performed by other ML techniques.
- bno1 5y agoI, for one, very much appreciate the comedy of this post.
- karxxm 5y agoI strongly agree with this paper. Regularizing a NN is the key for better performance. But first, one needs to know, what exactly to regularize. I don't think trial-and-error hyperparameter bingo should be the way to go. We need better insight and understanding about these networks, analyze their structure and find out, what exactly is wrong with it. Then, a TARGETED regularization (layerwise, or maybe per neuron) has a huge potential to let very simple networks perform extremely well. I even suggest, that adaptive regularizations (on/off/strength) should be researched even more. It is not necessary, that a network is regularized all the time the same way.
- nabusman 5y agoAny references/recommendations on the best practices to analyze a network, weights, etc.?
- DSingularity 5y agoAttention, regionalization, someone tell me what I really need.
- sunstone 5y agoI see what you did there.
- RocketSyntax 5y agoThis makes no sense. Neural nets already tear through tabular data like butter. We're literally dropping out half of the learned parameters. This points to the fundamental inefficiency of the approach. Better regularization is the answer, but it's not about round-robin of existing techniques.
- alpineidyll3 5y agoAnyone who finds this paper interesting should read: "self attention between datapoints" Which shows non-parametric transformers often beat GBDTs on tabular data. :https://www.google.com/url?sa=t&source=web&rct=j&url=https://arxiv.org/abs/2106.02584&ved=2ahUKEwin4cX0vazxAhVwMlkFHdGTDLkQFnoECAQQAg&usg=AOvVaw3MtQkGAsIAfyPIc4NdOJ3s https://www.google.com/url?sa=t&source=web&rct=j&url=https:/...
- xiaodai 5y agoif you spend the same amount of time in xgboost and lightgbm looking for hyperparameters and build trees with a small enough eta. will you achieve the same performance? that's not answered in the paper, and if I am a reviewer I would ask them to come compare time spent.
- jackylupino 5y agoAfter reading the paper I believe they do both aspects that you mention: i) they give XGboost and all the baselines up to 4 CPU days of hyperparameter optimization time on 20 CPU core servers per dataset, same compute as the author's proposed method. ii) the search space for the XGBoost hyperparameters includes low eta (starting from 0.001) and large num_round ranges (up to 1000 trees) in Table 5 at the appendix.
- whalesalad 5y ago> As a result, we propose regularizing plain Multilayer Perceptron (MLP) networks by searching for the optimal combination/cocktail of 13 regularization techniques for each dataset using a joint optimization over the decision on which regularizers to apply and their subsidiary hyperparameters. Pardon? https://youtu.be/RXJKdh1KZ0w https://youtu.be/RXJKdh1KZ0w
- dweinus 5y agoPer the paper, 37.5% of the regularization regimes were Batch Norm + Weight Decay. According to other findings, Batch Norm and Weight decay have an interesting interaction that results in a learning rate adjustment, not regularization: https://blog.janestreet.com/l2-regularization-and-batch-norm/ https://blog.janestreet.com/l2-regularization-and-batch-norm...
- tooheavymental 5y agoShort description: Overfit is All you need
- periheli0n 5y agoImages -> correlation in space -> Deep convolution networks Time series -> correlation in time -> Recursive networks Tabular data without clear correlation structure -> good old ML (ANN, SVM, DT, LR, KNN). This is obvious when following the field since 2006 or so. Deep Convolutional Networks were considered a special case for data with local correlations at a hierarchy of spatial scales. Same for RNNs in time, although they came much later (when was the LSTM rediscovery again? 2016?) For most data without clear spatial or temporal structure to exploit, the good old ML techniques will work just fine.
- carlmr 5y agoNot only will they work just fine, they'll save you compute costs, training time and lower your need for data.
- periheli0n 5y agoNot to mention the shedloads of “X is all you need” papers which you can ignore, because Bishop’s “Pattern Recognition and ML” book is actually all you need (plus perhaps a good reference on linear algebra).
- A-Train 5y agoNot necessarily true that MLPs are very compute expensive. It is maximum a couple of layers and if your input is sparse (categorical) you can gain even more. For some problems it can be the fastest and decent non linear model from my experience.
- periheli0n 5y agoI don’t think that was the claim… MLP/ANNs are fine except for the difficulties around interpretability. DTs and LR are preferable on that front. Or an SVM if you know a kernel/similarity metric that kills it in your data.
- jackylupino 5y agoIt is true that MLPs are classic, but the regularizaton methods that apparently make a big empirical difference at this paper are new concepts (data augmentation, skip connections/residual blocks, dropout, batch norm, lookahead, stochastic weight averaging, etc.). They compare againts a good old MLP without the bells and whistles at Table 2 and the classic MLP is quite a poor performer (XGBoost beats a classical MLP very significantly). Which leads to the conclusion that we need all these recent deep learning advances on innovative regularization techniques to make the difference.
- woadwarrior01 5y agoMassive hyperparameter sweeps without any cross validation are the perfect way to overfit holdout sets on small tabular datasets.
- cochne 5y agoBut they evaluate on a test set while sweeping on a dev set, so there is no risk of overfitting to the test set.
- stevofolife 5y agoTo piggy back on the topic of tabular data, has anyone experienced transfer learning on tabular data in their work or research?