3 ms·

Images -> correlation in space -> Deep convolution networks Time series -> correlation in time -> Recursive networks Tabular data without clear correlation st

Images -> correlation in space -> Deep convolution networks Time series -> correlation in time -> Recursive networks Tabular data without clear correlation structure -> good old ML (ANN, SVM, DT, LR, KNN). This is obvious when following the field since 2006 or so. Deep Convolutional Networks were considered a special case for data with local correlations at a hierarchy of spatial scales. Same for RNNs in time, although they came much later (when was the LSTM rediscovery again? 2016?) For most data without clear spatial or temporal structure to exploit, the good old ML techniques will work just fine.

carlmr 5y ago
Not only will they work just fine, they'll save you compute costs, training time and lower your need for data.
- periheli0n 5y ago
  Not to mention the shedloads of “X is all you need” papers which you can ignore, because Bishop’s “Pattern Recognition and ML” book is actually all you need (plus perhaps a good reference on linear algebra).
- A-Train 5y ago
  Not necessarily true that MLPs are very compute expensive. It is maximum a couple of layers and if your input is sparse (categorical) you can gain even more. For some problems it can be the fastest and decent non linear model from my experience.
  - periheli0n 5y ago
    I don’t think that was the claim… MLP/ANNs are fine except for the difficulties around interpretability. DTs and LR are preferable on that front. Or an SVM if you know a kernel/similarity metric that kills it in your data.
jackylupino 5y ago
It is true that MLPs are classic, but the regularizaton methods that apparently make a big empirical difference at this paper are new concepts (data augmentation, skip connections/residual blocks, dropout, batch norm, lookahead, stochastic weight averaging, etc.). They compare againts a good old MLP without the bells and whistles at Table 2 and the classic MLP is quite a poor performer (XGBoost beats a classical MLP very significantly). Which leads to the conclusion that we need all these recent deep learning advances on innovative regularization techniques to make the difference.

carlmr 5y ago

Not only will they work just fine, they'll save you compute costs, training time and lower your need for data.

periheli0n 5y ago
Not to mention the shedloads of “X is all you need” papers which you can ignore, because Bishop’s “Pattern Recognition and ML” book is actually all you need (plus perhaps a good reference on linear algebra).
A-Train 5y ago
Not necessarily true that MLPs are very compute expensive. It is maximum a couple of layers and if your input is sparse (categorical) you can gain even more. For some problems it can be the fastest and decent non linear model from my experience.
- periheli0n 5y ago
  I don’t think that was the claim… MLP/ANNs are fine except for the difficulties around interpretability. DTs and LR are preferable on that front. Or an SVM if you know a kernel/similarity metric that kills it in your data.

jackylupino 5y ago

It is true that MLPs are classic, but the regularizaton methods that apparently make a big empirical difference at this paper are new concepts (data augmentation, skip connections/residual blocks, dropout, batch norm, lookahead, stochastic weight averaging, etc.). They compare againts a good old MLP without the bells and whistles at Table 2 and the classic MLP is quite a poor performer (XGBoost beats a classical MLP very significantly). Which leads to the conclusion that we need all these recent deep learning advances on innovative regularization techniques to make the difference.