4 ms·

That's a strong unsubstantiated claim. On the other hand, there's been some nice theoretical work arguing that deep learning is a form of polynomial regression.

by tbenst 6y ago

That's a strong unsubstantiated claim. On the other hand, there's been some nice theoretical work arguing that deep learning is a form of polynomial regression. https://arxiv.org/abs/1806.06850 https://arxiv.org/abs/1806.06850

moultano 6y ago
Let us know when polynomial regression succeeds at any machine learning task. A lot of people publish results that say deep learning is "just" something else, where the something else doesn't work.
- tbenst 6y ago
  How about 80% accuracy on CIFAR-10 with unsupervised training? => logistic regression + Kmeans http://ai.stanford.edu/~acoates/papers/coatesleeng_aistats_2011.pdf http://ai.stanford.edu/~acoates/papers/coatesleeng_aistats_2...
blackbear_ 6y ago
They are both universal approximators. So are support vector machines, gaussian processes and gradient boosted trees. Yet the performance of neural networks is unrivaled in certain tasks, as has been proven over and over again. As a whole, that paper is quite bad (and still unpublished, probably blocked by peer review) because (1) it only considers fully connected networks (which are a minority of models used nowadays) and (2) the experimental validation was done on tasks where neural networks are not very strong. Show me examples of competitive polynomial regression models in language translation, image segmentation and Go playing and I will be convinced.
- tbenst 6y ago
  To be fair, most of the machine learning literature had no or a poor excuse for peer review. And many deep learning layers can be described by dense layers. For example, convolution. You almost certainly can frame Go playing as polynomial regression but there would undoubtedly be numerical precision & other gradient issues. Deep learning is a practice is remarkably effective, no disagreement there.

4 ms·

That's a strong unsubstantiated claim. On the other hand, there's been some nice theoretical work arguing that deep learning is a form of polynomial regression.

by tbenst 6y ago

moultano 6y ago
Let us know when polynomial regression succeeds at any machine learning task. A lot of people publish results that say deep learning is "just" something else, where the something else doesn't work.
- tbenst 6y ago
  How about 80% accuracy on CIFAR-10 with unsupervised training? => logistic regression + Kmeans http://ai.stanford.edu/~acoates/papers/coatesleeng_aistats_2011.pdf http://ai.stanford.edu/~acoates/papers/coatesleeng_aistats_2...
blackbear_ 6y ago
They are both universal approximators. So are support vector machines, gaussian processes and gradient boosted trees. Yet the performance of neural networks is unrivaled in certain tasks, as has been proven over and over again. As a whole, that paper is quite bad (and still unpublished, probably blocked by peer review) because (1) it only considers fully connected networks (which are a minority of models used nowadays) and (2) the experimental validation was done on tasks where neural networks are not very strong. Show me examples of competitive polynomial regression models in language translation, image segmentation and Go playing and I will be convinced.
- tbenst 6y ago
  To be fair, most of the machine learning literature had no or a poor excuse for peer review. And many deep learning layers can be described by dense layers. For example, convolution. You almost certainly can frame Go playing as polynomial regression but there would undoubtedly be numerical precision & other gradient issues. Deep learning is a practice is remarkably effective, no disagreement there.