5 ms·

You are missing the point. The surprising thing about deep learning is that it can generalize to unseen data so well. Polynomial regression cannot.

mamon 6y ago
If there is a pattern in data, such as it fitting on a curve, and you approximate that curve, then that should generalize to unseen data. What's surprising about that? A single polynominal regression wouldn't be able to do it, because some curves cannot be expressed as a polynominal, but superposition of multiple polynominals is apparently good enough.
- moultano 6y ago
  If that worked, people would use it, but it doesn't, so they don't.
- blackbear_ 6y ago
  > then that should generalize to unseen data. It's not guaranteed at all. Overcomplicated models will "overfit" the training data and generalize very poorly. > some curves cannot be expressed as a polynominal You can approximate any (continuous and blah blah) curve arbitrarily well with Taylor expansions. In fact, polynomials are one of the the most common examples to demonstrate overfitting. See figure 2 on wikipedia (https://en.wikipedia.org/wiki/Overfitting https://en.wikipedia.org/wiki/Overfitting)
mumblemumble 6y ago
It doesn't necessarily seem that surprising to me. If I can be really hand-wavy about it -- A lot of deep neural networks achieve their nonlinearity in a very constrained way. They stack linear models on top of each other, and the nonlinearity comes from using a a relatively simple nonlinear function such as logistic or tanh to scale the models' outputs before feeding them into the next one. (Without that step, you'd just have a linear combination of linear functions, which would itself be linear.) That's a pretty constrained form of nonlinearity compared to polynomial regression, which tries to directly fit some high-order polynomial. I don't have anything like the math chops to prove it this, but I believe that means that the neural network is going to tend to favor a relatively smoother decision boundary, whereas polynomial regression is a naturally high variance sort of affair.
- Darkharbourzz 6y ago
  No, it's not constrained at all. In fact, even single hidden layer networks with nearly arbitrary activation functions are universal approximators (Hornik et al. 1989). Polynomials are also universal approximators (Weierstrass).
  - mumblemumble 6y ago
    I'm not trying to say that neural networks are inherently constrained. I'm saying that, in typical usage, they tend to be used a certain way that I believe introduces some useful constraints. You can use a single hidden layer and an arbitrary activation functions, but, in practice, it's a heck of a lot more common to use multiple hidden layers and tanh. It's worth noting that neural networks didn't take off with Hornik et al. style simple-topology-complex-activation-function universal approximators. They took off a decade or so later, with LeCun-style complex-topology-simple-activation-function networks. That arguably suggests that the paper is of more theoretical than practical interest. It's also worth noting that one of the practical challenges with a single hidden layer and a complex activation function is that it's susceptible to variance. Just like polynomial regression.
    - bonoboTP 6y ago
      This kind of stuff is called inductive bias and is a sexy topic nowadays.
- blackbear_ 6y ago
  I agree this is one of the reasons for the success of neural networks. But it was not obvious at first, and it still is quite hard to formalize and explain in mathematical terms. That's what I meant with "surprising".
  - deleted 6y ago
    [deleted]
lallysingh 6y ago
How's that different from regular interpolation?
tbenst 6y ago
That's a strong unsubstantiated claim. On the other hand, there's been some nice theoretical work arguing that deep learning is a form of polynomial regression. https://arxiv.org/abs/1806.06850 https://arxiv.org/abs/1806.06850
- moultano 6y ago
  Let us know when polynomial regression succeeds at any machine learning task. A lot of people publish results that say deep learning is "just" something else, where the something else doesn't work.
  - tbenst 6y ago
    How about 80% accuracy on CIFAR-10 with unsupervised training? => logistic regression + Kmeans http://ai.stanford.edu/~acoates/papers/coatesleeng_aistats_2011.pdf http://ai.stanford.edu/~acoates/papers/coatesleeng_aistats_2...
- blackbear_ 6y ago
  They are both universal approximators. So are support vector machines, gaussian processes and gradient boosted trees. Yet the performance of neural networks is unrivaled in certain tasks, as has been proven over and over again. As a whole, that paper is quite bad (and still unpublished, probably blocked by peer review) because (1) it only considers fully connected networks (which are a minority of models used nowadays) and (2) the experimental validation was done on tasks where neural networks are not very strong. Show me examples of competitive polynomial regression models in language translation, image segmentation and Go playing and I will be convinced.
  - tbenst 6y ago
    To be fair, most of the machine learning literature had no or a poor excuse for peer review. And many deep learning layers can be described by dense layers. For example, convolution. You almost certainly can frame Go playing as polynomial regression but there would undoubtedly be numerical precision & other gradient issues. Deep learning is a practice is remarkably effective, no disagreement there.

mamon 6y ago

If there is a pattern in data, such as it fitting on a curve, and you approximate that curve, then that should generalize to unseen data. What's surprising about that? A single polynominal regression wouldn't be able to do it, because some curves cannot be expressed as a polynominal, but superposition of multiple polynominals is apparently good enough.

moultano 6y ago
If that worked, people would use it, but it doesn't, so they don't.
blackbear_ 6y ago
> then that should generalize to unseen data. It's not guaranteed at all. Overcomplicated models will "overfit" the training data and generalize very poorly. > some curves cannot be expressed as a polynominal You can approximate any (continuous and blah blah) curve arbitrarily well with Taylor expansions. In fact, polynomials are one of the the most common examples to demonstrate overfitting. See figure 2 on wikipedia (https://en.wikipedia.org/wiki/Overfitting https://en.wikipedia.org/wiki/Overfitting)

mumblemumble 6y ago

It doesn't necessarily seem that surprising to me. If I can be really hand-wavy about it -- A lot of deep neural networks achieve their nonlinearity in a very constrained way. They stack linear models on top of each other, and the nonlinearity comes from using a a relatively simple nonlinear function such as logistic or tanh to scale the models' outputs before feeding them into the next one. (Without that step, you'd just have a linear combination of linear functions, which would itself be linear.) That's a pretty constrained form of nonlinearity compared to polynomial regression, which tries to directly fit some high-order polynomial. I don't have anything like the math chops to prove it this, but I believe that means that the neural network is going to tend to favor a relatively smoother decision boundary, whereas polynomial regression is a naturally high variance sort of affair.

Darkharbourzz 6y ago
No, it's not constrained at all. In fact, even single hidden layer networks with nearly arbitrary activation functions are universal approximators (Hornik et al. 1989). Polynomials are also universal approximators (Weierstrass).
- mumblemumble 6y ago
  I'm not trying to say that neural networks are inherently constrained. I'm saying that, in typical usage, they tend to be used a certain way that I believe introduces some useful constraints. You can use a single hidden layer and an arbitrary activation functions, but, in practice, it's a heck of a lot more common to use multiple hidden layers and tanh. It's worth noting that neural networks didn't take off with Hornik et al. style simple-topology-complex-activation-function universal approximators. They took off a decade or so later, with LeCun-style complex-topology-simple-activation-function networks. That arguably suggests that the paper is of more theoretical than practical interest. It's also worth noting that one of the practical challenges with a single hidden layer and a complex activation function is that it's susceptible to variance. Just like polynomial regression.
  - bonoboTP 6y ago
    This kind of stuff is called inductive bias and is a sexy topic nowadays.
blackbear_ 6y ago
I agree this is one of the reasons for the success of neural networks. But it was not obvious at first, and it still is quite hard to formalize and explain in mathematical terms. That's what I meant with "surprising".
- deleted 6y ago
  [deleted]

lallysingh 6y ago

How's that different from regular interpolation?

tbenst 6y ago

That's a strong unsubstantiated claim. On the other hand, there's been some nice theoretical work arguing that deep learning is a form of polynomial regression. https://arxiv.org/abs/1806.06850 https://arxiv.org/abs/1806.06850

moultano 6y ago
Let us know when polynomial regression succeeds at any machine learning task. A lot of people publish results that say deep learning is "just" something else, where the something else doesn't work.
- tbenst 6y ago
  How about 80% accuracy on CIFAR-10 with unsupervised training? => logistic regression + Kmeans http://ai.stanford.edu/~acoates/papers/coatesleeng_aistats_2011.pdf http://ai.stanford.edu/~acoates/papers/coatesleeng_aistats_2...
blackbear_ 6y ago
They are both universal approximators. So are support vector machines, gaussian processes and gradient boosted trees. Yet the performance of neural networks is unrivaled in certain tasks, as has been proven over and over again. As a whole, that paper is quite bad (and still unpublished, probably blocked by peer review) because (1) it only considers fully connected networks (which are a minority of models used nowadays) and (2) the experimental validation was done on tasks where neural networks are not very strong. Show me examples of competitive polynomial regression models in language translation, image segmentation and Go playing and I will be convinced.
- tbenst 6y ago
  To be fair, most of the machine learning literature had no or a poor excuse for peer review. And many deep learning layers can be described by dense layers. For example, convolution. You almost certainly can frame Go playing as polynomial regression but there would undoubtedly be numerical precision & other gradient issues. Deep learning is a practice is remarkably effective, no disagreement there.