3 ms·

I'd just like to interject for moment. What you're refering to as Deep Learning, is in fact, non-convex optimization, or as I've recently taken to calling it, g

I'd just like to interject for moment. What you're refering to as Deep Learning, is in fact, non-convex optimization, or as I've recently taken to calling it, generalized function approximation. Deep Learning is not a technology unto itself, but rather a specific type of mathematical optimization that has been known about for decades, is based on stochastic gradient descent, and only now is being featured more prominently due to enhaced processing power. Many computer users run some form of mathematical optimization everyday, without realizing it. Through a peculiar turn of events, the optimizations which are widely used today are often called Machine Learning, and many of its users are not aware that it is basically non-convex optimization, or high dimensional function approximation. There really is Deep Learning, and these people are using it, but it is just a part of the optimization mathematics they use. Deep Learning is the kernel: the program in the system that allocates the machine's resources to the other programs that you run. The kernel is an essential part of an operating system, but useless by itself; it can only function in the context of a complete rigorous mathematical framework. Deep Learning is normally used in combination with other ideas in classical mathematical optimization: the whole system is basically stochastic gradient descent with GPUs added, or large-scale numerical optimization. All the so-called Deep Learning algorithms are really algorithms of generalized function approximation!

brandonb 10y ago
This seems like a sincere comment, so in case it's helpful, here's my interpretation of why others may be downvoting it. Deep learning practitioners are aware that they're using non-convex optimization, stochastic gradient descent, and so on. In fact, those topics are core to modern deep learning research and explicitly acknowledged in most published research: LSTMs were invented to solve the vanishing gradient problem, Restricted Boltzmann Machines were used as a pre-training step to avoid local minima, and optimizers like ADAM have explicit guarantees about things like convergence. You may know all this stuff already--not sure, based on your comment above. If not, here are a few example papers from mainstream AI researchers explicitly talking about deep learning as an optimization problem or function approximator: [1] Why Does Unsupervised Pre-training Help Deep Learning?http://www.jmlr.org/papers/volume11/erhan10a/erhan10a.pdf http://www.jmlr.org/papers/volume11/erhan10a/erhan10a.pdf [2] Multilayer feedforward networks are universal approximators: http://www.sciencedirect.com/science/article/pii/0893608089900208 http://www.sciencedirect.com/science/article/pii/08936080899... [3] Deep Learning, Nature. http://www.cs.toronto.edu/~hinton/absps/NatureDeepReview.pdf http://www.cs.toronto.edu/~hinton/absps/NatureDeepReview.pdf [4] Adam: A Method for Stochastic Optimization. https://arxiv.org/abs/1412.6980 https://arxiv.org/abs/1412.6980
- Guyag 10y ago
  I believe it's a version of an old copypasta [0]. [0] https://wiki.installgentoo.com/index.php/Interjection https://wiki.installgentoo.com/index.php/Interjection
  - brandonb 10y ago
    Well, I guess my sincerity detector is officially broken, huh?
- amelius 10y ago
  > Deep learning practitioners are aware that they're using non-convex optimization It really depends. It's now perfectly possible to use deep learning as a black box, and I've already seen many people use it that way. Just like you don't have to know how a web-browser works to be a web-programmer, you don't have to know how deep learning works to (e.g.) train a classifier.

brandonb 10y ago

This seems like a sincere comment, so in case it's helpful, here's my interpretation of why others may be downvoting it. Deep learning practitioners are aware that they're using non-convex optimization, stochastic gradient descent, and so on. In fact, those topics are core to modern deep learning research and explicitly acknowledged in most published research: LSTMs were invented to solve the vanishing gradient problem, Restricted Boltzmann Machines were used as a pre-training step to avoid local minima, and optimizers like ADAM have explicit guarantees about things like convergence. You may know all this stuff already--not sure, based on your comment above. If not, here are a few example papers from mainstream AI researchers explicitly talking about deep learning as an optimization problem or function approximator: [1] Why Does Unsupervised Pre-training Help Deep Learning?http://www.jmlr.org/papers/volume11/erhan10a/erhan10a.pdf http://www.jmlr.org/papers/volume11/erhan10a/erhan10a.pdf [2] Multilayer feedforward networks are universal approximators: http://www.sciencedirect.com/science/article/pii/0893608089900208 http://www.sciencedirect.com/science/article/pii/08936080899... [3] Deep Learning, Nature. http://www.cs.toronto.edu/~hinton/absps/NatureDeepReview.pdf http://www.cs.toronto.edu/~hinton/absps/NatureDeepReview.pdf [4] Adam: A Method for Stochastic Optimization. https://arxiv.org/abs/1412.6980 https://arxiv.org/abs/1412.6980

Guyag 10y ago
I believe it's a version of an old copypasta [0]. [0] https://wiki.installgentoo.com/index.php/Interjection https://wiki.installgentoo.com/index.php/Interjection
- brandonb 10y ago
  Well, I guess my sincerity detector is officially broken, huh?
amelius 10y ago
> Deep learning practitioners are aware that they're using non-convex optimization It really depends. It's now perfectly possible to use deep learning as a black box, and I've already seen many people use it that way. Just like you don't have to know how a web-browser works to be a web-programmer, you don't have to know how deep learning works to (e.g.) train a classifier.