5 ms·

tangent: all the backprop mysticism drives me nuts. it's just the chain rule...

bitL 8y ago
Most people couldn't pass calculus 101 so it's mysticism for them. Operations research people on the other hand must be shaking their heads that somebody still uses it and probably can't believe whole state-of-art is based off their most stupid method.
- adamnemecek 8y ago
  Ok what's state of the art OR?
- jefft255 8y ago
  You known, ML researchers are aware of the enormous body of research in optimization. A lot of these methods simply do not work or scale well to deep neural networks. Second order methods have been tried and are not worth the cost.
  - siekmanj 8y ago
    I mean, don’t most researchers use Adam? As far as I know that is a second-order method.
    - jefft255 8y ago
      Nope! Sorry I can’t take more time to explain but there is no second derivative used in Adam.
      - andbberger 8y ago
        From the paper > We propose Adam, a method for efficient stochastic optimization that only requires first-order gradients with little memory requirement. The method computes individual adaptive learning rates for different parameters from estimates of first and second moments of the gradients Most of the popular variants of SGD use approximations of the hessian in one way or another
        jefft255 8y ago
        Not to be peculiar, but I don't know if approximating the hessian using the gradient counts as a second order method. I was talking about "full-blown" second order methods where you compute de hessian through AD. Furthermore, I don't think by "moment of the gradients" they actually mean second derivatives. Also from the paper: We introduce Adam, an algorithm for first-order gradient-based optimization ofstochastic objective functions... It's written right in the abstract that the authors consider it a first-order method.
        andbberger 8y ago
        Seems legit
- andbberger 8y ago
  That gradient descent works so well for the non-convex loss landscapes involved is quite surprising and intriguing. There are a few OK handwavy explanations; supposedly most local minima of high-dimensional losses are saddle points, perhaps 'good models' have high entropy. But good theories are still far and few between
rramadass 8y ago
It is because of articles/books like this one which try to complicate basic ideas more than they need to be. People have forgotten how to convey concepts in a simple manner before formalizing with symbolic notation. As an example, i really understood the basics of NN only after reading Sandhya Samarasinghe's book "Neural Networks for Applied Sciences and Engineering: From Fundamentals to Complex Pattern Recognition" which explains everything simply and directly.

bitL 8y ago

Most people couldn't pass calculus 101 so it's mysticism for them. Operations research people on the other hand must be shaking their heads that somebody still uses it and probably can't believe whole state-of-art is based off their most stupid method.

adamnemecek 8y ago
Ok what's state of the art OR?
jefft255 8y ago
You known, ML researchers are aware of the enormous body of research in optimization. A lot of these methods simply do not work or scale well to deep neural networks. Second order methods have been tried and are not worth the cost.
- siekmanj 8y ago
  I mean, don’t most researchers use Adam? As far as I know that is a second-order method.
  - jefft255 8y ago
    Nope! Sorry I can’t take more time to explain but there is no second derivative used in Adam.
    - andbberger 8y ago
      From the paper > We propose Adam, a method for efficient stochastic optimization that only requires first-order gradients with little memory requirement. The method computes individual adaptive learning rates for different parameters from estimates of first and second moments of the gradients Most of the popular variants of SGD use approximations of the hessian in one way or another
      - jefft255 8y ago
        Not to be peculiar, but I don't know if approximating the hessian using the gradient counts as a second order method. I was talking about "full-blown" second order methods where you compute de hessian through AD. Furthermore, I don't think by "moment of the gradients" they actually mean second derivatives. Also from the paper: We introduce Adam, an algorithm for first-order gradient-based optimization ofstochastic objective functions... It's written right in the abstract that the authors consider it a first-order method.
        andbberger 8y ago
        Seems legit
andbberger 8y ago
That gradient descent works so well for the non-convex loss landscapes involved is quite surprising and intriguing. There are a few OK handwavy explanations; supposedly most local minima of high-dimensional losses are saddle points, perhaps 'good models' have high entropy. But good theories are still far and few between

rramadass 8y ago

It is because of articles/books like this one which try to complicate basic ideas more than they need to be. People have forgotten how to convey concepts in a simple manner before formalizing with symbolic notation. As an example, i really understood the basics of NN only after reading Sandhya Samarasinghe's book "Neural Networks for Applied Sciences and Engineering: From Fundamentals to Complex Pattern Recognition" which explains everything simply and directly.