3 ms·

This is an excellent post and I love the interactive demos. > gradient descent ... ball rolling down a hill Gradient descent is similar to a ball with no mass

This is an excellent post and I love the interactive demos. > gradient descent ... ball rolling down a hill Gradient descent is similar to a ball with no mass falling down a hill. Other methods (heavy-ball and Nesterov acceleration) are similar to a heavier ball rolling down a hill. I've written a post [0] on this with a visualization. Given a quadratic model (most examples in post) there are some strong theory on convergence rates. Nesterov acceleration has at least linear convergence (which really means exponential). More detail can be found in Noecdal and Wright. (I'm currently in Steve Wrights Nonlinear Optimization I) [0]:http://scottsievert.com/blog/2016/01/30/inverse-3/ http://scottsievert.com/blog/2016/01/30/inverse-3/

benfrederickson 10y ago
Your visualization of using momentum with gradient descent in that post is really great - nice work!
- skierscott 10y ago
  I've generated some other interactive visualizations. Some try to give intuition for regularzation [0] and another one [1] that explores colorspaces. I use ipywidgets-static to generate these and am looking forward to the upcoming release of iphwidgets that supports static widgets. [0]:http://scottsievert.com/blog/2015/12/09/inverse-part-2/ http://scottsievert.com/blog/2015/12/09/inverse-part-2/ [1]:http://scottsievert.com/blog/2015/04/23/image-sqrt/ http://scottsievert.com/blog/2015/04/23/image-sqrt/
santaclaus 10y ago
> Gradient descent is similar to a ball with no mass falling down a hill. Other methods (heavy-ball and Nesterov acceleration) are similar to a heavier ball rolling down a hill. I've written a post [0] on this with a visualization. I don't think mass is the right analogy -- if you non-dimensionalize a ball on a hill the mass shouldn't even appear. Maybe critical vs under damped (gradient descent isn't allowed to overshoot, Nestersov methods are)? Edit: OK, with a viscous drag force the analogy sort of works, a heavier sliding particle will overshoot. A massless particle will not move at all, however...
- deepnotderp 10y ago
  I think the best way to think of it is a heavy, rolling ball.

benfrederickson 10y ago

Your visualization of using momentum with gradient descent in that post is really great - nice work!

skierscott 10y ago
I've generated some other interactive visualizations. Some try to give intuition for regularzation [0] and another one [1] that explores colorspaces. I use ipywidgets-static to generate these and am looking forward to the upcoming release of iphwidgets that supports static widgets. [0]:http://scottsievert.com/blog/2015/12/09/inverse-part-2/ http://scottsievert.com/blog/2015/12/09/inverse-part-2/ [1]:http://scottsievert.com/blog/2015/04/23/image-sqrt/ http://scottsievert.com/blog/2015/04/23/image-sqrt/

santaclaus 10y ago

> Gradient descent is similar to a ball with no mass falling down a hill. Other methods (heavy-ball and Nesterov acceleration) are similar to a heavier ball rolling down a hill. I've written a post [0] on this with a visualization. I don't think mass is the right analogy -- if you non-dimensionalize a ball on a hill the mass shouldn't even appear. Maybe critical vs under damped (gradient descent isn't allowed to overshoot, Nestersov methods are)? Edit: OK, with a viscous drag force the analogy sort of works, a heavier sliding particle will overshoot. A massless particle will not move at all, however...

deepnotderp 10y ago
I think the best way to think of it is a heavy, rolling ball.