6 ms·

Gradient descent visualization

263 points by weinzierl 2y ago

DrNosferatu 2y ago
Looks great! Killer feature: allow arbitrary surfaces to be used (choice of 2D projection if in higher dimensions). And integrate in Python to allow use in production! Allow arbitrary zoom and level of detail of surface.
albertzeyer 2y ago
Note that such a 2D parameter space gives often the wrong intuition when thinking about applying gradient descent on high-dimensional parameter space. Also, mini-batch stochastic gradient descent behaves more stochastic than just gradient descent.
- GrantMoyer 2y ago
  Agreed, but I still think this is a useful tool for building intuition about a subset of problems gradient descent faces. The animation in the readme is similar to what I picture in my head for local minima for instance.
- GaggiX 2y ago
  >gives often the wrong intuition when thinking about applying gradient descent on high-dimensional parameter space Can you give some examples?
  - euW3EeBe 2y ago
    Not an expert in this field, but I'm guessing this is related to unintuitive nature of geometry in high-dimensional spaces. One rough example I can think of is the fact that the number of ways to move away from the origin increases exponentially (or even faster?) as the dimensionality goes up. There is _way_ more volume away from the origin than near it, I've seen this explained as something like "most of the volume of a high-dimensional orange is in the peel". One result of this is the fact that samples of a standard gaussian end up forming a "shell" as opposed to a "ball" that you would expect (this phenomenon is called the concentration of measure in general). Also, very roughly, high-dimensional objects have lots of corners, and these corners are also very sharp. I would guess that gradient descent would get stuck in these corners and have a hard time getting out. Some links related to this: - Spikey Spheres: http://www.penzba.co.uk/cgi-bin/PvsNP.py?SpikeySpheres#HN2 http://www.penzba.co.uk/cgi-bin/PvsNP.py?SpikeySpheres#HN2 - Thinking outside the 10-dimensional box - 3blue1brown: https://www.youtube.com/watch?v=zwAD6dRSVyI https://www.youtube.com/watch?v=zwAD6dRSVyI - This is a fairly long talk about HMC, but it does talk about some problems that come up when sampling high-dimensional distributions: https://www.youtube.com/watch?v=pHsuIaPbNbY https://www.youtube.com/watch?v=pHsuIaPbNbY
  - DeathArrow 2y ago
    Probably there are very few local minimums since the chances of all local derivatives to be 0 decrease with the number of dimensions.
- dukeofdoom 2y ago
  I vaguely remember something like this explained in one of my math classes (calculus or linear equations). Not sure if the math teacher was referring the same problem: If you were walking down a mountain following an algorithm that told you just to keep going down. You might get stuck in a local minimum. Since you might reach a valley. Sometimes you need to go up to get out of the valley to keep going down.
  - Feathercrown 2y ago
    When I took an ML class we solved that with a temperature parameter, to allow random deviations out of a local minimum. I wonder if there are novel methods now or if they're just improved versions of temperature?
  - Imnimo 2y ago
    In very high dimensional spaces (like trying to optimize a neural network with billions of parameters), to be "in a valley", you must be in a valley with respect to every one of the billions of dimensions. It turns out that in many practical settings, loss landscapes are pretty well-behaved in this regard, and you can almost always find some direction to continue going downward in that lets you go around the next hill rather than over it. This 2015 paper has some good examples (although it does sort of sweep some issues under the rug): https://arxiv.org/pdf/1412.6544 https://arxiv.org/pdf/1412.6544
    - jonathan_landy 2y ago
      Is the claim that there aren’t local many local minima for high dimensional problems eg in neural network loss functions?
      - Imnimo 2y ago
        Yes. To be more specific, it's that nearly all points where the derivative is zero are saddle points rather than minima. Note that some portion of this nice behavior seems to be due to design choices in modern architectures, like residual connections, rather than being a general fact about all high dimensional problems. https://arxiv.org/pdf/1712.09913 https://arxiv.org/pdf/1712.09913 This paper has some nice visualizations.
- pictureofabear 2y ago
  This visualization appears to be for batch gradient descent only, not stochastic or mini-batch. That said, your critique would be true of similar visualizations for gravity warping space-time, yet these visualizations are still widely used because they help with the initial understanding of the idea--why, for example, you hold one variable constant when taking the partial differential in another dimension, and what effect changing the step size for the gradient descent has on the ability to find a local minimum.
alok-g 2y ago
This is great! Which open source license is this under? (Absence of license by default implies 'copyrighted', which in this case could be in conflict with the Qt open source license. Note: I am not a lawyer.)
- deleted 2y ago
  [deleted]
GistNoesis 2y ago
Here is a project I created for myself (some time ago) to help visualize the gradient as a vector field. https://github.com/GistNoesis/VisualizeGradient https://github.com/GistNoesis/VisualizeGradient Probably best used as a support material with someone teaching along the way the right mental picture one should have. A great exercise is to have the student (or yourself) draw this visualization with pen and paper, (both in 2d and 3d), for various functions. And you can make the connection to the usual "tangent" on the curve of a derivative.
can16358p 2y ago
As a extremely visual thinker/learner, thank you for creating this! Many things that were too abstract on paper and formulas are MUCH easier to understand this way.
levocardia 2y ago
These are nice animations. However I've always hesitated to get too enamored with these simple 2D visualizations of gradient descent, because one of the strange takeaways from deep learning is that behavior in high dimensions is very different from behavior in low dimensions. In a 2D problem with many local minima, like the "eggholder functions" [1], gradient descent will be hopeless. But neural net optimization in high dimensions really is a similar situation with many local minima, except gradient descent does great. Gradient descent in high dimensions also seems to have the ability to "step over" areas of high loss, which you can see by looking at the loss of a linear interpolation between weights at successive steps of gradient descent. This, again, seems like extremely strange behavior with no low-dimensional analogue. [1] https://www.sfu.ca/~ssurjano/egg.html https://www.sfu.ca/~ssurjano/egg.html
- cafaxo 2y ago
  Does gradient descent really do well for deep learning when the gradient is computed with respect to the whole dataset? I assumed that the noise in SGD played an important role for escaping local minima.
  - cameldrv 2y ago
    There aren't really local minima in most deep networks. When you get into millions/billions of parameters, there will essentially always be some directions that point downwards. You have to get really really close to the true minimum for there to be no direction to go that improves the loss. Incidentally this same phenomenon is IMO how evolution is able to build things like the eye. Naively you'd think that since you need so many parts arranged so well, it's impossible to find a step by step path where fitness goes up at every step, i.e. if you just have a retina with no lens or vice-versa, it doesn't work. However, due to the high dimensionality of DNA, there is essentially guaranteed to be a path with monotonically increasing fitness just because there are so many different possible paths from A to B in the high dimensional space that at least one is bound to work. Now this isn't strictly true for every high dimensional system. You need to have a degree of symmetry or redundancy in the encoding for it to work. For example, in convolutional neural networks, you see this phenomenon where some filters get "stuck" in training, and those are local minima for that subspace. What happens though is that if one filter gets stuck, the network will just use another one that had a better initialization. This is why pruning works, lottery tickets, etc. Things like residual connections enhance this effect since you can even be stuck in a whole layer and the training process can just bypass it. You see the same thing with life, where you could put a sequence for the same protein in different parts of the genome and it could still be produced, regardless of the position. There are also many different ways to encode the exact same protein, and many different possible proteins that will have the same shape in the critical areas. Life finds a way.
j_bum 2y ago
I love creating things to solidify my intuition about topics. When I learned about gradient descent, I saw this repo and was inspired to create my own toy Python package for visualizing gradient descent. My package, which uses PyVista for visualization: https://github.com/JacobBumgarner/grad-descent-visualizer https://github.com/JacobBumgarner/grad-descent-visualizer
umvi 2y ago
I thought gradient descent always followed the steepest slope, this looks like a physics simulation where marbles are rolling down hills. In mathematical gradient descent do you really oscillate around the minimum like a marble rocking back and forth in a pit? Edit: Oh, I see. The animations are "momentum" gradient descent
xg15 2y ago
Ignoring the dimensionality issue for a moment, wouldn't this also represent neural network training on a single training example/batch rather than on the whole trainset? Still a beginner in this area, but my understanding is that the standard training procedure for neural networks used today involves computing the loss ("height" of the ball) and gradient ("slope" below the ball) for one instance, then moving the ball one step in the gradient's direction, then repeating the process with a different instance. (You will also typically group a small number of instances together in a batch and process them in the same iteration - but that's for efficiency reasons, not because it's essential to the train algorithm) So if you wanted to visualise the entire training procedure, I'd imagine an animation of a smoothly rolling ball but with a rapidly changing "landscape" underneath. Would that be correct? (Writing this out of genuine curiosity, not as a criticism of the project. I think a lot more work should be done on useful visualisations in this area - and this is an incredibly helpful start.)

263 points by weinzierl 2y ago

Looks great! Killer feature: allow arbitrary surfaces to be used (choice of 2D projection if in higher dimensions). And integrate in Python to allow use in production! Allow arbitrary zoom and level of detail of surface.

albertzeyer 2y ago

Note that such a 2D parameter space gives often the wrong intuition when thinking about applying gradient descent on high-dimensional parameter space. Also, mini-batch stochastic gradient descent behaves more stochastic than just gradient descent.

GrantMoyer 2y ago
Agreed, but I still think this is a useful tool for building intuition about a subset of problems gradient descent faces. The animation in the readme is similar to what I picture in my head for local minima for instance.
GaggiX 2y ago
>gives often the wrong intuition when thinking about applying gradient descent on high-dimensional parameter space Can you give some examples?
- euW3EeBe 2y ago
  Not an expert in this field, but I'm guessing this is related to unintuitive nature of geometry in high-dimensional spaces. One rough example I can think of is the fact that the number of ways to move away from the origin increases exponentially (or even faster?) as the dimensionality goes up. There is _way_ more volume away from the origin than near it, I've seen this explained as something like "most of the volume of a high-dimensional orange is in the peel". One result of this is the fact that samples of a standard gaussian end up forming a "shell" as opposed to a "ball" that you would expect (this phenomenon is called the concentration of measure in general). Also, very roughly, high-dimensional objects have lots of corners, and these corners are also very sharp. I would guess that gradient descent would get stuck in these corners and have a hard time getting out. Some links related to this: - Spikey Spheres: http://www.penzba.co.uk/cgi-bin/PvsNP.py?SpikeySpheres#HN2 http://www.penzba.co.uk/cgi-bin/PvsNP.py?SpikeySpheres#HN2 - Thinking outside the 10-dimensional box - 3blue1brown: https://www.youtube.com/watch?v=zwAD6dRSVyI https://www.youtube.com/watch?v=zwAD6dRSVyI - This is a fairly long talk about HMC, but it does talk about some problems that come up when sampling high-dimensional distributions: https://www.youtube.com/watch?v=pHsuIaPbNbY https://www.youtube.com/watch?v=pHsuIaPbNbY
- DeathArrow 2y ago
  Probably there are very few local minimums since the chances of all local derivatives to be 0 decrease with the number of dimensions.
dukeofdoom 2y ago
I vaguely remember something like this explained in one of my math classes (calculus or linear equations). Not sure if the math teacher was referring the same problem: If you were walking down a mountain following an algorithm that told you just to keep going down. You might get stuck in a local minimum. Since you might reach a valley. Sometimes you need to go up to get out of the valley to keep going down.
- Feathercrown 2y ago
  When I took an ML class we solved that with a temperature parameter, to allow random deviations out of a local minimum. I wonder if there are novel methods now or if they're just improved versions of temperature?
- Imnimo 2y ago
  In very high dimensional spaces (like trying to optimize a neural network with billions of parameters), to be "in a valley", you must be in a valley with respect to every one of the billions of dimensions. It turns out that in many practical settings, loss landscapes are pretty well-behaved in this regard, and you can almost always find some direction to continue going downward in that lets you go around the next hill rather than over it. This 2015 paper has some good examples (although it does sort of sweep some issues under the rug): https://arxiv.org/pdf/1412.6544 https://arxiv.org/pdf/1412.6544
  - jonathan_landy 2y ago
    Is the claim that there aren’t local many local minima for high dimensional problems eg in neural network loss functions?
    - Imnimo 2y ago
      Yes. To be more specific, it's that nearly all points where the derivative is zero are saddle points rather than minima. Note that some portion of this nice behavior seems to be due to design choices in modern architectures, like residual connections, rather than being a general fact about all high dimensional problems. https://arxiv.org/pdf/1712.09913 https://arxiv.org/pdf/1712.09913 This paper has some nice visualizations.
pictureofabear 2y ago
This visualization appears to be for batch gradient descent only, not stochastic or mini-batch. That said, your critique would be true of similar visualizations for gravity warping space-time, yet these visualizations are still widely used because they help with the initial understanding of the idea--why, for example, you hold one variable constant when taking the partial differential in another dimension, and what effect changing the step size for the gradient descent has on the ability to find a local minimum.

alok-g 2y ago

This is great! Which open source license is this under? (Absence of license by default implies 'copyrighted', which in this case could be in conflict with the Qt open source license. Note: I am not a lawyer.)

deleted 2y ago
[deleted]

GistNoesis 2y ago

Here is a project I created for myself (some time ago) to help visualize the gradient as a vector field. https://github.com/GistNoesis/VisualizeGradient https://github.com/GistNoesis/VisualizeGradient Probably best used as a support material with someone teaching along the way the right mental picture one should have. A great exercise is to have the student (or yourself) draw this visualization with pen and paper, (both in 2d and 3d), for various functions. And you can make the connection to the usual "tangent" on the curve of a derivative.

can16358p 2y ago

As a extremely visual thinker/learner, thank you for creating this! Many things that were too abstract on paper and formulas are MUCH easier to understand this way.

levocardia 2y ago

These are nice animations. However I've always hesitated to get too enamored with these simple 2D visualizations of gradient descent, because one of the strange takeaways from deep learning is that behavior in high dimensions is very different from behavior in low dimensions. In a 2D problem with many local minima, like the "eggholder functions" [1], gradient descent will be hopeless. But neural net optimization in high dimensions really is a similar situation with many local minima, except gradient descent does great. Gradient descent in high dimensions also seems to have the ability to "step over" areas of high loss, which you can see by looking at the loss of a linear interpolation between weights at successive steps of gradient descent. This, again, seems like extremely strange behavior with no low-dimensional analogue. [1] https://www.sfu.ca/~ssurjano/egg.html https://www.sfu.ca/~ssurjano/egg.html

cafaxo 2y ago
Does gradient descent really do well for deep learning when the gradient is computed with respect to the whole dataset? I assumed that the noise in SGD played an important role for escaping local minima.
- cameldrv 2y ago
  There aren't really local minima in most deep networks. When you get into millions/billions of parameters, there will essentially always be some directions that point downwards. You have to get really really close to the true minimum for there to be no direction to go that improves the loss. Incidentally this same phenomenon is IMO how evolution is able to build things like the eye. Naively you'd think that since you need so many parts arranged so well, it's impossible to find a step by step path where fitness goes up at every step, i.e. if you just have a retina with no lens or vice-versa, it doesn't work. However, due to the high dimensionality of DNA, there is essentially guaranteed to be a path with monotonically increasing fitness just because there are so many different possible paths from A to B in the high dimensional space that at least one is bound to work. Now this isn't strictly true for every high dimensional system. You need to have a degree of symmetry or redundancy in the encoding for it to work. For example, in convolutional neural networks, you see this phenomenon where some filters get "stuck" in training, and those are local minima for that subspace. What happens though is that if one filter gets stuck, the network will just use another one that had a better initialization. This is why pruning works, lottery tickets, etc. Things like residual connections enhance this effect since you can even be stuck in a whole layer and the training process can just bypass it. You see the same thing with life, where you could put a sequence for the same protein in different parts of the genome and it could still be produced, regardless of the position. There are also many different ways to encode the exact same protein, and many different possible proteins that will have the same shape in the critical areas. Life finds a way.

j_bum 2y ago

I love creating things to solidify my intuition about topics. When I learned about gradient descent, I saw this repo and was inspired to create my own toy Python package for visualizing gradient descent. My package, which uses PyVista for visualization: https://github.com/JacobBumgarner/grad-descent-visualizer https://github.com/JacobBumgarner/grad-descent-visualizer

umvi 2y ago

I thought gradient descent always followed the steepest slope, this looks like a physics simulation where marbles are rolling down hills. In mathematical gradient descent do you really oscillate around the minimum like a marble rocking back and forth in a pit? Edit: Oh, I see. The animations are "momentum" gradient descent

xg15 2y ago

Ignoring the dimensionality issue for a moment, wouldn't this also represent neural network training on a single training example/batch rather than on the whole trainset? Still a beginner in this area, but my understanding is that the standard training procedure for neural networks used today involves computing the loss ("height" of the ball) and gradient ("slope" below the ball) for one instance, then moving the ball one step in the gradient's direction, then repeating the process with a different instance. (You will also typically group a small number of instances together in a batch and process them in the same iteration - but that's for efficiency reasons, not because it's essential to the train algorithm) So if you wanted to visualise the entire training procedure, I'd imagine an animation of a smoothly rolling ball but with a rapidly changing "landscape" underneath. Would that be correct? (Writing this out of genuine curiosity, not as a criticism of the project. I think a lot more work should be done on useful visualisations in this area - and this is an incredibly helpful start.)