3 ms·
>gives often the wrong intuition when thinking about applying gradient descent on high-dimensional parameter space Can you give some examples?
by GaggiX 2y ago
>gives often the wrong intuition when thinking about applying gradient descent on high-dimensional parameter space
Can you give some examples?
- euW3EeBe 2y agoNot an expert in this field, but I'm guessing this is related to unintuitive nature of geometry in high-dimensional spaces. One rough example I can think of is the fact that the number of ways to move away from the origin increases exponentially (or even faster?) as the dimensionality goes up. There is _way_ more volume away from the origin than near it, I've seen this explained as something like "most of the volume of a high-dimensional orange is in the peel". One result of this is the fact that samples of a standard gaussian end up forming a "shell" as opposed to a "ball" that you would expect (this phenomenon is called the concentration of measure in general). Also, very roughly, high-dimensional objects have lots of corners, and these corners are also very sharp. I would guess that gradient descent would get stuck in these corners and have a hard time getting out. Some links related to this: - Spikey Spheres: http://www.penzba.co.uk/cgi-bin/PvsNP.py?SpikeySpheres#HN2 http://www.penzba.co.uk/cgi-bin/PvsNP.py?SpikeySpheres#HN2 - Thinking outside the 10-dimensional box - 3blue1brown: https://www.youtube.com/watch?v=zwAD6dRSVyI https://www.youtube.com/watch?v=zwAD6dRSVyI - This is a fairly long talk about HMC, but it does talk about some problems that come up when sampling high-dimensional distributions: https://www.youtube.com/watch?v=pHsuIaPbNbY https://www.youtube.com/watch?v=pHsuIaPbNbY
- DeathArrow 2y agoProbably there are very few local minimums since the chances of all local derivatives to be 0 decrease with the number of dimensions.