3 ms·
Random walks work well in high dimension soaces, where calculating the gradient gets to be very expensive. Depending on your metaparameters, random walk can al
by MereInterest 9y ago
Random walks work well in high dimension soaces, where calculating the gradient gets to be very expensive. Depending on your metaparameters, random walk can also try to jump out of a local minimum, where gradient descent only optimizes within the initial local mimimum.
- halflings 9y agoGradient descent can also jump out of a local minimum if the learning rate is large enough, so they're equal in that sense. But it does make sense that a random walk would be more efficient in very high dimensional problems!
- pavelchristof 9y agoAbsolutely no. Random search does not work in high dimensions because of the "curse of dimensionality" - the number of directions to search grows exponentially with dimension. Gradient descent avoids the problem because it knows the right direction. Evolutionary strategies are doing (natural) gradient descent using an estimate of the gradient. Here's a good article: http://www.inference.vc/evolutionary-strategies-embarrassingly-parallelizable-optimization/ http://www.inference.vc/evolutionary-strategies-embarrassing...