3 ms·
Stuff got stuck in local minima for years before we learned about stuff like momentum and dropout and dropped a ton of GPU power on it.
by Eliezer 9y ago
Stuff got stuck in local minima for years before we learned about stuff like momentum and dropout and dropped a ton of GPU power on it.
- llamaz 9y agoWhen I was implementing a neural network for a university assignment (2 years ago so my memory might fail me), we had to run our algorithm multiple time with different starting positions, then take the minimum of those local minima. I'm not sure what momentum and dropout are, but I agree with Eleizer, without these things (which I didn't use) local minima are a problem.
- eat_veggies 9y agoDropout is where you randomly remove neurons from your network during training, which prevents them from depending too much on specific neurons (making the output more generalizable). It was developed in 2014 so it would have been brand new tech back when you were in your class.
- alexcnwy 9y agoI think you're misinterpreting the parent who is saying that local minima are not a problem in high dimensions because there is always a dimension to move in that reduces the loss (unlike in lower dimensions where you can get stuck in a point across all dimensions that cannot be locally improved upon)
- llamaz 9y agoI still don't understand what the parent is talking about then. Could you please restate the explanation using math notation/terminology?