3 ms·
I'm not sure dropout has anything to do with local optima or removing greedily optimal paths, since it is random. The original dropout paper's handwavy justifi
by brianchu 10y ago
I'm not sure dropout has anything to do with local optima or removing greedily optimal paths, since it is random.
The original dropout paper's handwavy justification for dropout is that it prevents co-adaptation. It prevents individual units (nodes/neurons) in the network from relying on specific units in the previous layer firing as well. This is a bad thing because it's fragile (if one unit is off). I say handwavy because this is just intuition; there is not really any proof that this is actually what is happening.
Another commonly cited motivation is that dropout is like learning an ensemble of multiple networks.
The only paper I've seen that theoretically analyzes dropout is: https://arxiv.org/pdf/1506.02142v6.pdf https://arxiv.org/pdf/1506.02142v6.pdf, which proves it's equivalent to approximating gaussian processes (this is beyond me).
- zardeh 10y agoI think that you could consider the co-adaption as a form of greedy local optima (I think this is a very handwavy explanation, though, and I'm having trouble forming it better, so take with that what you will). Dropout prevents that by not allowing the 'greedily optimal paths' to form, since sometimes they don't exist, so you can't rely on them too much.
- brianchu 10y agoSure, but dropout actually increases training error (makes you less likely to find the globally optimal training error), with (sometimes) a decrease in generalization error (test error). So any connection is very thin.
- zardeh 10y agoOh actually that's a really good point and I think makes the difference clear (there isn't such a train/test divide in the driving example). Thank you :)