4 ms·

Isn't this the reason that dropout works so well as a regularizer? In dropout, you select 50% (say) of the nodes in one layer of your network at random and for

Isn't this the reason that dropout works so well as a regularizer? In dropout, you select 50% (say) of the nodes in one layer of your network at random and force them to pass on no signal (neither positive nor negative). You then train the network to still return the result. This is effectively training the network to be robust to failures (i.e. "harder to break"). While writing this, I've realized that we've both assumed that not only "networks which generalize better are harder to break" but also "networks which are harder to break generalize better". I think that's true, but worth being explicit. (Still developing my intuition for machine learning, so I could be wrong...)

arimorcos 9y ago
Author here. Indeed, our work is closely related to dropout. However, as we discuss in the paper (https://arxiv.org/abs/1803.06959 https://arxiv.org/abs/1803.06959), dropout doesn't really encourage the network to be robust to deletion generally; it only encourages this robustness up until the dropout fraction used in training. So, for example, if you train the network with a 50% dropout rate, dropout will encourage the network to be robust to dropping 50% of the units, but the network could completely fail once 51% of the units are deleted and the training objective would be perfectly happy. As a result, dropout doesn't change the shapes of ablation curves, but rather simply horizontally scales them such that the left edge of the curves is at the dropout fraction rather than 0. In contrast, we found that batch normalization actually pulls the curves up and to the right, rather than simply scaling them, though we only have hints as to why that is. Hope that was helpful!
- LoSboccacc 9y ago
  > but the network could completely fail once 51% of the units are deleted also the network could have weird results for dropout values around 20%-30% depending on how the robustness was 'learned'
- maffydub 9y ago
  Thanks for your response - that's very interesting! I haven't read the paper yet, but have added it to my queue.

arimorcos 9y ago

Author here. Indeed, our work is closely related to dropout. However, as we discuss in the paper (https://arxiv.org/abs/1803.06959 https://arxiv.org/abs/1803.06959), dropout doesn't really encourage the network to be robust to deletion generally; it only encourages this robustness up until the dropout fraction used in training. So, for example, if you train the network with a 50% dropout rate, dropout will encourage the network to be robust to dropping 50% of the units, but the network could completely fail once 51% of the units are deleted and the training objective would be perfectly happy. As a result, dropout doesn't change the shapes of ablation curves, but rather simply horizontally scales them such that the left edge of the curves is at the dropout fraction rather than 0. In contrast, we found that batch normalization actually pulls the curves up and to the right, rather than simply scaling them, though we only have hints as to why that is. Hope that was helpful!

LoSboccacc 9y ago
> but the network could completely fail once 51% of the units are deleted also the network could have weird results for dropout values around 20%-30% depending on how the robustness was 'learned'
maffydub 9y ago
Thanks for your response - that's very interesting! I haven't read the paper yet, but have added it to my queue.