3 ms·

I’m inexperienced with deep nets - isn’t this just dropout? Nets with dropout will generalize better? well, ok, nets that generalize better can deal with dropou

by rjeli 9y ago

I’m inexperienced with deep nets - isn’t this just dropout? Nets with dropout will generalize better? well, ok, nets that generalize better can deal with dropout

pure-awesome 9y ago
Not quite. This and dropout are a good idea to analyze together, but this is not the same thing. Dropout is done _during training_ of the neural network. In other words, for a given training example / batch of examples, some of the neurons are deactivated. Those that remain are forced to "pick up the slack" of the missing ones, and so this ends up preventing over-fitting. What this is, is taking a neural network which has _already been trained_ and seeing how the output differs if we now drop some of the neurons. It is true that neural networks that have been trained with dropout will probably fare much better under these circumstances.
- mabbo 9y ago
  If I'm understanding you, you're saying that this is more of a way to evaluate whether a network has generalized rather than a method to help it generalize. Is that right?
  - ashraymalhotra 9y ago
    This paper basically proves that neurons which people understand better are not any more "important" to the network than the ones we do not understand as well. It is absolutely true that a generalised network will have much lesser drop off in accuracy as compared to a network which basically just stored the input data.
    - bcheung 9y ago
      Is this a metric that is being used any where? Would be interesting to have 'resilency' metric in addition to the normal 'accuracy' and 'loss' metrics.
- mlazos 9y ago
  This just seems true by definition. If you have a learning algorithm that has generalized well, then adding noise to its input shouldn’t result in a large degradation in accuracy.
bcheung 9y ago
Very similar. Dropout during training will force alternative redundant "routes". This is using dropout to give clues to see what contributes to the final signal.

by rjeli 9y ago

I’m inexperienced with deep nets - isn’t this just dropout? Nets with dropout will generalize better? well, ok, nets that generalize better can deal with dropout

pure-awesome 9y ago

Not quite. This and dropout are a good idea to analyze together, but this is not the same thing. Dropout is done _during training_ of the neural network. In other words, for a given training example / batch of examples, some of the neurons are deactivated. Those that remain are forced to "pick up the slack" of the missing ones, and so this ends up preventing over-fitting. What this is, is taking a neural network which has _already been trained_ and seeing how the output differs if we now drop some of the neurons. It is true that neural networks that have been trained with dropout will probably fare much better under these circumstances.

mabbo 9y ago
If I'm understanding you, you're saying that this is more of a way to evaluate whether a network has generalized rather than a method to help it generalize. Is that right?
- ashraymalhotra 9y ago
  This paper basically proves that neurons which people understand better are not any more "important" to the network than the ones we do not understand as well. It is absolutely true that a generalised network will have much lesser drop off in accuracy as compared to a network which basically just stored the input data.
  - bcheung 9y ago
    Is this a metric that is being used any where? Would be interesting to have 'resilency' metric in addition to the normal 'accuracy' and 'loss' metrics.
mlazos 9y ago
This just seems true by definition. If you have a learning algorithm that has generalized well, then adding noise to its input shouldn’t result in a large degradation in accuracy.

bcheung 9y ago

Very similar. Dropout during training will force alternative redundant "routes". This is using dropout to give clues to see what contributes to the final signal.