4 ms·

Deep nets love noise, because with so many parameters they are very vulnerable to overfitting. For example, dropout, which is almost ubiquitous for deep learni

Deep nets love noise, because with so many parameters they are very vulnerable to overfitting. For example, dropout, which is almost ubiquitous for deep learning, basically makes activations 'wrong' 50% of the time during training.

p1esk 11y ago
That's not true. NNs don't like noise, there have been a lot of research done about effect of noise on NNs in the 90s. Random noise over a certain threshold will progressively degrade the performance of NNs, and below the threshold will have no effect. Dropout is not the same as random noise. By using dropout you eliminate some neurons from making contribution. As a result, you effectively train many smaller nets, each one adjusting its available weights to perform the same task. During testing, there's no noise - all neurons are back in business and contributing.
- srean 11y ago
  I am pretty sure that parent meant DNNs love to fit noise. So you two are in strong agreement.
  - p1esk 11y ago
    No, we are talking about a random electrical circuit noise in the analog NN hardware. Of course, if the noise is known and fixed, the net could learn to compensate (to a certain extent). The noise we are talking about is like when you put your finger on the chip, and raise its temperature by 10 degrees, the whole thing needs to be retrained.
    - taliesinb 11y ago
      > The noise we are talking about is like when you put your finger on the chip, and raise its temperature by 10 degrees, the whole thing needs to be retrained. What would change with temperature that would require retraining? Are you saying the output of an op could depend sensitively on temperature, or that higher temperatures would increase things like thermal or shot noise? Why would the latter require retraining?
      - p1esk 11y ago
        Current depends on threshold voltage and carrier mobility, which are temperature dependent.
- taliesinb 11y ago
  > Dropout is not the same as random noise. By using dropout you eliminate some neurons from making contribution. As a result, you effectively train many smaller nets, each one adjusting its available weights to perform the same task. During testing, there's no noise - all neurons are back in business and contributing. I was speaking loosely -- dropout is multiplicative Bernoulli noise on the hidden layers. > That's not true. NNs don't like noise, there have been a lot of research done about effect of noise on NNs in the 90s. Random noise over a certain threshold will progressively degrade the performance of NNs, and below the threshold will have no effect. I'd argue that dropout (and its predecessor in denoising autoencoders) are perfectly valid to see as noise, albeit multiplicative.
  - p1esk 11y ago
    You are missing my point - with dropout, you don't have any noise during the operation of the net. The noise we are talking about (circuit noise) is always present.

p1esk 11y ago

That's not true. NNs don't like noise, there have been a lot of research done about effect of noise on NNs in the 90s. Random noise over a certain threshold will progressively degrade the performance of NNs, and below the threshold will have no effect. Dropout is not the same as random noise. By using dropout you eliminate some neurons from making contribution. As a result, you effectively train many smaller nets, each one adjusting its available weights to perform the same task. During testing, there's no noise - all neurons are back in business and contributing.

srean 11y ago
I am pretty sure that parent meant DNNs love to fit noise. So you two are in strong agreement.
- p1esk 11y ago
  No, we are talking about a random electrical circuit noise in the analog NN hardware. Of course, if the noise is known and fixed, the net could learn to compensate (to a certain extent). The noise we are talking about is like when you put your finger on the chip, and raise its temperature by 10 degrees, the whole thing needs to be retrained.
  - taliesinb 11y ago
    > The noise we are talking about is like when you put your finger on the chip, and raise its temperature by 10 degrees, the whole thing needs to be retrained. What would change with temperature that would require retraining? Are you saying the output of an op could depend sensitively on temperature, or that higher temperatures would increase things like thermal or shot noise? Why would the latter require retraining?
    - p1esk 11y ago
      Current depends on threshold voltage and carrier mobility, which are temperature dependent.
taliesinb 11y ago
> Dropout is not the same as random noise. By using dropout you eliminate some neurons from making contribution. As a result, you effectively train many smaller nets, each one adjusting its available weights to perform the same task. During testing, there's no noise - all neurons are back in business and contributing. I was speaking loosely -- dropout is multiplicative Bernoulli noise on the hidden layers. > That's not true. NNs don't like noise, there have been a lot of research done about effect of noise on NNs in the 90s. Random noise over a certain threshold will progressively degrade the performance of NNs, and below the threshold will have no effect. I'd argue that dropout (and its predecessor in denoising autoencoders) are perfectly valid to see as noise, albeit multiplicative.
- p1esk 11y ago
  You are missing my point - with dropout, you don't have any noise during the operation of the net. The noise we are talking about (circuit noise) is always present.