3 ms·

Sure, but it’s effectively random which way the model learns the parameter impact and how values are used by subsequent layers. I’m assuming layers in the NN.

Sure, but it’s effectively random which way the model learns the parameter impact and how values are used by subsequent layers. I’m assuming layers in the NN. Adding noise with things like dropout encourages the model to ‘grow’ redundancy.

dragontamer 5y ago
Hmmm... I see you're coming in from another angle but its also related to noise. ------- I guess I was relating analog-noise to quantization noise of the FP16 format. For example, the number 4098 cannot be represented in FP16, you round that number to 4096 or 4100. This is seen as "noise" perhaps in your perspective, but I'm seeing it as "loss of precision". From my perspective, neural nets are clearly able to work even with low-precision (ie: numbers like 4098 being rounded to 4096 in every step of every calculation). "Noise" in my model is this "number shifting" effect. So from my side of things: neural nets seem to work with 8-bits or 16-bits of precision. And analog circuits are able to achieve that level of precision in practice (though its called 2-digits of precision or 5-digits of precision in the analog-world, maybe SNR ratio of 24db or 48db). --------- But from your perspective, noisy inputs are themselves a strategy that people use to train neural nets with. Literally adding noise and then compensating with a larger model / more training. Maybe its all noise... one artificial, the other caused by physical limitation (quantization noise is... noise after all), but it was a bit confusing for a post or two because I think we were coming in from different perspectives for a second.
- ttqidebg 5y ago
  It is counter intuitive. Noise is often an essential part to training, especially when it comes to generalization. Sometimes there is enough noise in the training data to be sufficient. Controlling how much noise and where it goes so that you end up with something usable is still a bit of an art. Not just in designing the model but also the training and distillation process. It’s a pain in the ass, but if the perf is there it can be worth it. I’m inference speed bottlenecked so I’m keeping an eye on this sort of tech.
  - dragontamer 5y ago
    I've done enough neural net stuff to know that noise can knock you out of local minima / local maxima. Well, not even neural nets... machine learning in general benefits. But there's a difference between noise that you can control (ie: on during training and you can increase/decrease at will) and noise that's fundamental to the system and uncontrollable. Still, since all neural nets show a resilience to a degree of noise, maybe that makes analog-circuits suddenly viable.
joe_the_user 5y ago
The problem I'd see is that the raw randomness of electronics is not the nice, uniform distribution that indeed may help NNs learns. It can be, often is, biased, skewed, time-varying and just about anything else. It could vary from one part of the chip to another, one chip to another and so-forth. The various circuits exists to turn a wide range of noise into digital ons and offs. Which means the raw circuits can have this wide range unless controlled. It's worth considering that no one uses raw electronic noise for random/pseudo-random number generators. And that's an application that's in demand.
- ttqidebg 5y ago
  Dropout, the example given, is not a normally distributed error - more like a salt noise. The impact on subsequent layers isn’t Normal either. My point is certain error modes other than normally distributed may not be so bad. E.g. skew as roughly half of the redundant paths for information will be inverted in the latent space. I’m not a hardware guy, but in my experience DNNs can be made rather resilient. To me it looks like it could work. If it does great.

dragontamer 5y ago

Hmmm... I see you're coming in from another angle but its also related to noise. ------- I guess I was relating analog-noise to quantization noise of the FP16 format. For example, the number 4098 cannot be represented in FP16, you round that number to 4096 or 4100. This is seen as "noise" perhaps in your perspective, but I'm seeing it as "loss of precision". From my perspective, neural nets are clearly able to work even with low-precision (ie: numbers like 4098 being rounded to 4096 in every step of every calculation). "Noise" in my model is this "number shifting" effect. So from my side of things: neural nets seem to work with 8-bits or 16-bits of precision. And analog circuits are able to achieve that level of precision in practice (though its called 2-digits of precision or 5-digits of precision in the analog-world, maybe SNR ratio of 24db or 48db). --------- But from your perspective, noisy inputs are themselves a strategy that people use to train neural nets with. Literally adding noise and then compensating with a larger model / more training. Maybe its all noise... one artificial, the other caused by physical limitation (quantization noise is... noise after all), but it was a bit confusing for a post or two because I think we were coming in from different perspectives for a second.

ttqidebg 5y ago
It is counter intuitive. Noise is often an essential part to training, especially when it comes to generalization. Sometimes there is enough noise in the training data to be sufficient. Controlling how much noise and where it goes so that you end up with something usable is still a bit of an art. Not just in designing the model but also the training and distillation process. It’s a pain in the ass, but if the perf is there it can be worth it. I’m inference speed bottlenecked so I’m keeping an eye on this sort of tech.
- dragontamer 5y ago
  I've done enough neural net stuff to know that noise can knock you out of local minima / local maxima. Well, not even neural nets... machine learning in general benefits. But there's a difference between noise that you can control (ie: on during training and you can increase/decrease at will) and noise that's fundamental to the system and uncontrollable. Still, since all neural nets show a resilience to a degree of noise, maybe that makes analog-circuits suddenly viable.

joe_the_user 5y ago

The problem I'd see is that the raw randomness of electronics is not the nice, uniform distribution that indeed may help NNs learns. It can be, often is, biased, skewed, time-varying and just about anything else. It could vary from one part of the chip to another, one chip to another and so-forth. The various circuits exists to turn a wide range of noise into digital ons and offs. Which means the raw circuits can have this wide range unless controlled. It's worth considering that no one uses raw electronic noise for random/pseudo-random number generators. And that's an application that's in demand.

ttqidebg 5y ago
Dropout, the example given, is not a normally distributed error - more like a salt noise. The impact on subsequent layers isn’t Normal either. My point is certain error modes other than normally distributed may not be so bad. E.g. skew as roughly half of the redundant paths for information will be inverted in the latent space. I’m not a hardware guy, but in my experience DNNs can be made rather resilient. To me it looks like it could work. If it does great.