3 ms·

Hmmm... I see you're coming in from another angle but its also related to noise. ------- I guess I was relating analog-noise to quantization noise of the FP16

Hmmm... I see you're coming in from another angle but its also related to noise. ------- I guess I was relating analog-noise to quantization noise of the FP16 format. For example, the number 4098 cannot be represented in FP16, you round that number to 4096 or 4100. This is seen as "noise" perhaps in your perspective, but I'm seeing it as "loss of precision". From my perspective, neural nets are clearly able to work even with low-precision (ie: numbers like 4098 being rounded to 4096 in every step of every calculation). "Noise" in my model is this "number shifting" effect. So from my side of things: neural nets seem to work with 8-bits or 16-bits of precision. And analog circuits are able to achieve that level of precision in practice (though its called 2-digits of precision or 5-digits of precision in the analog-world, maybe SNR ratio of 24db or 48db). --------- But from your perspective, noisy inputs are themselves a strategy that people use to train neural nets with. Literally adding noise and then compensating with a larger model / more training. Maybe its all noise... one artificial, the other caused by physical limitation (quantization noise is... noise after all), but it was a bit confusing for a post or two because I think we were coming in from different perspectives for a second.

ttqidebg 5y ago
It is counter intuitive. Noise is often an essential part to training, especially when it comes to generalization. Sometimes there is enough noise in the training data to be sufficient. Controlling how much noise and where it goes so that you end up with something usable is still a bit of an art. Not just in designing the model but also the training and distillation process. It’s a pain in the ass, but if the perf is there it can be worth it. I’m inference speed bottlenecked so I’m keeping an eye on this sort of tech.
- dragontamer 5y ago
  I've done enough neural net stuff to know that noise can knock you out of local minima / local maxima. Well, not even neural nets... machine learning in general benefits. But there's a difference between noise that you can control (ie: on during training and you can increase/decrease at will) and noise that's fundamental to the system and uncontrollable. Still, since all neural nets show a resilience to a degree of noise, maybe that makes analog-circuits suddenly viable.

ttqidebg 5y ago

It is counter intuitive. Noise is often an essential part to training, especially when it comes to generalization. Sometimes there is enough noise in the training data to be sufficient. Controlling how much noise and where it goes so that you end up with something usable is still a bit of an art. Not just in designing the model but also the training and distillation process. It’s a pain in the ass, but if the perf is there it can be worth it. I’m inference speed bottlenecked so I’m keeping an eye on this sort of tech.

dragontamer 5y ago
I've done enough neural net stuff to know that noise can knock you out of local minima / local maxima. Well, not even neural nets... machine learning in general benefits. But there's a difference between noise that you can control (ie: on during training and you can increase/decrease at will) and noise that's fundamental to the system and uncontrollable. Still, since all neural nets show a resilience to a degree of noise, maybe that makes analog-circuits suddenly viable.