3 ms·
This is very interesting. I have been wondering about this for years. I have asked neural network people whether it is possible to build a neural network from t
by GregarianChild 7y ago
This is very interesting. I have been wondering about this for years. I have asked neural network people whether it is possible to build a neural network from the non-linearity induced by rounding. I have never received a convincing answer. Would you be able to point me towards a write-up of this fact? And why is this not used in practice?
I imagine that how well this works also strongly depends on the kinds of rounding. I imagine that stochastic rounding, or the rounding used in Google's bfloat16 are different in this regard in comparison with standard IEEE floating point rounding.
- unlikelymordant 7y agohttps://openai.com/blog/nonlinear-computation-in-linear-networks/ https://openai.com/blog/nonlinear-computation-in-linear-netw...
- nestorD 7y agoSomeone else gave a link to an openai blog post, I personally first heard about it in a post by facebook (on their experiments with small precision). I believe it is not used because it requires 16bits precision which, nowadays, you only get on GPU. People usually train on GPU but then evaluate on CPU (in production) where the discontinuity would be much smaller (as you would use 32 bits precision). Furthermore I don't know if, in practice, that type of discontinuity trains as well as a classical activation function (the gradient propagation might be hindered by the limited precision).
- formalsystem 7y agoSee https://en.wikipedia.org/wiki/Analog_computer https://en.wikipedia.org/wiki/Analog_computer AFAIK analog computers are still the standard in radars for example and it sound like neural networks would benefit from similar hardware.