3 ms·

With the strong caveat that floating point numbers are technicaly discrete. Anecdotal consequence: 16bits floating point numbers have so much non-linearity in

With the strong caveat that floating point numbers are technicaly discrete. Anecdotal consequence: 16bits floating point numbers have so much non-linearity in their round-off error that you can use them to build neural network with no activation functions (which are traditionally needed to introduces non-linearity).

GregarianChild 7y ago
This is very interesting. I have been wondering about this for years. I have asked neural network people whether it is possible to build a neural network from the non-linearity induced by rounding. I have never received a convincing answer. Would you be able to point me towards a write-up of this fact? And why is this not used in practice? I imagine that how well this works also strongly depends on the kinds of rounding. I imagine that stochastic rounding, or the rounding used in Google's bfloat16 are different in this regard in comparison with standard IEEE floating point rounding.
- unlikelymordant 7y ago
  https://openai.com/blog/nonlinear-computation-in-linear-networks/ https://openai.com/blog/nonlinear-computation-in-linear-netw...
- nestorD 7y ago
  Someone else gave a link to an openai blog post, I personally first heard about it in a post by facebook (on their experiments with small precision). I believe it is not used because it requires 16bits precision which, nowadays, you only get on GPU. People usually train on GPU but then evaluate on CPU (in production) where the discontinuity would be much smaller (as you would use 32 bits precision). Furthermore I don't know if, in practice, that type of discontinuity trains as well as a classical activation function (the gradient propagation might be hindered by the limited precision).
- formalsystem 7y ago
  See https://en.wikipedia.org/wiki/Analog_computer https://en.wikipedia.org/wiki/Analog_computer AFAIK analog computers are still the standard in radars for example and it sound like neural networks would benefit from similar hardware.

GregarianChild 7y ago

This is very interesting. I have been wondering about this for years. I have asked neural network people whether it is possible to build a neural network from the non-linearity induced by rounding. I have never received a convincing answer. Would you be able to point me towards a write-up of this fact? And why is this not used in practice? I imagine that how well this works also strongly depends on the kinds of rounding. I imagine that stochastic rounding, or the rounding used in Google's bfloat16 are different in this regard in comparison with standard IEEE floating point rounding.

unlikelymordant 7y ago
https://openai.com/blog/nonlinear-computation-in-linear-networks/ https://openai.com/blog/nonlinear-computation-in-linear-netw...
nestorD 7y ago
Someone else gave a link to an openai blog post, I personally first heard about it in a post by facebook (on their experiments with small precision). I believe it is not used because it requires 16bits precision which, nowadays, you only get on GPU. People usually train on GPU but then evaluate on CPU (in production) where the discontinuity would be much smaller (as you would use 32 bits precision). Furthermore I don't know if, in practice, that type of discontinuity trains as well as a classical activation function (the gradient propagation might be hindered by the limited precision).
formalsystem 7y ago
See https://en.wikipedia.org/wiki/Analog_computer https://en.wikipedia.org/wiki/Analog_computer AFAIK analog computers are still the standard in radars for example and it sound like neural networks would benefit from similar hardware.