2 ms·
If floating point values were real numbers, simple matrix multiplications would not produce nonlinearities. However, researchers at OpenAI demonstrated that th
by stochastic_monk 8y ago
If floating point values were real numbers, simple matrix multiplications would not produce nonlinearities.
However, researchers at OpenAI demonstrated that the use of subnormal floating point values and their discontinuity [0] provides sufficient nonlinearity to learn nonlinear associations.
[0] https://blog.openai.com/nonlinear-computation-in-linear-networks/ https://blog.openai.com/nonlinear-computation-in-linear-netw...
- crankylinuxuser 8y agoThen, would it be better to store rationals as 2 ints with understood division, and irrationals as 2 ints as understood powers? Why are we working with these numbers so imprecisely, and with such discontinuity? When I've done math work in school, I always worked with the primitives without approximating.. 2^.5 was handled as such unless asked for an approximate answer. I also ran across this in AutoCAD. Because they force arbitrary precision, one cannot define a sqrt length. 1.414 doesn't reach all the way on a r-triangle with 1,1 .
- MauranKilom 8y agoThat post completely disregards the fact that subnormals are usually orders of magnitude slower to work with for common hardware.