4 ms·
16-bit floats with an 8-bit exponent and a 7+1 bit mantissa are popular for neural networks, because they have the same range as standard 32-bit floats while ta
by CodesInChaos 7y ago
16-bit floats with an 8-bit exponent and a 7+1 bit mantissa are popular for neural networks, because they have the same range as standard 32-bit floats while taking have the memory and memory bandwidth.
https://en.wikipedia.org/wiki/Bfloat16_floating-point_format https://en.wikipedia.org/wiki/Bfloat16_floating-point_format
- helltone 7y agoInteresting, although I believe most neural nets nowadays have moved to use linear (relu) activation, which again removes the need for exponent and would work really well with fixed point. Here's a reference using 16-bit fixed point neural nets http://ieeexplore.ieee.org/document/7011421/?part=1 http://ieeexplore.ieee.org/document/7011421/?part=1