4 ms·
Doesn't ML work fine with fixed point numbers? I don't think there is a particular use case for very large or very small numbers.
by sltkr 6y ago
Doesn't ML work fine with fixed point numbers? I don't think there is a particular use case for very large or very small numbers.
- sharpneli 6y agoIt's the exact opposite. It really loves large and small numbers, the precision of a number doesn't matter that much That's the reason why they introduced https://en.wikipedia.org/wiki/Bfloat16_floating-point_format https://en.wikipedia.org/wiki/Bfloat16_floating-point_format It's similar to normal half precision float but the exponent is the same as with 32bit float and mantissa has been sacrificed to get room for it.
- oivey 6y agoReferring to ML is so vague that it’s not particularly meaningful. The answer in general is no. There’s been some work showing neural networks can function with low precision floating point, but that’s still pretty different than fixed point.
- PeterisP 6y agoML optimization algorithms generally do require proper handling of very,very small numbers for e.g. calculation of gradients. There's a bunch of work on discretization of models after you've done optimizing them - that works, you can get high-performance inference with low-bit fixed point numbers; but that's after the learning has been done using proper floating point (however, you don't necessarily need high accuracy floating point, e.g. single precision may work betted than double precision simply because it's twice less bytes to copy over).