3 ms·

Could we get a Bfloat32?

• _Nat_ 7y ago
  Hard to infer humor here sometimes -- were you kidding? Just in case, since I can see someone else being serious about this, I think the gist is that neural-networks tend to be fairly approximate things such that we're not particularly concerned with having a lot of precision in many cases. This use-case wouldn't seem to demand a 32-bit variant too often. But.. if you want it anyway... Higher-bit extensions would seem to be floating-point values that favor range-over-precision more than typical floating-point numerics with the same bit-count. If we take that to an extreme, we can talk about ranges over infinities and infinitesimals -- this is, much like the hyperreal-number system [1]. And ya know what's funny? Some guy's been pushing for such a primitive numeric data type [2] since the early-2000's [3]! [1]: https://en.wikipedia.org/wiki/Hyperreal_number https://en.wikipedia.org/wiki/Hyperreal_number [2]: http://wwwinfo.deis.unical.it/yaro/EMSS_Sergeyev.pdf http://wwwinfo.deis.unical.it/yaro/EMSS_Sergeyev.pdf [3]: https://patents.google.com/patent/US7860914 https://patents.google.com/patent/US7860914
• ndesaulniers 7y ago
  You already have that; it's more formally called IEEE 754 single precision. Requires double the memory bandwidth (bad) for extra precision that back propagation would have corrected for anyways.
  • ScottBurson 7y ago
    Seems like BFloat32 would have the same exponent range as Float64 (double precision), and a correspondingly shorter significand.