3 ms·
> Google has also developed the bfloat16 for use in AI accelerators. ARM apparently also supports a nonstandard variation of binary16, but I’m hoping I will nev
by kstenerud 4y ago
> Google has also developed the bfloat16 for use in AI accelerators. ARM apparently also supports a nonstandard variation of binary16, but I’m hoping I will never have to learn the details.
The details of bfloat16 are very simple: Build a standard ieee754 float32, then take only the upper 16 bits. It's a LOT easier than trying to support half-precision float, a form that has so little demand that most platforms don't even support it.
- jb1991 4y agoDepends on your definition of “platform“… nearly all GPUs, including the one in your phone, support it. It’s also a native type on the CPU side on all Apple Silicon and iOS devices.
- stingraycharles 4y agoThere is a huge demand for float16 in machine learning, and as such all modern GPUs support it. I can imagine that reusing ieee754 float32 implies a lot of performance degradation over native instructions?
- mananaysiempre 4y agoThis is ancient history so I might be misremembering, but I think float16 first appeared in NVidia GPUs in the 00s when even GLSL was not yet mainstream and the thing to use were the {ARB,NV}_fragment_program* pseudo-assembly languages.
- shultays 4y ago7 bit only for mantissa?
- why_only_15 4y agoYes. The idea is that in machine learning you care more about dynamic range and being able to represent many numbers than you care about the precise number.