3 ms·
Bfloat is specifically designed for ML. It is the native type in Google's TPUs. It is quite good at ML; most models that work with fp32 work with bfloat with
by jlebar 7y ago
Bfloat is specifically designed for ML. It is the native type in Google's TPUs. It is quite good at ML; most models that work with fp32 work with bfloat with no adjustments; that's in contrast to IEEE fp16.
You're right that the mantissa is small. The trick is that you always accumulate into fp32 and then truncate down to 16 bits at the end. You'd do this for any 16-bit floating type.
Source: I work on this at Google.
https://cloud.google.com/tpu/docs/bfloat16 https://cloud.google.com/tpu/docs/bfloat16
- edwintorok 7y agoHave you evaluated alternatives such as posits, Kulisch accumulation, and zfp? https://arxiv.org/pdf/1805.08624.pdf https://arxiv.org/pdf/1805.08624.pdf https://arxiv.org/pdf/1811.01721.pdf https://arxiv.org/pdf/1811.01721.pdf https://insidehpc.com/2018/05/universal-coding-reals-alternatives-ieee-floating-point/ https://insidehpc.com/2018/05/universal-coding-reals-alterna... In particular the latter describes a generic framework that can be used to generate a lot of different number systems. Could hardware implement this, allowing us to compose and choose the number system by just setting some simple flags?