2 ms·
Does bfloat16 have any other uses than deep learning?
by im3w1l 7y ago
Does bfloat16 have any other uses than deep learning?
- londons_explore 7y agoIt isn't really very good for machine learning - machine learning doesn't need 8 bits of exponent. For weights during training, 7 bits of mantissa also seems a bit low - it's common for weights to adjust much less than 1% during a single batch of training, which this couldn't represent. I think this is more a "we want something which is faster but is compatible with existing code written for fp32's".
- jlebar 7y agoBfloat is specifically designed for ML. It is the native type in Google's TPUs. It is quite good at ML; most models that work with fp32 work with bfloat with no adjustments; that's in contrast to IEEE fp16. You're right that the mantissa is small. The trick is that you always accumulate into fp32 and then truncate down to 16 bits at the end. You'd do this for any 16-bit floating type. Source: I work on this at Google. https://cloud.google.com/tpu/docs/bfloat16 https://cloud.google.com/tpu/docs/bfloat16
- edwintorok 7y agoHave you evaluated alternatives such as posits, Kulisch accumulation, and zfp? https://arxiv.org/pdf/1805.08624.pdf https://arxiv.org/pdf/1805.08624.pdf https://arxiv.org/pdf/1811.01721.pdf https://arxiv.org/pdf/1811.01721.pdf https://insidehpc.com/2018/05/universal-coding-reals-alternatives-ieee-floating-point/ https://insidehpc.com/2018/05/universal-coding-reals-alterna... In particular the latter describes a generic framework that can be used to generate a lot of different number systems. Could hardware implement this, allowing us to compose and choose the number system by just setting some simple flags?