eoskx

7 ms·

1. 61.▲
   by eoskx 3y ago

   Someone commented below that with enough batchnorm/layernorm/etc. and/or gradient clipping you can manage it, but BF16 just makes life easier if you can live without some precision.
2. 62.▲
   Bfloat16 support coming to Apple's Metal and PyTorch [video](developer.apple.com)

   93 points by eoskx 3y ago|53 comments
3. 63.▲
   by eoskx 3y ago

   Somehow missed this from WWDC23, but it looks like Sonoma will add support for bfloat16 with Metal, and there's an active PR to add support with the PyTorch MPS back-end (PR #99272). Since M2 added bfloat16 support at the hardware leve