4 ms·
Somewhat because the few bits they save on things like Nan give them a sensible boost on those low precisions. But the best way to go is to explucitly fit the d
by nestorD 3y ago
Somewhat because the few bits they save on things like Nan give them a sensible boost on those low precisions.
But the best way to go is to explucitly fit the distribution of values you are expecting: precisely what Meta did when they introduced their own 16bits format.
- pmarreck 3y agoWhat is the name of the Meta custom 16 bit format?
- nestorD 3y agoHere is the relevant announcement (I believe they made it available in Pytorch as Bfloat16): https://engineering.fb.com/2018/11/08/ai-research/floating-point-math/ https://engineering.fb.com/2018/11/08/ai-research/floating-p...