3 ms·
Google's TPUs use them. But it has been for a year. I don't agree with the "new" or "Intel's" in the title.
by marcyb5st 8y ago
Google's TPUs use them. But it has been for a year. I don't agree with the "new" or "Intel's" in the title.
- masklinn 8y agoAnd TPU uses them because Tensorflow uses them, it's been present since the first public commit: https://github.com/tensorflow/tensorflow/blob/f41959ccb2d9d4c722fe8fc3351401d53bcf4900/tensorflow/core/framework/bfloat16.h https://github.com/tensorflow/tensorflow/blob/f41959ccb2d9d4...
- marcyb5st 8y agoThank you! I didn't know this. I thought they introduced them shortly after announcing TPU v1 in the 2016 (or 2017, can't remember) Google I/O.
- twtw 8y agoI would be extremely surprised if the motivation for putting bfloat16 in tensorflow was not the TPU. That first public commit was ~1.5 years before TPUv2 was announced at I/O, so it was almost certainly already in development.
- vrv 8y agobfloat16 was first in DistBelief, so it actually predates TensorFlow and TPUs (I worked on both systems). IIRC the motivation was more about minimizing parameter exchange bandwidth for large-scale CPU clusters rather than minimizing memory bandwidth within accelerators, but the idea generalized.