4 ms·
Intel's TPU2 and 3, as well as NVidia's V100 series get great tensor performance on FP16 and FP8. There are many implementation and algorithmic risks associate
by hpcjoe 8y ago
Intel's TPU2 and 3, as well as NVidia's V100 series get great tensor performance on FP16 and FP8. There are many implementation and algorithmic risks associated with using such small FP sizes, especially if you are working on problems related to convergence, and you have significant roundoff error.
Smaller FP sizes mean you can run more of these in parallel. Smaller FP sizes mean (much) larger roundoff error, fewer bits of precision kept.
- bitL 8y agoYup, does the lower precision in Deep Learning matter though? You are basically constructing some hyper-dimensional soft boundaries separating classes of objects, and even between training runs on FP64/128/256/512 one can end up with very different boundaries - whether these come from some random training/sample order or lowered precision probably doesn't play a large role, if it speeds up training 10x? If you really want more precise boundaries, you can train on 8/16-bit first and then use transfer learning on pre-trained low-precision model with higher precision as your last step anyway.
- drewg123 8y agoThe TPU is a google chip, not an Intel chip.