Yup, does the lower precision in Deep Learning matter though? You are basically constructing some hyper-dimensional soft boundaries separating classes of object

Yup, does the lower precision in Deep Learning matter though? You are basically constructing some hyper-dimensional soft boundaries separating classes of objects, and even between training runs on FP64/128/256/512 one can end up with very different boundaries - whether these come from some random training/sample order or lowered precision probably doesn't play a large role, if it speeds up training 10x? If you really want more precise boundaries, you can train on 8/16-bit first and then use transfer learning on pre-trained low-precision model with higher precision as your last step anyway.