3 ms·
What leads you to think that TPUs are not useful for training?
by halflings 10y ago
What leads you to think that TPUs are not useful for training?
- alfalfasprout 10y agoProbably memory. Unless their ASICs have some kind of ultra fast on-chip memory. That's why GPUs still reign supreme. FPGAs have started seeing use for inference for those of us without the billion dollar budgets needed to develop ASICs.
- ogrisel 10y agoThe aforementioned google translate paper suggests so. My hunch is that TPUs work with a mix of 8 bit and 16 bit integer arithmetic on quantized networks with quantized data [1]. As far as I know nobody has managed to make SGD work properly with integer weights and that could be a reason why TPUs are not used for training (yet). [1] https://petewarden.com/2016/05/03/how-to-quantize-neural-networks-with-tensorflow/ https://petewarden.com/2016/05/03/how-to-quantize-neural-net... I could be completely wrong though.