3 ms·

Those "inference" chips are capable of 16/32 fixed point MADS on par with a current Pascal GPU with better perf/W. So they could be used for training. Just sa

Those "inference" chips are capable of 16/32 fixed point MADS on par with a current Pascal GPU with better perf/W. So they could be used for training. Just sayin'... As to whether the TPU can perform efficient training or whether it chokes like current FPGAs due to insufficient memory bandwidth is a different question. That would be my guess but I'd love to be proven wrong.

yvdriess 10y ago
There is quite a gulf between inference and training, it is not just raw FLOPS. Inference is relatively simple data/control-flow, the weights/biases can be stored read-only and the entire network dataflow can be baked into the chip in case of FPGAs. Training adds the backpropagation step, which performs memory read-update-writes all over the place, has kernels with bad arithmetic intensity and requires SP/DP for most models to converge. I only know of Nervana's ASIC that is promising to be used for training and Google hinted at it for future work. Does anyone know of other processor, ASIC or FPGA's that are built for training?
- scottlegrand2 10y ago
  So like I said, they didn't sufficiently think over the memory subsystem? GPUs can transpose matrices in shared memory and register space in several different ways that are all efficient. I'm guessing that you are describing the transposed matrices required for calculating deltas and weight gradients here? If so, then I see, it's more than just the fact it has one-tenth the bandwidth of a consumer GPU it's just a less flexible architecture in the first place. One would hope they fix that for the second generation TPU which I assume they're working on already. That said I've yet to encounter a network that requires double-precision math to converge. Have you? Half Precision can be tricky, but the same trick Nervana uses (dynamic fixed point) can be emulated in software and it has been used for graphics for decades. I specifically made use of it in an OpenGL engine 20 years ago for high resolution texture sampling. As for Nervana IP, IMO they need to ship something soon or NVIDIA'S going to ship a GPU on par with it.

yvdriess 10y ago

There is quite a gulf between inference and training, it is not just raw FLOPS. Inference is relatively simple data/control-flow, the weights/biases can be stored read-only and the entire network dataflow can be baked into the chip in case of FPGAs. Training adds the backpropagation step, which performs memory read-update-writes all over the place, has kernels with bad arithmetic intensity and requires SP/DP for most models to converge. I only know of Nervana's ASIC that is promising to be used for training and Google hinted at it for future work. Does anyone know of other processor, ASIC or FPGA's that are built for training?

scottlegrand2 10y ago
So like I said, they didn't sufficiently think over the memory subsystem? GPUs can transpose matrices in shared memory and register space in several different ways that are all efficient. I'm guessing that you are describing the transposed matrices required for calculating deltas and weight gradients here? If so, then I see, it's more than just the fact it has one-tenth the bandwidth of a consumer GPU it's just a less flexible architecture in the first place. One would hope they fix that for the second generation TPU which I assume they're working on already. That said I've yet to encounter a network that requires double-precision math to converge. Have you? Half Precision can be tricky, but the same trick Nervana uses (dynamic fixed point) can be emulated in software and it has been used for graphics for decades. I specifically made use of it in an OpenGL engine 20 years ago for high resolution texture sampling. As for Nervana IP, IMO they need to ship something soon or NVIDIA'S going to ship a GPU on par with it.