So like I said, they didn't sufficiently think over the memory subsystem? GPUs can transpose matrices in shared memory and register space in several different

So like I said, they didn't sufficiently think over the memory subsystem? GPUs can transpose matrices in shared memory and register space in several different ways that are all efficient. I'm guessing that you are describing the transposed matrices required for calculating deltas and weight gradients here? If so, then I see, it's more than just the fact it has one-tenth the bandwidth of a consumer GPU it's just a less flexible architecture in the first place. One would hope they fix that for the second generation TPU which I assume they're working on already. That said I've yet to encounter a network that requires double-precision math to converge. Have you? Half Precision can be tricky, but the same trick Nervana uses (dynamic fixed point) can be emulated in software and it has been used for graphics for decades. I specifically made use of it in an OpenGL engine 20 years ago for high resolution texture sampling. As for Nervana IP, IMO they need to ship something soon or NVIDIA'S going to ship a GPU on par with it.