4 ms·

The ASIC processors Google use are for inference, not training. Having efficient inference is great, but training is what uses up most of the compute power. Th

The ASIC processors Google use are for inference, not training. Having efficient inference is great, but training is what uses up most of the compute power. The battle between GPUs and CPUs in the datacenter is real, but I wouldn't count out regular CPUs yet. Even with GPU-ready frameworks like Tensorflow, Theano, and Pytorch, it still requires a fair amount of domain expertise to get good performance out of GPUs. These frameworks can automatically offload compute kernels to the GPU, but actually getting good performance still requires you to understand the memory model, what data to pin in GPU memory, how to use the limited GPU memory effectively, etc. The new instruction sets that Intel have been adding (e.g. all of the recent vectorized FMA instructions) go a long way to making Intel CPUs competitive for a lot of this ASIC/GPU stuff out of the box for things like neural networks. Nvidia has been off to a really great start in this area, and they have carved out a very respectable lead, but I think it's premature to say that Intel (or AMD) are having an existential crisis.

puzzle 10y ago
Correction: the ASIC processors that Google used in 2015 and just talked about, two years later, were for inference. We don't know what they use now and how. The paper hints that training might happen in a later revision. You have to wonder why they started with inference and not the other way round. Perhaps it's the case that you train a model once, then use it for predictions many, many, many times, thus using more aggregate compute power. (I used to work at Google, but the above is based on public information and speculation alone.) The paper does confirm that CPUs are closer to GPUs for smaller or latency-sensitive workloads than most people assumed. At least on 2015-vintage hardware, that is. And they're a lot less trouble to set up and provision.
- jtmcmc 10y ago
  Training can happen asynchronously and in batch - inference needs to be low latency and have massive parallelism because it's powering millions of android devices voice to text recognition - for instance.
- _delirium 10y ago
  > You have to wonder why they started with inference and not the other way round. At least from what I know of ASICs (which is admittedly pretty high-level), this direction is the natural way I'd expect it to go, because the inference problem is much closer in scale to the kind of problem ASICs are already widely used for and good at. Fast inference can be done with a chip of a size similar to what you see in existing ASICs, like specialized DSP chips. Speeding up training is a lot more speculative. You could try scaling up to massively powerful individual chips that can each replace an NVidia GPU (but with lower cost / lower power budget), or improve distributed training so you can effectively use a much larger number of much weaker customized chips. And there are more significant memory and memory-bandwidth issues with training, so it's less of the pure-compute type of problem that ASICs are more often clear wins for. All of which is probably possible with enough effort, but less straightforward, especially to end up with something that beats GPUs on relevant metrics.
Analemma_ 10y ago
> The ASIC processors Google use are for inference, not training. Having efficient inference is great, but training is what uses up most of the compute power. I think that was the GP's point? The training is done on GPUs, the execution is done on ASICs, and neither step involves Intel. That's what alarms them. Dedicated hardware might help this, but that seems iffy to me. ML and neural networks are still a fast-evolving field, and instructions that might help today could be useless next year. ASICs can be thrown out and replaced, but instructions that Intel adds to its ISA will be there forever.
paulsutter 10y ago
There's nothing about the TPU architecture that prevents use for training. Training is also done as tensor operations, and training can be done at limited precision[1]. Just because the first TPU was memory-limited doesn't mean TPUs can never be used for training. [1] https://arxiv.org/abs/1606.06160 https://arxiv.org/abs/1606.06160
- shaklee3 10y ago
  Sure, but it's a garbage in, garbage out thing. If you're using those in production in something like, say, google images, you wouldn't want to lose accuracy on a one-time pass. Anything can be quantized lower, but you start making the tradeoff of losing information that may affect the quality of your product later.
- pjscott 10y ago
  The key problem with the TPU architecture for training is that its on-die buffer space is fairly limited and its memory bandwidth is currently nowhere near that of a GPU. This matters if you've got a lot of intermediate buffers that need to be alive at the same time for backpropagation. Also, there's the less fundamental matter of convenience: while there are some clever ways to train with limited precision gradients, it's not yet as easy as not doing that. One style of training that would be a good fit for the current TPU architecture is some of the recent work on reinforcement learning without gradients: https://arxiv.org/abs/1703.03864 https://arxiv.org/abs/1703.03864
ganfortran 10y ago
> for inference True, but for training, in near future, it is still going to be GPU, probably Nvidia GPU only, not too much Intel could do. > requires a fair amount of domain expertise to get good performance out of GPUs Rightfully so, but invisible to end user/programmer, if you ever use those frameworks and understand their abstraction level. This falls into the realm of few experienced engineers in Google/FAIR/Nvidia and other respected institutions, who understand their business well. I think Intel is indeed having an existential crisis: they are losing the future, the future of computation is shifting, away from their comfort zone, and they yet figured out how to stop or catch up with the new trend. If CPU becomes a stock controller component, just like a WiFi-chip, how is Intel going to justify its hefty marketing valuation? It is about time for Intel to do some hard soul searching.
scottlegrand2 10y ago
Those "inference" chips are capable of 16/32 fixed point MADS on par with a current Pascal GPU with better perf/W. So they could be used for training. Just sayin'... As to whether the TPU can perform efficient training or whether it chokes like current FPGAs due to insufficient memory bandwidth is a different question. That would be my guess but I'd love to be proven wrong.
- yvdriess 10y ago
  There is quite a gulf between inference and training, it is not just raw FLOPS. Inference is relatively simple data/control-flow, the weights/biases can be stored read-only and the entire network dataflow can be baked into the chip in case of FPGAs. Training adds the backpropagation step, which performs memory read-update-writes all over the place, has kernels with bad arithmetic intensity and requires SP/DP for most models to converge. I only know of Nervana's ASIC that is promising to be used for training and Google hinted at it for future work. Does anyone know of other processor, ASIC or FPGA's that are built for training?
  - scottlegrand2 10y ago
    So like I said, they didn't sufficiently think over the memory subsystem? GPUs can transpose matrices in shared memory and register space in several different ways that are all efficient. I'm guessing that you are describing the transposed matrices required for calculating deltas and weight gradients here? If so, then I see, it's more than just the fact it has one-tenth the bandwidth of a consumer GPU it's just a less flexible architecture in the first place. One would hope they fix that for the second generation TPU which I assume they're working on already. That said I've yet to encounter a network that requires double-precision math to converge. Have you? Half Precision can be tricky, but the same trick Nervana uses (dynamic fixed point) can be emulated in software and it has been used for graphics for decades. I specifically made use of it in an OpenGL engine 20 years ago for high resolution texture sampling. As for Nervana IP, IMO they need to ship something soon or NVIDIA'S going to ship a GPU on par with it.

puzzle 10y ago

Correction: the ASIC processors that Google used in 2015 and just talked about, two years later, were for inference. We don't know what they use now and how. The paper hints that training might happen in a later revision. You have to wonder why they started with inference and not the other way round. Perhaps it's the case that you train a model once, then use it for predictions many, many, many times, thus using more aggregate compute power. (I used to work at Google, but the above is based on public information and speculation alone.) The paper does confirm that CPUs are closer to GPUs for smaller or latency-sensitive workloads than most people assumed. At least on 2015-vintage hardware, that is. And they're a lot less trouble to set up and provision.

jtmcmc 10y ago
Training can happen asynchronously and in batch - inference needs to be low latency and have massive parallelism because it's powering millions of android devices voice to text recognition - for instance.
_delirium 10y ago
> You have to wonder why they started with inference and not the other way round. At least from what I know of ASICs (which is admittedly pretty high-level), this direction is the natural way I'd expect it to go, because the inference problem is much closer in scale to the kind of problem ASICs are already widely used for and good at. Fast inference can be done with a chip of a size similar to what you see in existing ASICs, like specialized DSP chips. Speeding up training is a lot more speculative. You could try scaling up to massively powerful individual chips that can each replace an NVidia GPU (but with lower cost / lower power budget), or improve distributed training so you can effectively use a much larger number of much weaker customized chips. And there are more significant memory and memory-bandwidth issues with training, so it's less of the pure-compute type of problem that ASICs are more often clear wins for. All of which is probably possible with enough effort, but less straightforward, especially to end up with something that beats GPUs on relevant metrics.

Analemma_ 10y ago

> The ASIC processors Google use are for inference, not training. Having efficient inference is great, but training is what uses up most of the compute power. I think that was the GP's point? The training is done on GPUs, the execution is done on ASICs, and neither step involves Intel. That's what alarms them. Dedicated hardware might help this, but that seems iffy to me. ML and neural networks are still a fast-evolving field, and instructions that might help today could be useless next year. ASICs can be thrown out and replaced, but instructions that Intel adds to its ISA will be there forever.

paulsutter 10y ago

There's nothing about the TPU architecture that prevents use for training. Training is also done as tensor operations, and training can be done at limited precision[1]. Just because the first TPU was memory-limited doesn't mean TPUs can never be used for training. [1] https://arxiv.org/abs/1606.06160 https://arxiv.org/abs/1606.06160

shaklee3 10y ago
Sure, but it's a garbage in, garbage out thing. If you're using those in production in something like, say, google images, you wouldn't want to lose accuracy on a one-time pass. Anything can be quantized lower, but you start making the tradeoff of losing information that may affect the quality of your product later.
pjscott 10y ago
The key problem with the TPU architecture for training is that its on-die buffer space is fairly limited and its memory bandwidth is currently nowhere near that of a GPU. This matters if you've got a lot of intermediate buffers that need to be alive at the same time for backpropagation. Also, there's the less fundamental matter of convenience: while there are some clever ways to train with limited precision gradients, it's not yet as easy as not doing that. One style of training that would be a good fit for the current TPU architecture is some of the recent work on reinforcement learning without gradients: https://arxiv.org/abs/1703.03864 https://arxiv.org/abs/1703.03864

ganfortran 10y ago

> for inference True, but for training, in near future, it is still going to be GPU, probably Nvidia GPU only, not too much Intel could do. > requires a fair amount of domain expertise to get good performance out of GPUs Rightfully so, but invisible to end user/programmer, if you ever use those frameworks and understand their abstraction level. This falls into the realm of few experienced engineers in Google/FAIR/Nvidia and other respected institutions, who understand their business well. I think Intel is indeed having an existential crisis: they are losing the future, the future of computation is shifting, away from their comfort zone, and they yet figured out how to stop or catch up with the new trend. If CPU becomes a stock controller component, just like a WiFi-chip, how is Intel going to justify its hefty marketing valuation? It is about time for Intel to do some hard soul searching.

scottlegrand2 10y ago

Those "inference" chips are capable of 16/32 fixed point MADS on par with a current Pascal GPU with better perf/W. So they could be used for training. Just sayin'... As to whether the TPU can perform efficient training or whether it chokes like current FPGAs due to insufficient memory bandwidth is a different question. That would be my guess but I'd love to be proven wrong.

yvdriess 10y ago
There is quite a gulf between inference and training, it is not just raw FLOPS. Inference is relatively simple data/control-flow, the weights/biases can be stored read-only and the entire network dataflow can be baked into the chip in case of FPGAs. Training adds the backpropagation step, which performs memory read-update-writes all over the place, has kernels with bad arithmetic intensity and requires SP/DP for most models to converge. I only know of Nervana's ASIC that is promising to be used for training and Google hinted at it for future work. Does anyone know of other processor, ASIC or FPGA's that are built for training?
- scottlegrand2 10y ago
  So like I said, they didn't sufficiently think over the memory subsystem? GPUs can transpose matrices in shared memory and register space in several different ways that are all efficient. I'm guessing that you are describing the transposed matrices required for calculating deltas and weight gradients here? If so, then I see, it's more than just the fact it has one-tenth the bandwidth of a consumer GPU it's just a less flexible architecture in the first place. One would hope they fix that for the second generation TPU which I assume they're working on already. That said I've yet to encounter a network that requires double-precision math to converge. Have you? Half Precision can be tricky, but the same trick Nervana uses (dynamic fixed point) can be emulated in software and it has been used for graphics for decades. I specifically made use of it in an OpenGL engine 20 years ago for high resolution texture sampling. As for Nervana IP, IMO they need to ship something soon or NVIDIA'S going to ship a GPU on par with it.