4 ms·

Correction: the ASIC processors that Google used in 2015 and just talked about, two years later, were for inference. We don't know what they use now and how. Th

by puzzle 10y ago

Correction: the ASIC processors that Google used in 2015 and just talked about, two years later, were for inference. We don't know what they use now and how. The paper hints that training might happen in a later revision. You have to wonder why they started with inference and not the other way round. Perhaps it's the case that you train a model once, then use it for predictions many, many, many times, thus using more aggregate compute power. (I used to work at Google, but the above is based on public information and speculation alone.) The paper does confirm that CPUs are closer to GPUs for smaller or latency-sensitive workloads than most people assumed. At least on 2015-vintage hardware, that is. And they're a lot less trouble to set up and provision.

jtmcmc 10y ago
Training can happen asynchronously and in batch - inference needs to be low latency and have massive parallelism because it's powering millions of android devices voice to text recognition - for instance.
_delirium 10y ago
> You have to wonder why they started with inference and not the other way round. At least from what I know of ASICs (which is admittedly pretty high-level), this direction is the natural way I'd expect it to go, because the inference problem is much closer in scale to the kind of problem ASICs are already widely used for and good at. Fast inference can be done with a chip of a size similar to what you see in existing ASICs, like specialized DSP chips. Speeding up training is a lot more speculative. You could try scaling up to massively powerful individual chips that can each replace an NVidia GPU (but with lower cost / lower power budget), or improve distributed training so you can effectively use a much larger number of much weaker customized chips. And there are more significant memory and memory-bandwidth issues with training, so it's less of the pure-compute type of problem that ASICs are more often clear wins for. All of which is probably possible with enough effort, but less straightforward, especially to end up with something that beats GPUs on relevant metrics.

by puzzle 10y ago

jtmcmc 10y ago

Training can happen asynchronously and in batch - inference needs to be low latency and have massive parallelism because it's powering millions of android devices voice to text recognition - for instance.

_delirium 10y ago

> You have to wonder why they started with inference and not the other way round. At least from what I know of ASICs (which is admittedly pretty high-level), this direction is the natural way I'd expect it to go, because the inference problem is much closer in scale to the kind of problem ASICs are already widely used for and good at. Fast inference can be done with a chip of a size similar to what you see in existing ASICs, like specialized DSP chips. Speeding up training is a lot more speculative. You could try scaling up to massively powerful individual chips that can each replace an NVidia GPU (but with lower cost / lower power budget), or improve distributed training so you can effectively use a much larger number of much weaker customized chips. And there are more significant memory and memory-bandwidth issues with training, so it's less of the pure-compute type of problem that ASICs are more often clear wins for. All of which is probably possible with enough effort, but less straightforward, especially to end up with something that beats GPUs on relevant metrics.