10 ms·

Train TensorFlow models faster and at lower cost on Cloud TPU Pods

105 points by frankchn 8y ago

polskibus 8y ago
What a pity Google TPUs are locked in Google Cloud - many would like to use them on their own, in already owned servers. It kind of slows the ML progress, such restriction of access to the product.
- zak 8y ago
  You can also use Cloud TPUs for free in your browser via Colab. Several sample notebooks are available here: https://www.tensorflow.org/tfrc/ https://www.tensorflow.org/tfrc/
  - api 8y ago
    I think they mean actually buying the chips. This model of putting revolutionary new hardware in the cloud only is kind of worrisome given the future implications.
    - eanzenberg 8y ago
      "Revolutionary"
- kajecounterhack 8y ago
  How these chips work is pretty reliant on Google's specific infrastructure and there's a stark difference between the work needed to offer a service and selling chips. It isn't a matter of bad will or trying to slow ML progress but a matter of what's practical.
  - polskibus 8y ago
    Maybe Google could just license the tech to parties that would be interested in doing the hardware for the masses? Or maybe we just have to wait to see what Nvidia is brewing?
  - ori_b 8y ago
    > How these chips work is pretty reliant on Google's specific infrastructure. I find that hard to believe. Can you be more specific?
    - lawrenceyan 8y ago
      At Berkeley, we had a guest lecture where one of the major contributors in developing Google’s cloud infrastructure gave a talk on building scalability and robustness into their systems. The majority of the public facing products/services that you find on GCP actually started much earlier as internal tools for employees. There is a huge amount of work that is put into ensuring that everything works reliably, with significant underlying infrastructure needed to achieve it. The problems you face at the level of scale Google works in on a day to day basis turns even the most mundane tasks into extraordinarily difficult algorithmic challenges that have to be solved.
      - ori_b 8y ago
        ...er. So, by that reasoning, there is nothing that can exist outside of Google, because Google does things internally at scale.
        lawrenceyan 8y ago
        I think the point is that a lot of the products and services wouldn't work outside the Google Cloud platform without a significant amount of underlying infrastructure to support it, such that it would be very difficult to just provide some ready to go package for people to buy and implement themselves.
        ori_b 8y ago
        Which is largely bullshit. There's complexity in managing failures at scale -- but strip away the scale, and you strip away the complexity in managing failures. You just get normal, bog standard server-grade hardware, with the usual downtime.
  - psds2 8y ago
    Agreed. TPU will come on premise when Google launches a similar offering to AWS Outposts and ships you an appliance.
- thesandlord 8y ago
  Probably not what you are looking for, but: https://cloud.google.com/edge-tpu/ https://cloud.google.com/edge-tpu/ If you look at an image of a Cloud TPU board [0], I'm not sure how it would even work with off the shelf servers. It's 100% custom for a Google Datacenter. Nvidia also has the full supply chain set up, as well as things like driver support and all the things required to sell this kind of complex physical hardware. Google also has a program to give research scientists free access to TPUs: https://www.tensorflow.org/tfrc/ https://www.tensorflow.org/tfrc/ [0] https://storage.googleapis.com/gweb-uniblog-publish-prod/images/tpu-V2-hero.max-1000x1000.png https://storage.googleapis.com/gweb-uniblog-publish-prod/ima... (I work on GCP, but not on ML and TPU stuff)
  - deleted 8y ago
    [deleted]
KenoFischer 8y ago
Julia runs on these ;). Well, it will once multicore access for non-TF frameworks is public.
- zak 8y ago
  We're on it! = D
riku_iki 8y ago
Curious what will be the answer from Nvidia.
- eanzenberg 8y ago
  Their existing cards?
- ianhowson 8y ago
  > the largest Cloud TPU Pod configuration tested (256 chips) delivers a 200X speedup over an individual V100 GPU So a Cloud TPU chip is slower than a V100 chip? Seems like that's the answer.
  - KenoFischer 8y ago
    Well, you lose some performance to distributed memory overhead of course, plus the TPU measure for that number is a previous generation TPU. The new ones are a decent bit faster. Plus you get four chips per "Cloud TPU". Putting all that together, TPUs are certainly a very competitive offering.
  - wskinner 8y ago
    They seem roughly comprable. If you can get 85% scale out efficiency, based on the numbers in this post, 256 V100s would be at parity with 256 TPUs. In practice, you can get 90% scale out on Resnet with last year’s GPU and networking hardware. Network bandwidth is the bottleneck, and with better network hardware that could be improved. Maybe that’s why Google didn’t publish a benchmark for 256 V100s. I’m also curious what the is the performance of the network interconnects in the TPU pod. I couldn’t find it documented, though I didn’t look super hard. [1] https://github.com/uber/horovod/blob/master/docs/benchmarks.md https://github.com/uber/horovod/blob/master/docs/benchmarks....
- david-gpu 8y ago
  This is what NVidia says [0]. Notice how Google did not submit benchmark results for several tests. [0] https://blogs.nvidia.com/blog/2018/12/12/record-breaking-mlperf-ai-benchmarks/ https://blogs.nvidia.com/blog/2018/12/12/record-breaking-mlp...
ethikal 8y ago
Full ML Perf results can be found here: https://mlperf.org/results/ https://mlperf.org/results/
AlexCoventry 8y ago
I thought TPUs were mostly used for inference, because they're low- precision. Has that changed?
- gradys 8y ago
  Yep. That was the first generation that was only for inference. Subsequent generations can do inference and training.
etaioinshrdlu 8y ago
Does this require distributed training? My understanding and experience is that it is not always trivial to get linear training speedups with additional machines. It can be sometimes hard to get any speedup at all. The difference with gradient descent can be described simply as: * Single-machine training: take more steps * Distributed training: take fewer but more confident and accurate steps. This could allow you to take bigger steps (learning rate) as well, but there is a limit to this as well. They are not equivalent processes and it is an area of active research how to get an equivalent result with distributed training.
- antognini 8y ago
  I worked on a research project that set out to answer that question: https://arxiv.org/abs/1811.03600 https://arxiv.org/abs/1811.03600 We looked at a bunch of different model architectures and datasets and found that you can get speedups from larger batch sizes up until a certain point, but that point is different for different datasets and architectures. The range of good hyperparameters is also narrower for larger batch sizes, which makes tuning harder.
  - gdahl 8y ago
    To clarify, we found the range shrink if one trains for a fixed number of epochs and expand if one trains for a fixed number of steps. So tuning can be easier or harder depending on the budget.
fredguth 8y ago
TF only? Any reason why, besides Google tech lock in?

105 points by frankchn 8y ago

What a pity Google TPUs are locked in Google Cloud - many would like to use them on their own, in already owned servers. It kind of slows the ML progress, such restriction of access to the product.

zak 8y ago
You can also use Cloud TPUs for free in your browser via Colab. Several sample notebooks are available here: https://www.tensorflow.org/tfrc/ https://www.tensorflow.org/tfrc/
- api 8y ago
  I think they mean actually buying the chips. This model of putting revolutionary new hardware in the cloud only is kind of worrisome given the future implications.
  - eanzenberg 8y ago
    "Revolutionary"
kajecounterhack 8y ago
How these chips work is pretty reliant on Google's specific infrastructure and there's a stark difference between the work needed to offer a service and selling chips. It isn't a matter of bad will or trying to slow ML progress but a matter of what's practical.
- polskibus 8y ago
  Maybe Google could just license the tech to parties that would be interested in doing the hardware for the masses? Or maybe we just have to wait to see what Nvidia is brewing?
- ori_b 8y ago
  > How these chips work is pretty reliant on Google's specific infrastructure. I find that hard to believe. Can you be more specific?
  - lawrenceyan 8y ago
    At Berkeley, we had a guest lecture where one of the major contributors in developing Google’s cloud infrastructure gave a talk on building scalability and robustness into their systems. The majority of the public facing products/services that you find on GCP actually started much earlier as internal tools for employees. There is a huge amount of work that is put into ensuring that everything works reliably, with significant underlying infrastructure needed to achieve it. The problems you face at the level of scale Google works in on a day to day basis turns even the most mundane tasks into extraordinarily difficult algorithmic challenges that have to be solved.
    - ori_b 8y ago
      ...er. So, by that reasoning, there is nothing that can exist outside of Google, because Google does things internally at scale.
      - lawrenceyan 8y ago
        I think the point is that a lot of the products and services wouldn't work outside the Google Cloud platform without a significant amount of underlying infrastructure to support it, such that it would be very difficult to just provide some ready to go package for people to buy and implement themselves.
        ori_b 8y ago
        Which is largely bullshit. There's complexity in managing failures at scale -- but strip away the scale, and you strip away the complexity in managing failures. You just get normal, bog standard server-grade hardware, with the usual downtime.
- psds2 8y ago
  Agreed. TPU will come on premise when Google launches a similar offering to AWS Outposts and ships you an appliance.
thesandlord 8y ago
Probably not what you are looking for, but: https://cloud.google.com/edge-tpu/ https://cloud.google.com/edge-tpu/ If you look at an image of a Cloud TPU board [0], I'm not sure how it would even work with off the shelf servers. It's 100% custom for a Google Datacenter. Nvidia also has the full supply chain set up, as well as things like driver support and all the things required to sell this kind of complex physical hardware. Google also has a program to give research scientists free access to TPUs: https://www.tensorflow.org/tfrc/ https://www.tensorflow.org/tfrc/ [0] https://storage.googleapis.com/gweb-uniblog-publish-prod/images/tpu-V2-hero.max-1000x1000.png https://storage.googleapis.com/gweb-uniblog-publish-prod/ima... (I work on GCP, but not on ML and TPU stuff)
- deleted 8y ago
  [deleted]

KenoFischer 8y ago

Julia runs on these ;). Well, it will once multicore access for non-TF frameworks is public.

zak 8y ago
We're on it! = D

riku_iki 8y ago

Curious what will be the answer from Nvidia.

eanzenberg 8y ago
Their existing cards?
ianhowson 8y ago
> the largest Cloud TPU Pod configuration tested (256 chips) delivers a 200X speedup over an individual V100 GPU So a Cloud TPU chip is slower than a V100 chip? Seems like that's the answer.
- KenoFischer 8y ago
  Well, you lose some performance to distributed memory overhead of course, plus the TPU measure for that number is a previous generation TPU. The new ones are a decent bit faster. Plus you get four chips per "Cloud TPU". Putting all that together, TPUs are certainly a very competitive offering.
- wskinner 8y ago
  They seem roughly comprable. If you can get 85% scale out efficiency, based on the numbers in this post, 256 V100s would be at parity with 256 TPUs. In practice, you can get 90% scale out on Resnet with last year’s GPU and networking hardware. Network bandwidth is the bottleneck, and with better network hardware that could be improved. Maybe that’s why Google didn’t publish a benchmark for 256 V100s. I’m also curious what the is the performance of the network interconnects in the TPU pod. I couldn’t find it documented, though I didn’t look super hard. [1] https://github.com/uber/horovod/blob/master/docs/benchmarks.md https://github.com/uber/horovod/blob/master/docs/benchmarks....
david-gpu 8y ago
This is what NVidia says [0]. Notice how Google did not submit benchmark results for several tests. [0] https://blogs.nvidia.com/blog/2018/12/12/record-breaking-mlperf-ai-benchmarks/ https://blogs.nvidia.com/blog/2018/12/12/record-breaking-mlp...

ethikal 8y ago

Full ML Perf results can be found here: https://mlperf.org/results/ https://mlperf.org/results/

AlexCoventry 8y ago

I thought TPUs were mostly used for inference, because they're low- precision. Has that changed?

gradys 8y ago
Yep. That was the first generation that was only for inference. Subsequent generations can do inference and training.

etaioinshrdlu 8y ago

Does this require distributed training? My understanding and experience is that it is not always trivial to get linear training speedups with additional machines. It can be sometimes hard to get any speedup at all. The difference with gradient descent can be described simply as: * Single-machine training: take more steps * Distributed training: take fewer but more confident and accurate steps. This could allow you to take bigger steps (learning rate) as well, but there is a limit to this as well. They are not equivalent processes and it is an area of active research how to get an equivalent result with distributed training.

antognini 8y ago
I worked on a research project that set out to answer that question: https://arxiv.org/abs/1811.03600 https://arxiv.org/abs/1811.03600 We looked at a bunch of different model architectures and datasets and found that you can get speedups from larger batch sizes up until a certain point, but that point is different for different datasets and architectures. The range of good hyperparameters is also narrower for larger batch sizes, which makes tuning harder.
- gdahl 8y ago
  To clarify, we found the range shrink if one trains for a fixed number of epochs and expand if one trains for a fixed number of steps. So tuning can be easier or harder depending on the budget.

fredguth 8y ago

TF only? Any reason why, besides Google tech lock in?