6 ms·
Cloud TPU VMs are generally available
- Reubend 4y agoDo these TPU VMs have preemptible/spot pricing? I'm just a hobbyist, but it always seemed annoying before that one would have to orchestrate VMs to spin up/down alongside spot instances of TPUs manually.
- zak 4y agoYes, there are a couple of ways to use Cloud TPUs at lower priority and lower cost. If you are a hobbyist, I highly recommend trying out Cloud TPUs for free via the TPU Research Cloud: https://sites.research.google/trc/ https://sites.research.google/trc/ If you are flexible, you (or your scripts) can access a lot of compute power at odd hours!
- Filligree 4y agoThat’s for researchers, however. For hobbyists, I guess we just have colab?
- zak 4y agoI started the TRC program alongside the Cloud TPU program to make interesting amounts of ML compute available to a broad group of creative people, not only to academic researchers. The TRC program welcomes hobbyists, artists, students, independent learners, technical writers, and a variety of others. We love it when the TPU Research Cloud enables people to do something that wouldn't have been possible otherwise. I definitely recommend applying to the TRC program - please feel free to say directly that you are a hobbyist. The sign-up form is short, and it's likely that you can access a lot of compute if you are flexible and persistent.
- usmannk 4y agoTPU VMs are an incredible, incredible product. If you were put off by the previous TPU Node architecture, you should definitely come back and give this a shot. The dev/debug loop is just so much nicer now that the TPU is local to your VM. That said, the one thing I'm missing on these is orchestration. GKE supports TPU Nodes but it does not support TPU VMs. It seems to me like this is a must-have feature so I'm sure it's on the roadmap. If anyone from the team is here, do you have updates on this? And if anyone reading this has orchestration techniques they use and recommend for TPU VMs, please let me know what they are!
- zak 4y agoThanks for the very kind feedback! We've wanted to provide TPU VMs since the beginning of the Cloud TPU program, and I'm delighted that you're enjoying them. Many people across Google contributed to this launch. We're definitely working on GKE integration, and we intend to provide more support for orchestration over time since individual Cloud TPU workloads are getting bigger and bigger. We'd love to hear any more detailed feedback you might have.
- bigcat12345678 4y agoHi Zak, great achievement. Xoogler, had connection with people working on TPU. AFAIK, TPU support on GCP has been started in 2018? Or earlier? It's been 4 years to GA. Rumors have it that the project has been incredibly difficult to get out due to Googler's poor cross-org collaboration support. It has been a sad story, another one in the list of things that Google could have been putting more resources and made available sooner to the public.
- zak 4y agoI started pitching the Cloud TPU program in 2016. Many, many people have contributed since then to build the products that are available today. Google is a large and complicated place, but we're getting closer to providing the magical interactive supercomputing experience we've wanted for a long time. The deep learning landscape is evolving very rapidly, and there is a lot of interest in scaling up further, so the next few years will be exciting.
- carbocation 4y agoDoes this solve the data-loading issues raised by sillysaurus in this prior thread? https://news.ycombinator.com/item?id=24721229 https://news.ycombinator.com/item?id=24721229
- zak 4y agoYes, TPU VMs dramatically improve the Cloud TPU user experience. You now have direct access to the VM on each TPU host whether you are using JAX, PyTorch, or TensorFlow, which provides a lot more flexibility and control and can often improve performance.
- zwaps 4y agoI struggle to understand precisely what you mean by user experience and ‘often improved performance‘. Previously, there was no actual support for crucial features of the TPU related to data loading when using PyTorch, say. In turn, using a TPU over a GPU on that setup was frequently not worth it due to that exact issue. Your answer suggests it might be different now: are TF, Jax and PyTorch now on par in all stages?
- zak 4y agoIn the previous Cloud TPU architecture, PyTorch and JAX users had to create a separate CPU VM for every remote TPU host and arrange for these CPU hosts to communicate indirectly with the TPU hosts via gRPC. This was cumbersome and made debugging difficult. With TPU VMs, none of this is necessary. You can SSH directly into each TPU host machine and install arbitrary software on a VM there to handle data loading and other tasks with much greater flexibility. The blog post provides an example of training cost improvement using PyTorch / XLA on TPU VMs in the "Local execution of input pipeline" section. Hopefully we will be able to provide more tutorials on using PyTorch / XLA with TPU VMs soon. With TPU VMs, workloads that require lots of CPU-TPU communication can now do that communication locally instead of going over the network, which can improve performance.
- ultrons 4y ago
- mahastore 4y agoFirst thing you should explain is what the hell a TPU is/
- zak 4y agoHere's a quick overview of TPUs from last year's Google I/O keynote: https://www.youtube.com/watch?v=XFFrahd05OM&t=1565s https://www.youtube.com/watch?v=XFFrahd05OM&t=1565s
- ChadNauseam 4y agoThe section on error-correcting qubits right after the bit you timestamped is really interesting too, I wonder what the status of that is
- bpodgursky 4y agoIt's an inter-dimensional temporal portal which tunnels into the future and borrows incomprehensible alien technology to execute deep learning algorithms on GCP client datasets.
- armchairhacker 4y agohttps://en.wikipedia.org/wiki/Tensor_Processing_Unit https://en.wikipedia.org/wiki/Tensor_Processing_Unit
- AnonMO 4y agoGoogle cloud is part of a company called alphabet which also own a little tool called google search you should try it. it’s very handy. Also if you don’t know what a tpu is the product probably is targeted at you
- zak 4y agoFounder of the Cloud TPU program here. If you'd like to experiment with TPU VMs for free and are willing to share your work with the world somehow (e.g. via publications or open-source projects), you can apply to participate in the TPU Research Cloud (TRC) program here: https://sites.research.google/trc/ https://sites.research.google/trc/
- BooneJS 4y agoCongrats Zak!
- zak 4y agoThanks, and congratulations to many others across many teams who have supported the Cloud TPU program over the years!
- teleforce 4y agoThanks Zak, already applied. Just wondering does TPU VM support Vectorflow? https://github.com/Netflix/vectorflow https://github.com/Netflix/vectorflow
- zak 4y agoNo, Vectorflow is not supported out of the box, and I'm not sure the workloads it targets are the right fit for Cloud TPU hardware. However, be sure to check out the "Ranking and recommendation" section of the linked blog post above - Cloud TPUs are able to accelerate the ML models with very large embeddings that are increasingly common in state-of-the-art ranking and recommendation systems.
- baybal2 4y ago
- jasonphang 4y agoHi Zak, this question is a little out of your scope, but perhaps you may know the answer: Do you know if/when Colab TPU runtimes are likely to be updated to support newer JAX functionality like pjit? (Or put another way: are Colab and Cloud TPU runtimes expected to be in-sync at all?) I'd written some model code that worked great on a TPU-VM and that I was excited to share on Colabs (which is likely far more accessible), but then I found out that Colabs simply don't support pjit (https://github.com/google/jax/issues/8300 https://github.com/google/jax/issues/8300). Other than that, really like TRC!
- ramchip 4y agoFor those like me who had no idea what this means: > Tensor Processing Units (TPUs) are Google's custom-developed application-specific integrated circuits (ASICs) used to accelerate machine learning workloads.
- barbarbar 4y agoI wish I had seen this comment earlier. Maybe here tags/labels could be helpful.
- hoosieree 4y agoIs this[1] the best resource for understanding TPU (or TPU-like) architectures in general? I would love to see a TPU implementation in the style of the Nand2Tetris[2] CPU, reduced to its bare essentials for learning purposes. Are systolic array processors the best place to start for this kind of info? Thanks in advance if anyone answers. :) [1]: https://drive.google.com/file/d/0Bx4hafXDDq2EMzRNcy1vSUxtcEk/view?resourcekey=0-ulCsvFTNky29UIPJ3pHyCw https://drive.google.com/file/d/0Bx4hafXDDq2EMzRNcy1vSUxtcEk... [2]: https://www.nand2tetris.org/course https://www.nand2tetris.org/course
- joelthelion 4y agoAny benchmarks comparing this to regular Nvidia gpus?
- zak 4y agoThe MLPerf 1.0 results provided an apples-to-apples comparison of large-scale TPU and GPU systems across several ML workloads: https://cloud.google.com/blog/products/ai-machine-learning/google-wins-mlperf-benchmarks-with-tpu-v4 https://cloud.google.com/blog/products/ai-machine-learning/g... In MLPerf 1.1, we showcased model training at larger scale: https://cloud.google.com/blog/topics/tpus/google-showcases-cloud-tpu-v4-pods-for-large-model-training https://cloud.google.com/blog/topics/tpus/google-showcases-c... The deep learning workloads that people find most interesting and the underlying hardware and software systems are all changing very rapidly. In addition to following MLPerf, we generally recommend that people run rigorous performance and cost comparisons on the actual workloads that they care about accelerating.
- formerly_proven 4y agoI'd be more interested in the edge TPU / coral stuff for local inference, but it seems perpetually unavailable.
- ramoz 4y agoAvailable in new regions other than us-central?
- moritonal 4y agoMostly a jest, but now you're releasing these to the public, does that mean Google's moved onto something better?
- Havoc 4y agoIt mentions tensorflow specifically. Is there a penalty for using PyTorch instead?
- zak 4y agoNo, Cloud TPUs support JAX, PyTorch, and TensorFlow, and the new TPU VM architecture provides enough low-level access that users could add support for additional frameworks themselves if they are willing to put in substantial effort.
- echelon 4y agoDo existing PyTorch models work out of the box with cloud TPU, or does it require some tweaking? Are there cost savings over traditional GPU workloads? When will GKE support TPU? And will there be preemptable instances?
- johndough 4y agoLast time I checked, you had to change a few lines to use torch_xla. If there are cost savings depends on your workload. If you are training huge models, it might be cost effective, but honestly most tasks I have seen would have been fine on a CPU. The pricing page lists preemptible instances: https://cloud.google.com/tpu/pricing#single-device-pricing https://cloud.google.com/tpu/pricing#single-device-pricing
- _9rq6 4y agoare google cloud TPUs available in mumbai region?
- rg111 4y agoCloud TPUs are the ultimate form of vendor lock-in. I vehemently oppose using it for anything truly critical.
- zak 4y agoWe try to make it easy to switch back and forth between Cloud TPUs and other hardware platforms using JAX, PyTorch, and TensorFlow. This is a difficult technical challenge, but the XLA compiler helps a lot, and switching is easier now than it has ever been.