8 ms·
TPU v4 provides exaFLOPS-scale ML with efficiency gains
- ThorsBane 4y agoThis is really cool!
- reaperman 4y agoI am very impressed with what Google has done for the state of machine learning infrastructure. I'm looking forward to future models based on OpenXLA which can run between Nvidia, Apple Silicon, and google's TPU's. My main limiter to using TPU more often is model compatibility. The TPU hardware is clearly the very best, just not always cost-effective for those of us who are starved of available engineering hours. OpenXLA may fix this if it lives up to its promise. That said, it's also incredible how fast things move in this space: > Midjourney, one of the leading text-to-image AI startups, have been using Cloud TPU v4 to train their state-of-the-art model, coincidentally also called “version four”. Midjourney is already on v5 as of the date of publication of this press release.
- nestorD 4y ago> model compatibility JAX will compile your model with XLA today, letting you run it on Nvidia, Apple Silicon (CPU for now), google's TPU, etc. Overall, XLA really shines on TPUs, is doing very well on GPUs and sadly lags behind on CPUs (reaching about single core performance). As a bonus, JAX is basically Numpy++: you can use it to build scientific/numerical software and run it easily on GPU and TPU!
- gallabytes 4y agoyeah we trained v5 on TPUs and continue to train on them.
- xiphias2 4y ago,,Midjourney, one of the leading text-to-image AI startups, have been using Cloud TPU v4 to train their state-of-the-art model, coincidentally also called “version four'' This sounds quite bad in a press release when Midjourney is at v5. Why did they move away?
- sebzim4500 4y agoSounds like they are just out of date, it is possible MJv5 was also trained on TPUs.
- gallabytes 4y agoWe didn't, v5 was trained on TPUs too
- obblekk 4y agoThis is very impressive technology and engineering. However, I remain a bit skeptical of the business case for TPUs for 3 core reasons: 1) 100000x lower unit production volume than GPUs means higher unit costs 2) Slow iteration cycle - these TPUv4 were launched in 2020. Maybe Google publishes one gen behind, but that would still be a 2-3 year iteration cycle from v3 to v4. 3) Constant multiple advantage over GPUs - maybe 5-10x compute advantage over off the shelf GPU, and that number isn't increasing with each generation. It's cool to get that 5-10x performance over GPUs, but that's 4.5yrs of Moore's Law, and might already be offset today due to unit cost advantages. If the TPU architecture did something to allow fundamentally faster transistor density scaling, it's advantage over GPUs would increase each year and become unbeatable. But based on the TPUv3 to TPUv4 perf improvement over 3 years, it doesn't seem so. Apple's competing approach seems a bit more promising from a business perspective. The M1 unifies memory reducing the time commitment required to move data and switch between CPU and GPU processing. This allows advances in GPUs to continue scaling independently, while decreasing the user experience cost of using GPUs. Apple's version also seems to scale from 8GB RAM to 128GB meaning the same fundamental process can be used at high volume, achieving a low unit cost. Are there other interesting hardware for ML approaches out there?
- kccqzy 4y ago> Are there other interesting hardware for ML approaches out there? Google also has Coral, which is a non-cloud mobile-focused TPU that you can buy and plug in (USB or PCIe). https://coral.ai/products/ https://coral.ai/products/ Naturally, this is orders of magnitude less powerful than the kind of TPU being discussed here but I just like the idea of local compute.
- 1MachineElf 4y agoJust received my mSATA Coral TPU in the mail. I'd been waiting 11 months for it after backordering on Digikey. Perhaps this speaks to the parent comment's concerns over unit volume and iteration cycle? Hopefully that will improve in the future and modules like these will become widespread.
- cubefox 4y ago
- tinco 4y agoThey're so non confrontational. Their performance comparisons are against "CPU". Just come out and say it, even if it's not apples to apples. If the 3D-torus interconnect is so much better, just say how it compares to NVidia's latest and greatest. It's cool that midjourney committed to building on TPU, but I have a hard time betting my company on a technology that's so guarded that they won't even post a benchmark against their main competitor.
- jeffbee 4y agoThe paper compares it to A100. https://arxiv.org/pdf/2304.01433.pdf https://arxiv.org/pdf/2304.01433.pdf
- ttt3ts 4y agoNvidia's last gen...
- jsnell 4y agoThat's addressed in section 7.4 of the paper: > The Google tradition is to write retrospective papers after systems are deployed and running production apps. Both TPU v4s and A100s deployed in 2020 and both use 7nm technology. The newer, 700W H100 was not available at AWS, Azure, or Google Cloud in 2022. The appropriate H100 match would be a successor to TPU v4 deployed in a similar time frame and technology (e.g., in 2023 and 4 nm).
- deleted 4y ago[deleted]
- revelio 4y agoThat's not really addressing the point though, is it. That tradition at Google evolved when they had technology that was well ahead of their competitors, which they currently are not and therefore this paper is more like a marketing document than a recruiting resource. It's inevitable that people will want comparisons before buying TPU access and giving them a mapreduce style paper that comes out years after the technology is mature isn't going to satisfy them.
- KeplerBoy 4y agoI refuse to care about them until they sell them on PCIe Cards. The lock-in is bad enough when dealing with niche hardware on-prem, i certainly won't deal with niche hardware in the cloud.
- synergy20 4y agocan't agree you, until then, I have no interests in its TPUx
- photonbeam 4y agoThis is part of why I’m hesitant to use ARM on AWS, its not easy enough to just buy a representative equivalent for onprem
- my123 4y agoIt is easy especially now. (With Altra availability ramping up) https://www.ipi.wiki/products/com-hpc-ampere-altra?variant=43194533773474 https://www.ipi.wiki/products/com-hpc-ampere-altra?variant=4...
- amrb 4y agohttps://tenstorrent.com/ https://tenstorrent.com/
- deleted 4y ago[deleted]
- amrb 4y ago[flagged]
- dekhn 4y agothe product merely needs to ensure that the ads division trains their serving models on it to obtain longevity
- deleted 4y ago[deleted]