7 ms·
Google TPU v5p beats Nvidia H100
- nabla9 3y agoGoogle is one generation behind? Google announces TPU v5p with specs competing against H100 when NVIDIA GH200 Grace Hopper is announced. >The new pods also have 95GB of high-bandwidth memory (HBM) GH200 has 282GB of HBM3e memory >These new pods provide a throughput of 4,800Gbps. That equals 600 GB/sec. GH200 has 10TB/sec of combined bandwidth HBM3e memory.
- miohtama 3y agoI guess performance per watt is also a major factor?
- bhewes 3y agoI am more interested in TCO of TPU v5p. Right now it seems only Nvidia is making money everyone else loses.
- Y_Y 3y agotable of contents?
- kirubakaran 3y agoI think they mean TCO: https://en.wikipedia.org/wiki/Total_cost_of_ownership https://en.wikipedia.org/wiki/Total_cost_of_ownership
- bhewes 3y agoThanks.
- deleted 3y ago[deleted]
- nabla9 3y agoI think Broadcomm designs and builds TPUs for Google. Their stock was up 100% in 2023 and their profit margin was 39.31%.
- bhewes 3y agoThey bought VMware for $61B in Nov of last year. Hard to think TPUs are driving much of this at Broadcom.
- surajrmal 3y agoGoogle's TPUs are entirely in house designed. Not sure where you heard broadcomm is involved.
- rreichman 3y agohttps://siliconangle.com/2023/09/21/google-reaffirms-broadcom-chip-partnership-report-claims-may-end-tpu-deal/ https://siliconangle.com/2023/09/21/google-reaffirms-broadco...
- windexh8er 3y agoGoogle doesn't build their TPUs. Broadcom does.
- lordswork 3y agoThe article mentions TPU v5p is 2.1 x perf/TCO of H100.
- bhewes 3y agoActual dollars would be nice, since that is what the end user understands. Otherwise what are they talking about 1 million unit orders? But nice catch I didn't see it.
- jsnell 3y agoI don't think it does? > Google's v5p TPUs are up to 2.8 times faster at training large language models than TPU v4, and offer 2.1-times value-for-money. That's a comparison of TPUv5p to TPUv4, not to H100.
- wmf 3y agoH100, H200, and GH200 are the same generation although the next generation B100 is coming this year.
- nabla9 3y agoThese comparisons are "Google Pod" vs. "Nvidia Supership" not single chips. Pod/Superships are collections GPU/TPU's, memory and high speed interconnect. One Grace Hopper has: H100 chip, Grace CPU with 72 cores, 282GB of HBM3e memory and 480 GB LPDDR5X for the CPU.
- wmf 3y agoYour numbers are still off. A GH200 pod has 144 terabytes of RAM; it isn't even measured in gigabytes. It looks like TPUv5p may have 851 terabytes which would be a generation ahead but you didn't show those numbers.
- deleted 3y ago[deleted]
- versteegen 3y agonabla9 said "one Grace Hopper" not "one GH200 pod". Actually, DGX GH200 seem to come in different sizes, something that I can't find clearly stated on Nvidia's website, but what I do see are entirely inconsistent specs. I'm thinking they've changed it a few times. https://developer.nvidia.com/blog/announcing-nvidia-dgx-gh200-first-100-terabyte-gpu-memory-system/ https://developer.nvidia.com/blog/announcing-nvidia-dgx-gh20... describes the DGX GH200 as: -256 NVIDIA Grace Hopper Superchips (1 CPU + 1 GPU) -each Grace CPU has 480 GB LPDDR5 CPU memory -each H100 GPU has 96 GB of HBM3 -each GPU can access all of the CPU and GPU memory in the whole system (144TB), at 900 GBps At https://www.nvidia.com/en-us/data-center/dgx-gh200/ https://www.nvidia.com/en-us/data-center/dgx-gh200/ (linked to from the DGX page from the website) which has a Datasheet pdf they say the DGX GH200 has: -32 NVIDIA Grace Hopper Superchips (1 CPU + 1 GPU) -each Grace CPU has 72 "ARM Neoverse V2 Cores with SVE2 4X 128" -each GPU can access "19.5TB" shared memory -that's 624GB per superchip, which is weird. I expect it's actually 96GiB HBM3 + 512GiB LPDDR, a total of 19456GiB = 19.0TiB And other people have found completely different specs elsewhere on the website!
- modeless 3y agoA few notes: Nvidia always announces their hardware in advance of availability, while Google typically announces some time after they've started using it internally. Also TPU has been more cost-focused than Nvidia, since Google uses these chips internally to serve their web traffic while Nvidia is supply constrained and can name their price right now, plus they don't pay the electricity bills. There are also a few things wrong with the specs you quoted. 282 GB is split between two GPUs, it's 144 GB per GPU (not sure where the extra 6 GB went). The TPU pod throughput number you quoted is interconnect bandwidth which you compared to GH200's memory bandwidth. Those numbers are not comparable. I believe GH200 does beat TPU v5p per chip, however the differences are not anywhere near as large as your comparison suggested. And it's likely that TPU v5p is dramatically more cost effective but we don't have Google's internal numbers to prove that.
- nabla9 3y ago> 282 GB is split between two GPUs There is only one GPU in GH200 and it gets all 282 GB HBM3e memory. There is another 480 GB LPDDR5X memory for the another chip (Grace CPU, not GPU). GH200 interconnect bandwidth trough NVLink is 900 GB/s.
- modeless 3y agoI don't believe this is true. GH200's GPU has 144GB of HBM3e according to their datasheet. https://resources.nvidia.com/en-us-grace-cpu/grace-hopper-superchip https://resources.nvidia.com/en-us-grace-cpu/grace-hopper-su... The 282 GB figure comes from a "dual configuration" server with two CPU chips and two GPU chips: > the new platform will be available in a wide range of configurations. The dual configuration [...] comprises a single server with 144 Arm Neoverse cores, eight petaflops of AI performance and 282GB of the latest HBM3e memory technology. Note that one Grace Hopper CPU has 72 cores so this configuration clearly has two CPUs and two GPUs in one server, and the 282 GB HBM3e total is split between the two GPUs with 144 GB each (again, not sure where the extra 6 GB went, maybe disabled for yield issues?). https://nvidianews.nvidia.com/news/gh200-grace-hopper-superchip-with-hbm3e-memory https://nvidianews.nvidia.com/news/gh200-grace-hopper-superc...
- 3y ago
- tgtweak 3y agoDGX GH200 is next gen, but afaik not available to anyone aside from internal partners. All the quoted data here is for several of them together in a chassis. >Each NVIDIA Grace Hopper Superchip in NVIDIA DGX GH200 has 480 GB LPDDR5 CPU memory, at an eighth of the power per GB, compared with DDR5, and 96 GB of fast HBM3. (https://developer.nvidia.com/blog/announcing-nvidia-dgx-gh200-first-100-terabyte-gpu-memory-system/ https://developer.nvidia.com/blog/announcing-nvidia-dgx-gh20...) Looks like a single superchip has 96GB of hbm3, similar to a single v5p tpu. The H100's in the DGX H100 were 80GB per board.
- porphyra 3y agoIt is pretty exciting that various companies are making compute for deep learning. I was starting to get worried that the TPUs were lagging behind Nvidia. More diversity in hardware is always nice. Meanwhile I am still waiting for Tesla Dojo to be an actual thing...
- ShamelessC 3y agoTesla’s dojo is deployed already I thought? I don’t recall them ever having plans to make it publicly available.
- modeless 3y agoElon's comments on the latest earnings call, if you read between the lines, strongly imply that Dojo v1, while functioning, has been a disappointment. He seems to hold out hope for future versions but he is markedly less optimistic than for other programs he talked about, and considering his typical unjustifiably optimistic attitude toward future releases of AI-related products that seems to bode poorly for the project. I would not be surprised if it was cancelled or rebooted in the next year or two. I don't recall if they made comments about public availability for Dojo before, but given the difficulty of competing in this market and the huge benefits scale brings in hardware, it would seem foolish to limit the system to internal customers only.
- ShamelessC 3y ago> Elon's comments on the latest earnings call, if you read between the lines, strongly imply that Dojo v1, while functioning, has been a disappointment Maybe they shouldn’t have gone with an in-house design. > it would seem foolish to limit the system to internal customers only. My understanding from working with similar HPC systems is that public use cases are likely niche and many tasks don’t benefit from the sort of fast interconnect their setup gives them. If anything being quite public about their design was meant to allow other companies to copy their design and compete with nvidia on a united front perhaps. It’s not really as versatile as AWS, GCP are to profitably work as a compute cloud provider though. Perhaps it could be a provider for other companies who don’t want to own their hardware to do training runs on.
- stonogo 3y agoI don't see how we're this far in and so many competitors are still trying to ignore the massive market penetration of CUDA. I don't care how fast your chip is if I can only run 20% of the already tiny amount of software capable of leveraging this acceleration.
- collegeburner 3y agosome stuff is still CUDA-dependent (a lot of scientific code). other stuff is not (major ML frameworks and many math libraries). nobody said CUDA was obsolete, just that there are great options for many use cases that no longer rely on it.
- vessenes 3y agoI think this was a more accurate assessment a couple of years ago. AMD historically under invested in their software stack, as is well documented. But, they are catching up. The prevalence of pytorch makes it easier for hardware vendors to target a single set of libraries for implementation and get broad distribution. Even Apple has made major progress getting mps support in more broadly, first directly into pytorch, and they are now charting their own course with MPX with some early interesting successes. Zuck recently used the word "H100 equivalent" in compute, and my memory is he indicated roughly 35% of that compute was not from NVIDIA -- that will all be AMD. There's still work to do -- lots of repositories still contain `if device=="cuda"` type language, but my own experience is that manging code around to use apple gpus has gotten vastly easier this year, and I see more and more AMD GPU owners floating around github issues with resolvable problems. A year ago they were barely present. All that said - people aren't ignoring it - and entrepreneurs and cloud monopolies are putting real resources into opening things up. I think the playing field will continue to level / get back to competitive over the next two to three years.
- ShamelessC 3y agoThe main issue is until you solve 99% of all the problems, you’re still introducing substantial friction to every potential user who may have had to go from zero coding knowledge needed to very specialized coding knowledge needed. And while many problems are trivial to fix at eg the PyTorch layer, lots of stuff like flash attention, DeepSpeed, etc. are coded directly in CUDA kernels.
- sadhorse 3y agoMore compute can't save you from dumb methods. Memorizing the whole of human text while being fooled by a prompt crafted by a 14 year old doesn't mean its intelligent.
- crakenzak 3y agoPlenty of intelligent and educated humans have been fooled by "prompts" aka lies/social engineering by others many years their younger.
- ilaksh 3y agoCan we access TPUv4 and/or v5 in Google Cloud? What frameworks support them?
- sxp 3y agoIs there any good way to program TPUs for non-ML work? The only official way appears to involve encoding the compute operation into a TF graph and loading that. Is there a better way to directly use the hardware?
- counters 3y agoJust write your code in JAX and run as usual?
- pjmlp 3y agoLooking forward to see Nanite implementation in JAX.
- lordswork 3y agoAFAIK, XLA is the only compiler that can target TPUs, so anything written for TPUs would need to lower to HLO. That said, why would you want to to run non-ML workloads on such an ML-specialized chip?
- sxp 3y agoI was interested in seeing if raytracing or similar embarrassingly parallel workloads could be ported to a TPU.
- renewiltord 3y agoCan only use on Google Cloud, no? Nvidia is the only self-hostable product available. But Google can build good framework integration. Will wait and see.
- wmf 3y agoGaudi, MI300, Cerebras...
- renewiltord 3y agoHave you bought these? Could not find Gaudi or Cerebras device available anywhere. And MI300 risky since software support is poor. But if you have managed to make it work, perhaps there's potential. What stack are you using and what workload?
- wmf 3y agoI don't work in ML but I know these are "call for pricing" products.
- renewiltord 3y agoI don't know of anyone who uses these products. I would probably avoid.
- wmf 3y agoJust don't complain about monopoly if you're not willing to try anything.
- renewiltord 3y agoWould like to try others, but investment too big. Would have to build entire software ecosystem. I think most people like this. Even geohot suffered trying to do this.
- jchonphoenix 3y agoNone of this matters if they can't get the hardware stack to work correctly. The media keeps missing the real lock in Nvidia has: CUDA. It's not the hardware. It's the ability for someone to use it painlessly.
- lordswork 3y agoWhat's painful about using TPUs?
- refulgentis 3y agoin a sentence: google ai stuff is the vendor lockin of 2024 apple with the ecosystem value of 1994 apple. c.f. https://news.ycombinator.com/item?id=39149854 https://news.ycombinator.com/item?id=39149854
- postalrat 3y agoSo you trade one vendor lockin for another. Nothing lost.
- moffkalast 3y agoAnd Nvidia does actually sell their hardware. Nobody will ever get their hands on one of these outside Google Cloud. It might as well not exist.
- buildbot 3y agoWell, sometimes they fall of the back of trucks I guess: https://www.ebay.com/itm/134540730431 https://www.ebay.com/itm/134540730431 Archive link: https://archive.ph/7dPFo https://archive.ph/7dPFo
- sidibe 3y agoDoesn't really matter. Google's infra is all the client you need to continue pouring tens of billions into a project like this, bonus if others start using it more in the cloud, but they have so much use for accelerators across their own projects they aren't going to stop
- nisten 3y agoMeh, the comparison is somewhat pointless when it doesn't account for the slowdown that the vast majority of pytorch codebases experience on TPU's vs using JAX and it's accelerations specific to TPUs, and vice versa. https://arxiv.org/pdf/2309.07181.pdf https://arxiv.org/pdf/2309.07181.pdf Meaning the TPU v5p is likely slower than H100 for most ML workloads that depend on pytorch.
- tgtweak 3y agoTwo things matter at cloud-scale: Compatability - does using a tpu require reworking significant parts of your software stack and application stack? If so, that sucks for most companies who's researchers are used to using nvidia libraries, tooling and hardware to get their models running, and reworking the entire bottom end of that to work on an esoteric platform needs to equate to a huge cost savings at scale to be worth it. Cost per tensor flop delivered - likely very low if google has optimized the silicon, memory, voltage, power and temperature envelope, networking, boards and chassis for the server running it, as well as optimizing for optimal process node efficiency/cost. They're probably not on bleeding edge tsmc process, but instead optimizing for total deployed running cost per pflop over 2-3 years. It's also now public (as of November) that microsoft/azure have been working for many years on their own ai chip, dubbed "Maia" [1], and appliance with the obvious goal of taking some of that nvidia margin in-house (and with openai/bing/copilot being a massive consumer of capacity). I think this will become even more commonplace with cloud vendors - even medium size ones - than it already is. The knowledge and complexity barrier to designing a tile processor unit seems pretty low and it looks like most of the hard stuff is in the drivers and software - something cloud providers designing and integrating internally can bypass and control to a great degree. It's also very hard to benchmark side by side on these since I'm sure cuda/nvidia hardware can do compute that a TPU cannot. AMD's machine learning accelerators look good on paper too, as do the tensor processors in the apple silicon, but on real world applications and use cases they don't often measure up save for a few optimal workflows. [1] https://www.geekwire.com/2023/microsoft-unveils-custom-ai-chip-with-help-from-openai-playing-catch-up-with-amazon/ https://www.geekwire.com/2023/microsoft-unveils-custom-ai-ch...
- geor9e 3y agoHasn't this always been the case? I remember they released TPUs at IO 2018, and they were nice if you build FP16/FP32 models in Google Cloud TensorFlow/CoLab and never ever port them to anything else. Meanwhile the cool new open source stuff coming out every week usually requires a GPU and is rarely compatible with TPU without major changes. If you wait a few months a TPU compatible copycat appears on the Keras demos page but by then you've lost interest.
- 7e 3y agoThere are no benchmarks in this s*t article, just SWAGs. If this chip doesn't support FP8, there's no way it's going to beat the H100 on workloads that matter.
- 7e 3y agoGooglers, stop downvoting me and get back to work!
- wslh 3y agoI understand that OpenAI used Azure for processing and not Google Cloud right? Using Google Cloud would mean, assuming Google as a bad actor, that they can get the model if OpenAI trains their models with it.
- surajrmal 3y agoNo one would Google cloud if they were worried about IP theft. This also seems very much irrelevant to the posted article.
- wslh 3y agoMy intention of writing that comment was different, it was my mistake, so I correct and rephrase it. I think from a cybersecurity perspective. Imagine a future where there are many players in the cloud training business. These players have the chance to retrieve de LLM model and since this could be a high stake business that is dangerous. I underastand that currently cloud players can observe everything that happens in their clouds including secret information and this is not happenning but it could not be "Google" by themselves but an inside job.
- surajrmal 3y agoI'm not sure this will ever be a solved problem. If it's in your threat model to worry about such things, you necessarily need to have your own servers, in a secured physical location you control. No amount of attestation and auditing will allow you to overcome that sort of threat. For the most part, I believe audit logs and severely restricted permissions provided to employees greatly disincentive the chances of an inside job occurring successfully.
- qeternity 3y agoA lot of comments in this thread are parroting the common CUDA moat but that really only applies for training and R&D. The majority of spend is on inference and the world is standardizing around a handful of common architectures that have or are being implemented performantly in non CUDA stacks
- nl 3y agoTPUs are mostly used outside of Google for training. There are better inference options.
- krasin 3y ago+1. My favorite essay on the topic: https://petewarden.com/2023/09/10/why-nvidias-ai-supremacy-is-only-temporary/ https://petewarden.com/2023/09/10/why-nvidias-ai-supremacy-i...
- gitfan86 3y agoI sold my GOOG shares from 2005 to buy NVDA last year and definitely agree with the article. The thing is that Wall St doesn't understand any of the technical details and will just see huge profit growth from NVDA and drive the price crazy high.
- Mistletoe 3y agoI don’t understand; hasn’t that already happened and you need to get out now?
- brucethemoose2 3y agoThis needs to be taken with a massive grain of salt, as LLM training performance hugely depends on the framework used. And by "performance" I mean the quality of the end result, the training speed, and what features device is physically capable of handling. While kinda discrete aspects in other fields, all of these are highly correlated in ML training. One specific example I am thinking of is support for efficient long context training. If your stack doesn't, for instance, support flash attention (or roll its own custom implementation), then good luck with that. Another I am thinking of is quantization. Sometimes you want to train with a specific quantization scheme, and sometimes doing that is just a performance compromise. And vendors really love to fudge their numbers with the fastest, most primitive quantization on the device.
- bogwog 3y agoAdditionally, Google has not been very honest recently with regards to their AI tech.
- habibur 3y agoI was seeking power consumption and price comparison with it, as otherwise only faster doesn't mean much.
- RantyDave 3y agoBut you can only buy it as a cloud service, yes? So presumably: * Whether something 'beats' something else is actually a question of price/performance and... * Whether or not the vendor in question is famous for dropping people in the shit.
- killingtime74 3y agoCompetitors will also never use it
- dev1ycan 3y ago[flagged]
- singularity2001 3y agoGiven the excessive run on GPUs its staggering that Google doesn't sell their TPUs (with some ludicrous exceptions).
- emadm 3y agoTraining on 1k+ v5p and h100 this isn’t true. We get 400 tflops per chip on v5p int8 and 650 fp8 for h100s H100s are less stable training though. V5ps are about the same as intel gaudi2s and both are cheaper than h100s price/perf Gaudi3s are 3-4x faster…
- wslh 3y agoDo you know how Intel Gaudi [1] could be purchased. I see a "Contact us" in their place but any inside or public information about prices and the budget required for bulk purchases would be great. Also, how Gaudi3 compares to Google TPUs? Thanks! I know that Gaudi3 is not readily available yet. [1] https://habana.ai/products/gaudi/ https://habana.ai/products/gaudi/
- maxwell22 3y agoIs there enough volume of this chip for it to mean anything to anyone? If there are only a few that no one can’t buy, and they are only better than nvidia at 1 out of 10’000 workloads, can people even use their existence to try to get lower H100 prices?
- wslh 3y agoI am currently analyzing the feasibility and economics of building a cloud service for AI training. This involves "day to day" analysis changes since all the "parts" are moving.
- vgblr 3y agohttps://www.supermicro.com/en/products/system/AI/4U/SYS-420GH-TNGR https://www.supermicro.com/en/products/system/AI/4U/SYS-420G...