4 ms·
Google is one generation behind? Google announces TPU v5p with specs competing against H100 when NVIDIA GH200 Grace Hopper is announced. >The new pods also h
by nabla9 3y ago
Google is one generation behind?
Google announces TPU v5p with specs competing against H100 when NVIDIA GH200 Grace Hopper is announced.
>The new pods also have 95GB of high-bandwidth memory (HBM)
GH200 has 282GB of HBM3e memory
>These new pods provide a throughput of 4,800Gbps.
That equals 600 GB/sec. GH200 has 10TB/sec of combined bandwidth HBM3e memory.
- miohtama 3y agoI guess performance per watt is also a major factor?
- bhewes 3y agoI am more interested in TCO of TPU v5p. Right now it seems only Nvidia is making money everyone else loses.
- Y_Y 3y agotable of contents?
- kirubakaran 3y agoI think they mean TCO: https://en.wikipedia.org/wiki/Total_cost_of_ownership https://en.wikipedia.org/wiki/Total_cost_of_ownership
- bhewes 3y agoThanks.
- deleted 3y ago[deleted]
- nabla9 3y agoI think Broadcomm designs and builds TPUs for Google. Their stock was up 100% in 2023 and their profit margin was 39.31%.
- bhewes 3y agoThey bought VMware for $61B in Nov of last year. Hard to think TPUs are driving much of this at Broadcom.
- surajrmal 3y agoGoogle's TPUs are entirely in house designed. Not sure where you heard broadcomm is involved.
- rreichman 3y agohttps://siliconangle.com/2023/09/21/google-reaffirms-broadcom-chip-partnership-report-claims-may-end-tpu-deal/ https://siliconangle.com/2023/09/21/google-reaffirms-broadco...
- windexh8er 3y agoGoogle doesn't build their TPUs. Broadcom does.
- lordswork 3y agoThe article mentions TPU v5p is 2.1 x perf/TCO of H100.
- bhewes 3y agoActual dollars would be nice, since that is what the end user understands. Otherwise what are they talking about 1 million unit orders? But nice catch I didn't see it.
- jsnell 3y agoI don't think it does? > Google's v5p TPUs are up to 2.8 times faster at training large language models than TPU v4, and offer 2.1-times value-for-money. That's a comparison of TPUv5p to TPUv4, not to H100.
- wmf 3y agoH100, H200, and GH200 are the same generation although the next generation B100 is coming this year.
- nabla9 3y agoThese comparisons are "Google Pod" vs. "Nvidia Supership" not single chips. Pod/Superships are collections GPU/TPU's, memory and high speed interconnect. One Grace Hopper has: H100 chip, Grace CPU with 72 cores, 282GB of HBM3e memory and 480 GB LPDDR5X for the CPU.
- wmf 3y agoYour numbers are still off. A GH200 pod has 144 terabytes of RAM; it isn't even measured in gigabytes. It looks like TPUv5p may have 851 terabytes which would be a generation ahead but you didn't show those numbers.
- deleted 3y ago[deleted]
- versteegen 3y agonabla9 said "one Grace Hopper" not "one GH200 pod". Actually, DGX GH200 seem to come in different sizes, something that I can't find clearly stated on Nvidia's website, but what I do see are entirely inconsistent specs. I'm thinking they've changed it a few times. https://developer.nvidia.com/blog/announcing-nvidia-dgx-gh200-first-100-terabyte-gpu-memory-system/ https://developer.nvidia.com/blog/announcing-nvidia-dgx-gh20... describes the DGX GH200 as: -256 NVIDIA Grace Hopper Superchips (1 CPU + 1 GPU) -each Grace CPU has 480 GB LPDDR5 CPU memory -each H100 GPU has 96 GB of HBM3 -each GPU can access all of the CPU and GPU memory in the whole system (144TB), at 900 GBps At https://www.nvidia.com/en-us/data-center/dgx-gh200/ https://www.nvidia.com/en-us/data-center/dgx-gh200/ (linked to from the DGX page from the website) which has a Datasheet pdf they say the DGX GH200 has: -32 NVIDIA Grace Hopper Superchips (1 CPU + 1 GPU) -each Grace CPU has 72 "ARM Neoverse V2 Cores with SVE2 4X 128" -each GPU can access "19.5TB" shared memory -that's 624GB per superchip, which is weird. I expect it's actually 96GiB HBM3 + 512GiB LPDDR, a total of 19456GiB = 19.0TiB And other people have found completely different specs elsewhere on the website!
- modeless 3y agoA few notes: Nvidia always announces their hardware in advance of availability, while Google typically announces some time after they've started using it internally. Also TPU has been more cost-focused than Nvidia, since Google uses these chips internally to serve their web traffic while Nvidia is supply constrained and can name their price right now, plus they don't pay the electricity bills. There are also a few things wrong with the specs you quoted. 282 GB is split between two GPUs, it's 144 GB per GPU (not sure where the extra 6 GB went). The TPU pod throughput number you quoted is interconnect bandwidth which you compared to GH200's memory bandwidth. Those numbers are not comparable. I believe GH200 does beat TPU v5p per chip, however the differences are not anywhere near as large as your comparison suggested. And it's likely that TPU v5p is dramatically more cost effective but we don't have Google's internal numbers to prove that.
- nabla9 3y ago> 282 GB is split between two GPUs There is only one GPU in GH200 and it gets all 282 GB HBM3e memory. There is another 480 GB LPDDR5X memory for the another chip (Grace CPU, not GPU). GH200 interconnect bandwidth trough NVLink is 900 GB/s.
- modeless 3y agoI don't believe this is true. GH200's GPU has 144GB of HBM3e according to their datasheet. https://resources.nvidia.com/en-us-grace-cpu/grace-hopper-superchip https://resources.nvidia.com/en-us-grace-cpu/grace-hopper-su... The 282 GB figure comes from a "dual configuration" server with two CPU chips and two GPU chips: > the new platform will be available in a wide range of configurations. The dual configuration [...] comprises a single server with 144 Arm Neoverse cores, eight petaflops of AI performance and 282GB of the latest HBM3e memory technology. Note that one Grace Hopper CPU has 72 cores so this configuration clearly has two CPUs and two GPUs in one server, and the 282 GB HBM3e total is split between the two GPUs with 144 GB each (again, not sure where the extra 6 GB went, maybe disabled for yield issues?). https://nvidianews.nvidia.com/news/gh200-grace-hopper-superchip-with-hbm3e-memory https://nvidianews.nvidia.com/news/gh200-grace-hopper-superc...
- 3y ago
- tgtweak 3y agoDGX GH200 is next gen, but afaik not available to anyone aside from internal partners. All the quoted data here is for several of them together in a chassis. >Each NVIDIA Grace Hopper Superchip in NVIDIA DGX GH200 has 480 GB LPDDR5 CPU memory, at an eighth of the power per GB, compared with DDR5, and 96 GB of fast HBM3. (https://developer.nvidia.com/blog/announcing-nvidia-dgx-gh200-first-100-terabyte-gpu-memory-system/ https://developer.nvidia.com/blog/announcing-nvidia-dgx-gh20...) Looks like a single superchip has 96GB of hbm3, similar to a single v5p tpu. The H100's in the DGX H100 were 80GB per board.