18 ms·
Google Launches AI Supercomputer Powered by Nvidia H100 GPUs
- amelius 3y agoI'm tired of hearing the same name again and again. Where is the competition?
- jacooper 3y agoReplying "ROCm doesn't support your GPU model" in GitHub issues
- zo1 3y agoWe need to stop feeding the advertising machine. That'll starve Google and other advertising parasites. First step in doing that is opening up the Android ecosystem and legislating Google's hands out of that pie. I can't even so much as shit on an android phone without requiring a valid Google account. /crass joke
- Georgelemental 3y agoI think the name in question is Nvidia, not Google.
- zo1 3y agoThis was launched by Google, hence my comment. But yeah, guess it's just as likely the other comment was about Nvidia.
- Aissen 3y agoPeople want to rent the pricey NVIDIA DGX H100. So Google just put it in their DC, letting customers pay its full price it every ~3 months; plus they don't have to operate it, which is win (or is it ?).
- MuffinFlavored 3y agothey have to pay for the power and the staff to set them up/manage it
- verdverm 3y agoThat's generally the cloud in a nutshell, they price accordingly
- plumeria 3y agoI wish that humanity gets to harvest static energy one day, so that everyone is able to run the experiments required in large-scale deep learning research, not only a handful of deep pocketed organizations.
- m00dy 3y agodon't worry, smartest people I know are working on this problem.
- plumeria 3y agoOn the other hand, liquid networks [0] seem promising in not requiring huge amounts of energy (by not requiring huge numbers of parameters). [0] https://www.youtube.com/watch?v=p1NpGC8K-vs https://www.youtube.com/watch?v=p1NpGC8K-vs
- MichaelZuo 3y agoYou likely would not want to live in a future where every individual could each 'harvest' several gigajoules of 'static energy'.
- politician 3y agoCue calls to reclassify balloons as assault weapons. /s
- totoglazer 3y ago
- mimd 3y agoMore info in these articles. Nice tidbits on design. https://www.hpcwire.com/2023/05/10/googles-new-ai-focused-a3-supercomputer-has-26000-gpus/ https://www.hpcwire.com/2023/05/10/googles-new-ai-focused-a3... https://www.nextplatform.com/2023/05/11/when-push-comes-to-shove-google-invests-heavily-in-gpu-compute/ https://www.nextplatform.com/2023/05/11/when-push-comes-to-s... https://www.nextplatform.com/2023/03/21/inside-the-infrastructure-that-microsoft-builds-to-run-ai/ https://www.nextplatform.com/2023/03/21/inside-the-infrastru...
- vsareto 3y agoNVIDIA really getting up there in importance with the likes of ASML
- abudabi123 3y agoThis type of thinking machine needs measurements like time to model convergence at ability for riding a bicycle or conducting an orchestra.
- metadat 3y agoHow many H100s is required to get to 26 exaflops?
- jleahy 3y ago250
- metadat 3y agoInteresting, so what is the compute power of the 1000-node A100 super cluster my team has been allocated at work? I was expecting Google to be much bigger than us.
- sanxiyn 3y agoThis is for Google Cloud users. My understanding is that Google mostly uses TPU internally.
- Aissen 3y agoBack of the envelope math is that H100 is twice as fast as A100 (task may vary). So your 1000-node A100 very, very fast. Now, the GPU-to-GPU links (NVLink) might often give them a big advantage for some workloads, letting them exchange data without going through the CPU, and virtually address more memory if your want to manipulate very large models. So it's hard to answer properly without knowing the topology of your cluster. Also, note that this "supercomputer", is probably "just" a DGX H100 in Google's DC.
- lwkl 3y agoThey use their own TPUs like described in this paper [0]. They talk about 4096-chip supercomputers so this should give you an idea about what we are talking here. The paper is pretty fascinating stuff. They are using optical interconnects for example, which sounded like science fiction a few years ago. [0] https://arxiv.org/pdf/2304.01433.pdf https://arxiv.org/pdf/2304.01433.pdf
- planetafro 3y agoDid you read the article? It says 8...
- bushbaba 3y agoDoes this mean google just deprecated TPUs? Not surprised.
- endisneigh 3y agoWhy would you assume that?
- cubefox 3y agoEither GPUs are better for most AI tasks or TPUs. Both being overall approximately equally good is very unlikely.
- panarky 3y ago> most AI tasks Different workloads require different infrastructure. Can your workload saturate the TPU without getting throttled by memory or network? Great! Use TPUs and reduce training cost. But if your TPUs are idle 70% of the time because the constraint is getting data to them ... "A3 represents the first production-level deployment of its GPU-to-GPU data interface, which allows for sharing data at 200 Gbps while bypassing the host CPU. This interface, which Google calls the Infrastructure Processing Unit (IPU), results in a 10x uplift in available network bandwidth for A3 virtual machines (VM) compared to A2 VMs."
- cubefox 3y agoThat's why I said "most" and "overall". Of course TPUs will have a niche. But it looks like the vast majority of money spent on ML compute is converging on GPUs.
- vlovich123 3y agoBut you could do an equivalent TPU<>TPU interlink. Surely that can’t be the reason.
- cavisne 3y agoTPU's have a TPU-TPU interconnect that is faster and lower latency than any GPU cluster [1]. That said this is a huge leap for GPU's on GCP. For A100's SOTA is 1.6tbit per host over Infiniband (which azure and some smaller gpu clouds provided), AWS had 400-800 Gbit and GCP had .... ~100gbit. SOTA seems to be 3.2Tbit for H100 clusters so this still seems a bit slow? (Tricky as they don't give us a clear number just 10x). H100's are much more powerful per chip though so at least initially the clusters will be smaller and not network bound. The tricky thing is no one other than Azure of the big providers seems willing to pay Nvidia's margins for RDMA switches, it seems this is still the case. [1] https://arxiv.org/pdf/2304.01433.pdf https://arxiv.org/pdf/2304.01433.pdf
- ChuckNorris89 3y agoSo this is why Nvidia isn't lowering the price on the GPUs despite them sitting on the shelves and not selling. They make enough money from customers in the data center and supercomputer businesses that gaming is just a small market.
- adam_arthur 3y agoGaming is a huge chunk of their revenue, around $2B in recent quarters, with datacenter around $3.5B. Despite the AI hype, Nvidia’s datacenter revenue was down QoQ and only up 10% YoY. It remains to be seen if the growth trajectory has changed meaningfully over the last quarter, because the stock is priced for massive earnings growth while their revenue and earnings have been actually shrinking. We’ll find out on the upcoming earnings call https://www.macrotrends.net/stocks/charts/NVDA/nvidia/revenue https://www.macrotrends.net/stocks/charts/NVDA/nvidia/revenu... https://www.macrotrends.net/stocks/charts/NVDA/nvidia/eps-earnings-per-share-diluted https://www.macrotrends.net/stocks/charts/NVDA/nvidia/eps-ea...
- HybridCurve 3y ago>Gaming is a huge chunk of their revenue, around $2B in recent quarters, with datacenter around $3.5B. Is it? I remember hearing they didn't make much money from their consumer GPU products a few years back. This was one of the reasons why they tried to clamp down so aggressively on people using desktop GPUs for computing. They had made a number of driver changes which restricted the capabilities of anything but the tesla and quadro products. They were also restricting bulk purchases of their cards.
- UncleEntity 3y agoIIRC they did that because the crypto miners were buying them all up and they wanted normal folks to be able to buy them too. Not that big of a deal these days and I doubt they’d make it so someone couldn’t take an off the shelf GPU and play around with llamas.
- Ologn 3y ago
- IanCal 3y agoKinda feels like the main thing google launches is waiting lists.
- ttul 3y agoAnd my main worry is: are they just going to cancel the new thing that my company invested six months and $250,000 of engineering time integrating with…
- noahl 3y agoNo, not for GCP stuff. I don't know of a single GCP product that's been shut down, although I could be missing something. But their track record for GCP is, I think, what you would want a cloud provider's record to be. (I should mention that I work for GCP. But this is just based on my own memory.)
- snek_case 3y agoProbably doesn't matter that much because it's just hardware. Presumably not that hard to run your software on another intel box with nvidia GPUs. There's also plenty of demand for nvidia GPUs right now, still no guarantee given it's Google, but it would be hard not to make money with this.
- ukuina 3y agoThis is the way.
- smrtinsert 3y agoQuote of the year
- crazysim 3y agoThe Gmail waiting list was one of the most legendary waiting lists. Anyone else remember inviting people to Google Docs?
- stOneskull 3y agoi sold quite a few gmail invites for 99c each on ebay. it was fun.
- naillo 3y agoExcited for Google to have gotten this kick in its rear and might finally do some really interesting publicly available things in ML.
- deleted 3y ago[deleted]
- ksec 3y agoGoing Slightly Off Topic. This is why Leading Edge Node will continue to be well funded. Consumer Electronics ( Mainly Smartphone ) Silicon usage has been the main push behind the development of Pure Play leading edge foundry in the past 10 years. Despite the predicted / expected drop of Smartphone sales, considering the potential shown by ChatGPT or Bard, GPU or Wafers dedicated for AI will continue to be in demand for at least another 5 years. In terms of lead time into the investment of silicon development that means we can continue to expect progress all the way till 2030, either 1nm or 0.8nm.
- Our_Benefactors 3y agoCan you elaborate on the supposed “wonky physics” that goes on when things get small? I’ve seen it thrown around that 3nm is “almost” the smallest size that can be made before different classes of physical errors are introduced due to the extremely small distance between gates.
- ksec 3y agoRead [1] from 2020, I have replied there along with the economics issues I was referring to which AI demand will likely solve, or at least part of the solution. [1] https://news.ycombinator.com/item?id=24618031 https://news.ycombinator.com/item?id=24618031
- binkHN 3y agoTechnical detail: > Each A3 supercomputer is packed with 4th generation Intel Xeon Scalable processors backed by 2TB of DDR5-4800 memory. But the real "brains" of the operation come from the eight Nvidia H100 "Hopper" GPUs, which have access to 3.6 TBps of bisectional bandwidth by leveraging NVLink 4.0 and NVSwitch.
- zrm 3y agoInterestingly the "4th generation Intel Xeon Scalable processors" themselves have up to 2.45 TBps in memory bandwidth, with the 8-socket configuration, or 2 TBps with 2-socket Xeon Max and HBM. If they'd make an 8-socket Xeon Max it would have 8 TBps. Considering that the Xeon Max 9462 is $8000 vs. the H100 going for north of $40,000, that could be interesting.
- jocaal 3y agoThe throughput these gpu's have make the price pretty competitive, but I think AMD is working on a APU in their instinct lineup. That could be pretty competitive since Nvidia is overcharging for memory and you could just use sticks instead
- smoldesu 3y agoNvidia is also working on a tightly integrated datacenter solution, FWIW: https://www.nvidia.com/en-us/data-center/grace-cpu/ https://www.nvidia.com/en-us/data-center/grace-cpu/
- zrm 3y agoA lot of this is workload-dependent. LLMs for example seem to be memory-bound, so a fast CPU with HBM or a large number of memory channels should do well. Socket SP5 has 12 channels, which is 461 GBps per socket at DDR5-4800. Intel is getting 1 TBps from HBM, but then you're paying for HBM. $8000 for the cheapest Xeon Max vs. $3000 for the Epyc 9334 with the same number of cores or ~$1000 for the least expensive thing that will fit in the 12-channel socket. CPUs also have a cost advantage because then you don't need a CPU and a GPU. Other things might be more compute bound. Then a fast GPU in a socket with a lot of memory channels worth of cheap sticks should be fun.
- ilaksh 3y agoI think the most interesting AI hardware stuff is about memristors or some type of compute-in-memory. https://arxiv.org/pdf/2303.07470.pdf https://arxiv.org/pdf/2303.07470.pdf https://ieeexplore.ieee.org/abstract/document/9669041 https://ieeexplore.ieee.org/abstract/document/9669041 Maybe there will be something like transformers but more suited to crossbar arrays of memristors. If that actually makes sense.
- narrator 3y agoI think HP has all the patents on these. Maybe when their patents expire some company that can actually release a product will make good use of them instead of having a business model consisting of bricking printers that use off-brand ink.
- ip26 3y agoI have yet to see a proposal for compute-in-memory that isn’t actually compute-near-memory and keeps the density of memory arrays. If you’re still doing row-column access, it’s just another Von Neumann machine. If you have compute hardware within each row to perform operations on every row in parallel, it’s now just another ALU.
- segmondy 3y agoI'm more interested in what normal folks are running at home. What are your builds?
- kwerk 3y ago2x 3090 but just getting started with fine tuning so I’m not sure how far I can push it
- ArtWomb 3y agoHonestly it's a $79 Lonovo 3 Chromebook running a Gcloud A3 virtual workstation over 5G from the golf course ;)
- rawoke083600 3y agoWhats the battery life on that :)
- kcb 3y agoI feel like if you go that route you should at least get something with a bigger nicer screen.
- NavinF 3y ago4090 and 3090 on personal desktop; 4 x 2080Ti in data center
- zirgs 3y agoRTX 3080 on a laptop. 8 GB was more than enough for gaming, but I get out of vram errors quite frequently.
- Havoc 3y agoFYI rumour has it next round of titan GPUs are supposedly coming with 48GB Of course there is always something better on horizon, but if you're building soon that may be worth the wait
- nullsense 3y ago
- MichaelRazum 3y agoSo can we train now 10t or 100t LLM models? I mean assuming that the dataset is large enough
- lordofgibbons 3y agoDoes this mean Google is giving up on TPUs? TPUs were supposed to be their unfair advantage in the cloud ML/DL space. But from what I've experienced, and have heard from other engineers, there's always some subtle incompatibility with TPUs that requires modifying the training/eval scripts. I wonder why they didn't try to polish the rough edges with Pytorch, et al. If they're admitting TPUs aren't their competitive advantage, then why not sell it to other hosting providers, or hell, even directly to ML scientists and enthusiasts? They'll finally get economies of scale, and take business (and mind share) away from NVidia's monopoly.
- totoglazer 3y agoThere’s demand in GCP for H100s so they offer them. I doubt Google itself is a big user.
- jsnell 3y agoThere's a few comments to this effect in the thread, and I don't entirely understand where they're coming from. There's nothing in the article suggesting they've changed their strategy with TPUs in any way. The word TPU isn't even mentioned here. There's no suggestion they're actually using this internally either. There's no benchmarks showing that it's more cost-effective or scales better. And isn't your second paragraph the obvious reason for why this product (A3) exists? It's something they expect to sell to cloud customers who have an existing GPU-based workflow, and just want to run it as-is as fast/cheap/scalable as possible, without worrying about compatibility, and making sure they can always move the workload to some other cloud provider or on-prem if needed. It's like suggesting Sony releasing some of their games on the PC means they're deprecating Playstation. (Maybe there would be more details in the IO talk. Does anyone know which one this announcement is from?)
- joseph_grobbles 3y ago>There's nothing in the article suggesting they've changed their strategy with TPUs in any way Google owns and designs their own TPUs. They offer these TPUs in the cloud. I've seen many comments in here about how next-level TPUs are (despite zero evidence indicating that). Google even disclaims their TPU by saying that you shouldn't compare it with the H100 given node levels et al. Their premiere offering is an nvidia H100 offering. Yes, of course this is a pretty telling indication. If Google was all in on TPUs they'd be building mega TPU systems and pushing those. Instead they're pushing nvidia AI offerings.
- marban 3y agoGonna ask GPT how big exaFlops are...
- andirk 3y agoGood luck. OK Google just told me to put a "terabyte" of salt on my air fryer broccoli.
- noogle 3y agoShould we buy Nvidia stock then? The greatest technological advancement in recent years critically depends on the hardware from a single company with no competition. yet Nvidia stock is still below its 2021 peak. How so?
- ddorian43 3y agoWatch its PE and forward PE. And look at earnings after 2 weeks.
- gitfan86 3y agoIt is unknown how much pricing power NVDA has. Can they 3x the price of everything And still sell out?
- wmf 3y agoIt sounds like they already did that. A100 was very expensive and H100 is even more expensive.
- noogle 3y agoWhy not? They seem to be a lot of leeway before any specific company will find it cheaper to design their own chips, or even to move to AMD (ROCm is not as well supported). Perhaps someone like OpenAI has both the expertise and incentive to do so, but not many others.
- gitfan86 3y agoSo if you think that maybe OpenAI has other options if NVDA increases their prices why do you think that one of the other big names ( MSFT,GOOG,IBM,TSLA,AMD,INTC,Facebook ) also cannot do the same thing? I'm not saying you are wrong or right, btw
- noogle 3y agoI guess for the same reason most of them keep buying from Intel - their market position allows them to pass on the cost to their customers, so it's not worth the distraction. OpenAI is more of a "one-(very impressive)-trick-pony", so they have a stronger incentive.
- 1024core 3y agoY U no use TPU??
- qwertox 3y agoAMD should be gifting their GPUs by the dozens to the most prolific Open Source contributors if they want a piece of the cake. Their lack of access to CUDA is really harming them badly.
- blihp 3y agoIt's more than just that: for the money, their consumer GPUs don't compete in compute tasks (especially inference/training) and their Linux compute drivers are a pile of steaming garbage on consumer hardware. It's really interesting/depressing to watch as they've done a nice job of supplying good open source graphics drivers. They really seem to be lacking something at a leadership level in terms of understanding GPU compute outside of specific enterprise/scientific use cases.
- nr2x 3y agoWhereas NVIDIA is basically just “we’re an AI company now”.
- ChuckNorris89 3y agoNo, they've become a general compute company selling pickaxes for whatever the current goldrush tends to be. Now it's AI, yesterday it was crypo-currencies, the day before is was PC games and video editing. They've been trying to push their GPUs as CPU alternatives everywhere especially in the datacenters where their presence grew since the acquisition of Mellanox. They also tried to acquire ARM, to squeeze both Intel and AMD out of the CPU market completely. I hate what they've done to the PC gamers, but as a company trying to grow in more markets and make even more money, they've executed insanely well strategically, leaps ahead of AMD.
- nr2x 3y agoI mean their public messaging is pure AI now, but I also think they have been very smart about running the right direction since Alexnet came out and showed what you could do with a GPU.
- deleted 3y ago[deleted]
- 29athrowaway 3y agoIt is interesting how the definition of a supercomputer changes over time. Compared to decades ago now everyone carries a supercomputer.
- Frummy 3y agoChange is the constant
- nullsense 3y agoApparently RTX4090 with FP8 is equivalent to worlds fastest supercomputer from 2007. So, in some sense I have a supercomputer on my desk:)
- 29athrowaway 3y agoThat is worth a HN post.
- pookah 3y agoWe're all lectured to look side-eye at bitcoin while these machine learning processes consume more energy than Las Vegas on meth. LOL.
- andirk 3y agoApples and oranges to an extent. 1) Knowledge is being derived from said energy use, and 2) nerds aren't extremely bitter that they didn't pick up that millionaire-making space cash when it was handed to them on a platter, before the rest of the world, a handful of years ago.
- diimdeep 3y agoWhat kind of compute so much compute is good for ? except AI