13 ms·
Nvidia H100 GPUs: Supply and Demand
- statguy 3y agoWhat is with this webpage, it appears to be blocked by pretty much every browser?
- reaperman 3y agoFWIW, works fine on a MacOS Safari client with zero customization.
- buro9 3y agoWorks fine on Firefox on Android Also works on Chrome on Android
- nanidin 3y agoIt's working fine in Chrome on Windows.
- helsinkiandrew 3y agoWorks fine on Chrome/Mac - it is however a very strange article structure. It was obviously very well researched with a lot of interesting information but I found it very hard to read - Q/A style, hardly any paragraphs with more than a sentence, sections where the heading is longer than the content, a huge number of quotes.
- KirillPanov 3y agoThis is a pretty awful article. AFAICT it consists of a bunch of anecdotes by thought-leader types followed by a corny-ass song. HN, you can do better. I believe in you. Try harder.
- visarga 3y agoIt is a well researched report on the GPU availability crisis. The song is funny, especially Mark riding a LLaMA and the cabaret dance.
- reaperman 3y agoShockingly well-researched, even. I found it super informative towards understanding the current market. Really impressive aggregation of sources. I wish it had some authors listed so that I could see some of their other work.
- _m5co 3y agoThanks! (Author here, see other work in my HN submissions and comments)
- matzab 3y agoIt looks like they didn't scroll past the embedded video, so they didn't even get to the table of contents. >"HN, you can do better" - indeed.
- deleted 3y ago[deleted]
- Tepix 3y agoI just forwarded this solid article to my colleagues. Did you perhaps miss the meat of the article that's after the video? PS: The song is also very good.
- KirillPanov 3y agoI bet you listen to stuff like http://www.openbsd.org/lyrics.html http://www.openbsd.org/lyrics.html
- sjsdaiuasgdia 3y agoI love a good lyric swap, I like the original song, and I couldn't make it through one minute of the song. Song is not good.
- deleted 3y ago[deleted]
- tin7in 3y agoA lot of it is from people with first hand knowledge of buying or selling capacity/hardware.
- ThatMedicIsASpy 3y agoIt has parts where I thought it's AI generated
- KirillPanov 3y agothat would explain a lot
- Symmetry 3y agoIf you scroll down past the corny song you'll find the table of contents for a very thorough article. The table sort of looks like footer text to the web page if you don't look too carefully, I almost missed it myself.
- slushh 3y ago>Who is going to take the risk of deplying 10,000 AMD GPUs or 10,000 random startup silicon chips? That’s almost a $300 million investment. Ironically, Jensen Huang did something like this many years ago. In an interview for his alma mater, he tells the story about how he had bet the existence of Nvidia on the successful usage of a new circuit simulation computer from a random startup that allowed Nvidia to complete the design of their chip.
- urthor 3y agoHonestly, that's a really excellent point. Successful startups are successful because they do exactly that. Successfully.
- klelatti 3y agoRisk / reward though. What's the reward? Are the alternatives better than Nvidia? Doesn't seem so and the risks are significant. On the other hand if you can't get H100s then nothing to lose!
- toasted-subs 3y agoSuccessful startups cater to specific customers and provider one on one care that isn't typically possible with larger companies.
- anewhnaccount2 3y ago> >Who is going to take the risk of deplying 10,000 AMD GPUs or 10,000 random startup silicon chips? That’s almost a $300 million investment. Lumi: https://www.lumi-supercomputer.eu/lumis-full-system-architecture-revealed/ https://www.lumi-supercomputer.eu/lumis-full-system-architec...
- toasted-subs 3y agoAMD GPUs are relatively well tested. Anybody who's looked at nvidia's architecture could tell you it's not perfect for every application. Similarly AMD's isn't either. If you know what your application would be and have the $300 million custom chips may be way more wise. Something you'd only get if you make things in-house/at startups.
- holoduke 3y agoIts really time for some competition. Either AMD or some chinese company like 'more threads' need to speed up and get something on the market to break the Nvidia dominance. Nvidia is showing already some nasty typical evil behavior that has to be stopped. I know not easy with fully booked partners at Samsung/tmct/etc
- ukd1 3y agohttps://tinygrad.org https://tinygrad.org is trying something around this; currently working on getting AMD GPUs to get on MLPerf. Info on what they're up to / why is mostly here - https://geohot.github.io/blog/jekyll/update/2023/05/24/the-tiny-corp-raised-5M.html https://geohot.github.io/blog/jekyll/update/2023/05/24/the-t... - though there are some older interesting bits too.
- blitzar 3y agoAMD / Intel should be throwing them 10's or 100's millions (or jsut straight up human hours of work) to make that work. If / when it does it would be 10's or 100's of billions of (additional) market cap for them.
- spiderfarmer 3y agoSo why doesn't AMD invest more in quality software? Everybody knows it's holding them back. Right now a tiny corporation has to jump through hoops to do work they could have done at any point in the last decade. Why don't they pull a great team together and just do the work, if the pay off is that big?
- blitzar 3y agoI honestly do not know. Perhaps they think using GPUs for computation is a passing fad? They hate money? Their product is actually terrible and they dont want to get found out (that one might be true for intel)?
- throwaway2990 3y ago
- MrBuddyCasino 3y agoIf the bottleneck isn’t TSMC wafer starts but CoWoS, where exactly does that bottleneck come from? From what I understand, its the interposer connecting GPU and HBM wafers. Are they hard to make, is the yield bad, are there insufficient production lines, …?
- kanwisher 3y agoNah cause AMD could be being used if they had software. Also Intel is totally different fabs and wafers, but both are not close in software to Nvidia
- gyrovagueGeist 3y agoIt's weird that this article ignores the entire traditional HPC market/DoE/DoD demand for H100s.
- _m5co 3y ago(Author here) I'd be interested in writing about this in the future. I need help though because I don't know people in those spaces. Email is in my profile. I had a section on this in early drafts but removed it as I didn't feel confident enough in my research.
- cavisne 3y agoJenson could write one of the clouds a license to use 4090s in a DC and make this crunch disappear overnight (would be rough for gamers though)
- treprinum 3y agoThere's A6000 Ada for that (you can rent servers with 4xA6000 at Lambda Labs). Moreover, 4090 has only 24GB memory, H100 has 80GB.
- mk_stjames 3y ago4090s have 24GB of 384-bit-wide GDDR6 with no ability to interconnect that memory to other 4090s except thru PCIe bandwidth. H100s have 80GB of 5120-bit HBM with SXM NVLink for 8-at-a-time in a rack. HUGE difference in bandwidth when doing anything where the inferring the model needs to be spread over multiple GPUs, which all LLM's are. And even more of a difference when training is in play.
- paxys 3y ago4090 (and all consumer chips of its class) have terrible efficiency and are not suitable for use in a DC.
- rawoke083600 3y agoVery good article ! Nice insight as to who/what/where and how much :)
- _m5co 3y agoI appreciate it, thanks for the comment
- cesaref 3y agoHaving worked with Quants before, the reality is that however big your compute farm, they will want more. I think this is what is going on with these large AI companies - they are simply utilising all of the resource they have. Of course they could do with more GPUs. If you gave them 1,000x their current number, they'd think up ways of utilising all of them, and have the same demand for more. This is how it should be.
- danuker 3y agoThere must be some point at which the cost of the endeavor becomes greater than the profits to be made.
- cesaref 3y agoAbsolutely! But thats for management to work through, the engineers just want more :)
- paul-nai 3y agoFrom what I've seen the utilisation is still pretty poor, they're being used so poorly and most companies can get away with less GPUs. Instead of looking at how to optimise their workflow they just slap GPUs on
- bushbaba 3y agoI’ve noticed the same. Very low utilization. But they are all used at peak every few weeks. For many companies having more GPU cost to unblock velocity of innovation here is worth it. As the benefits of improvement to your top line revenue far exceeds the GPU cost.
- spott 3y agoAre you talking poor utilization from a "we have a bunch of GPUs sitting idle" sense, or poor utilization from a performance standpoint (can't keep their opus fed with data kinda thing)?
- 3y ago
- paulcjh 3y agoBy far the biggest issue it utilisation of the GPUs, if people worked on that instead of throwing more power at problems this would be way less of a problem
- credit_guy 3y agoAnd you are under the impression that people are not working on that?
- paulcjh 3y agoI know people are working on it - but IMO it's not spoken about anywhere near as much as slapping more GPUs on a problem. I'm on about enterprise workload orchestration not just making a model faster
- jjoonathan 3y agoIt's spoken about all over github and the open source AI ecosystem. HN and casual media are heavily attention biased towards big money and big datacenters, yes.
- pas 3y agohow good is "historically" the open source AI ecosystem? usually community projects have different target constraints, right? (for example Mastodon doesn't even want to be Twitter, it wants its own thing, it wants to be a different answer to the social network/media question, even if there are obvious and fundamental similarities.) how does this play out in the open source AI communities?
- CurrentB 3y agoIs Nvidia even able to capture a proportionately significant amount of revenue from increases in demand for GPU cycles? As the article describes, there are real bottlenecks, but how does this play out? My assumption is that Nvidia doesn't have proportional pricing power for some reason. If demand increases 10x, they can't raise prices to the same extent (correct me if I'm wrong). How would that even play out then? Is everyone in the world simply stuck waiting for Nvidia's capacity to meet demand? There is obviously a huge incentive now to be competitive here, but is it realistic that anyone else might meaningfully meet demand before Nvidia can?
- binarymax 3y agoTheir prices are already high enough :) Base price of the H100 is something like $36000 USD.
- AnotherGoodName 3y agoA a ballpark you can guess ~100x the margin of gaming GPUs since the transistor counts and architecture are similar enough to high end gaming GPUs that sell for a fraction of the price and the extra RAM isn't going to close that gap. When you have 100x the margin on a product you only need to sell 1% as many to almost double profit. 10% would be more than 10x profit.
- nl 3y agoIt's weird not more is made of the fact the Google's TPUs aren't the only real, shipping, credible alternative to NVidia. I wonder how much a TPU company would be worth if Google spun it off and it started selling them?
- gravypod 3y ago(opinions are my own) https://coral.ai/products/ https://coral.ai/products/
- sargun 3y agoGoogle kind of has done this with Coral: https://coral.ai/about-coral/ https://coral.ai/about-coral/ These TPUs obviously aren't the ones deployed in Google's datacenters. That being said, I'm not sure how practical it would be to deploy TPUs elsewhere. Also, Amazon's Infinera (sp?) gets a fair bit of usage in industrial settings. It's just that these nvidia GPUs offer an amazing breeding ground for research and cutting edge work.
- dimatura 3y agoThe Coral line of products targets embedded/IoT inference applications, and are on the low end as far as processing power goes. AFAICT the recent surge of demand, mostly fueled by LLMs, is for GPUs on the opposite end of processing power and RAM size.
- deleted 3y ago[deleted]
- atty 3y agoI think the author has missed a pretty large segment of demand. Non-cloud/non-tech enterprises are also buying large quantities of H100s and A100s for their own machine learning and simulation workloads. Where I work, we are going to have more than 1000 H100s by the end of the year, I am very excited to start benchmarking them soon :)
- _m5co 3y agoI agree. (I'm the author) Touched on that briefly here https://news.ycombinator.com/item?id=36955403 https://news.ycombinator.com/item?id=36955403. Need help with that research; please email - email is in profile. Had a section on it in early drafts; didn't feel confident enough; removed it. Would be good to have more on enterprise companies like Pepsi, BMW, Bentley, Lowes, as well as other HPC uses like oil and gas, others in manufacturing, others in automotive, weather forecasting.
- YetAnotherNick 3y agoJust curious, what price do Nvidia charge enterprises? For $40k, it just doesn't make sense to buying it compared to renting from lambda labs or some other place for $2/hour(or $17k/year).
- atty 3y agoThat’s a good question and I can’t give a straight answer, because we are actually on three year lease plans through a vendor (think HPE). I know we get a pretty decent deal in general from them since we have basically an entire data center worth of hardware from them, but I don’t know how good the deal is on the GPUs, since they’re in such high demand.
- Uehreka 3y agoI know it’s off topic, forgive me HN Gods, but this was right at the top of the article and threw me off: > Elon Musk says that “GPUs are at this point considerably harder to get than drugs.” Does Elon have a hard time getting drugs?
- sixothree 3y agoSounds like no to me.
- latchkey 3y agoWhat nobody is talking about here is that there is no more power available in the US. All the FAANGS have scooped up the space and power contracts. You can buy all the GPUs you can possibly find. If you want to deploy 10MW+, it just doesn't exist. These things need redundant power/cooling, real data centers, and can't just be put into chicken farms. Anything less than 10MW isn't enough compute now either for large scale training and you can't spread it across data centers because all the data needs to be in one place. So yea... good luck.
- vorpalhex 3y agoWith things like solar and wind installs becoming more off-the-shelf, is there any path there? What does 10MW of solar/wind look like? Are we talking the size of a big ranch or the size of a small county?
- latchkey 3y agoIt doesn't matter, this stuff wants/needs to be deployed yesterday, not in 2-3 years. I'm starting with raw power, but even that isn't the limiting factor... it goes deeper... try to buy a bunch of large transformers, those are year long waitlists. Texas has a lot of wind. At this scale, it is mostly grid power anyway. Grid is a mixture of everything. Oh and solar has this pesky issue of not working in the evening, so then you have another problem... storage. ;-) I should add... you want backup generators for your UPS systems? Those are a 4.5 years backlog.
- vorpalhex 3y agoI'm sort of assuming this is like most other booms and will last several years. Sure the best time to be ready to jump in was yesterday, the the second best time is to get the ball rolling now even if your lead time is years. And yes obviously renewables won't cover 24/7 but if I have a choice between no data center and a 60% time data center.. give me the 60%.
- latchkey 3y ago
- zoogeny 3y agoThe real gut-punch for this is a reminder how far behind most engineers are in this race. With web 1.0 and web 2.0 at least you could rent a cheap VPS for $10/month and try out some stuff. There is almost no universe where a couple of guys in their garage are getting access to 1000+ H100s with a capital cost in the multiple millions. Even renting at that scale is $4k/hour. That is going to add up quickly. I hope we find a path to at least fine-tuning medium sized models for prices that aren't outrageous. Even the tiny corp's tinybox [1] is $15k and I don't know how much actual work one could get done on it. If the majority of startups are just "wrappers around OpenAI (et al.)" the reason is pretty obvious. 1. https://tinygrad.org/ https://tinygrad.org/
- tedivm 3y agoI'd argue that you really don't need 1000+ H100s to test things out and make a viable product. When I was at Rad AI we managed just fine. We took a big chunk of our seed round and used it to purchase our own cluster, which we setup at Colovore in Santa Clara. We had dozens, not hundreds, of GPUs and it set us back about half a million. The one thing I can't stress enough- do not rent these machines. For the cost of renting a machine from AWS for 8 months you can own one of these machines and cover all of the datacenter costs- this basically makes it "free" from the eight month to three year mark. Once we decoupled our training from cloud prices we were able to do a lot more training and research. Maintenance of the machines is surprisingly easy, and they keep their value too since there's such a high demand for them. I'd also argue that you don't need the H100s to get started. Most of our initial work was on much cheaper GPUs, with the A100s we purchased being reserved for training production models rapidly. What you need, and is far harder to get, is researchers who actually understand the models so they can improve the models themselves (rather than just compensating with more data and training). That was what really made the difference for Rad AI.
- latchkey 3y agoYour (ex)-company literally has the name AI in it, so yea, it makes sense to buy compute, not rent. That said, a lot of other businesses don't want to take on the capex, but they do need to train some models... and those models can't run on just a half a million worth of hardware. In that case, someone else is going to have to do it for you. It works both ways and there are no absolutes here.
- tim_sw 3y agoThis is a very high quality writeup.
- _m5co 3y agoThank you
- suierklepye 3y agoThe discussion on how different industries are vying for the same limited supply of components adds another layer of complexity to the situation. It's intriguing to see how GPUs are being sought after by diverse sectors, and this phenomenon showcases the versatility and importance of these components in modern technology.