11 ms·
Nvidia H100 and A100 GPUs – comparing available capacity at GPU cloud providers
- pulkas 3y agoAnother cloud gpu comparison web site : https://cloud-gpus.com/ https://cloud-gpus.com/
- sabalaba 3y agoGreat summary of the industry. I do want to point out that we actually have the lowest price for H100s: 1.89/hr. Coreweave’s pricing requires long term commitment and you’re comparing our on demand PCIe cloud to their ‘reserved’ h100 cloud. Our reserved H100 + InfiniBand cloud is priced as follows and is the lowest in the world: $1.89 / hour for 100% upfront $2.04 / hour for 80% upfront $2.15 / hour for monthly payment See: https://lambdalabs.com/service/gpu-cloud/reserved https://lambdalabs.com/service/gpu-cloud/reserved
- sabalaba 3y agoWe just dropped our H100 on-demand prices to 1.99/hr. It's now the lowest in the world, you don't need to talk to a sales person, and it's available now: https://t.co/AcqRIITS1s https://t.co/AcqRIITS1s
- nkingsy 3y agoPeople talk about cards not being worth the electricity vs cloud. Seems like an a100 pulls 300w, costs $1.50/hr ish to rent, and costs $12,000 to buy, meaning it pays for itself with 1 year of constant use.
- JoeOfTexas 3y agoLabor and infrastructure add much more costs. But I'm sure with the current hype they are all making bank.
- floatinglotus 3y agoMost of the new providers are putting them in liquid cooling to cut down on electric.
- Eisenstein 3y agoWhy would liquid cooling cut down on electricity cost? The only difference is fan power vs pump power -- is it really much different?
- frankreyes 3y agoIf I had to guess the difference is made in building cooling, not per device.
- coolspot 3y agoIt is much easier to transfer X amount of heat with a liquid than with air. Liquid travels in pipes and has enormous mass heat capacity (Per unit of volume, water has about 3200 times the specific heat capacity of dry air (at 77°F)). Air needs wide straight ducts and fans every 10 meters to keep air flowing.
- Eisenstein 3y agoSure, but removing that heat to the outside still takes the same amount of energy. Unless they just pour the water down the drain it has to be transferred somewhere, no?
- burmanm 3y agoWouldn't that depend on the sink? If they can cool down to the water, then running a water-water-heatexchanger would require less power (due to better heat transfer) than simply cooling it to the air. Same I guess if they could use evaporation to do some of the cooling.
- Eisenstein 3y ago
- TechBro8615 3y agoSounds like a business opportunity. Of course, the risk is that by the time you make your money back, the cards you've been renting to customers are obsolete and you need to recapitalize...
- jpgvm 3y agoThe only think I am happy about all this AI hype is Infiniband is getting some love again. A lot of people using RoCE on Connect-X HBAs but still a lot of folk doing native IB. If HPC becomes more commonplace maybe we get better subnet managers, IB routing, i.e all the stuff we were promised ~10+ years ago that never had a chance to materialise because HPC became so niche and the machines had different availability etc requirements than OLTP systems that didn't demand that stuff getting built out. Especially the subnet managers as most HPC cluster just compute a static torus or clos-tree topology. There was a time I was running QDR Infiniband (40G) at home while everyone else was still dreaming of 10G at home because the adapters and switches were so expensive.
- pclmulqdq 3y agoQDR infiniband at home was such a great hack, and I was doing the same thing with my small network a little while ago. As long as you run connections point-to-point (with no switch) and keep the computers near each other (in twinax range), it's incredibly cheap and blazing fast.
- jpgvm 3y agoYeah I ran it for years with no issues. I modified a Mellanox switch to use quieter fans and eventually switched to fibre SFPs so I could make longer cable runs (also twinax is bulky/heavy). Was absolutely epic to use my ZFS NAS at essentially native speeds from my desktop and media PC. I was using the IB SRP target to export ZVOLs so with fast all-flash storage you really couldn't tell it wasn't running on a (giant) local SSD.
- floatinglotus 3y agoIB is still so expensive. The new Sprwctrum Ethernet switches are a better choice IMO.
- jpgvm 3y agoI haven't kept up with the pricing for the last decade, back when I was working with it was a fraction of the cost of comparable Ethernet. Maybe things have changed drastically though.
- garciasn 3y agoThey’re saying there is a 16 GPU min for GCP. I don’t see that at all. https://cloud.google.com/compute/gpus-pricing https://cloud.google.com/compute/gpus-pricing Am I misunderstanding the GCP pricing or how this analysis was done?
- loosescrews 3y agoThey also said that they couldn't find pricing when it is listed on the page you linked to.
- cavisne 3y agoThe (sad) reality of GPU capacity in public clouds at the moment is your basically need a sales call and a reservation to get any capacity. So the on demand price is not really relevant, maybe thats why they excluded it.
- pavelstoev 3y agoWhen capacity or cost constrained (or both) consider running ML on “lesser” GPUs like A40, A10s and even consumer grade 3090/4090s. With compiler optimizations these uncover surprising price-performance ratios.
- nickstinemates 3y agoYeah a home cluster of a few 3090's performs remarkably well.
- qeternity 3y ago> If you want access to H100s, you'll have to talk to a sales rep. None that we looked at offered instant access. Unless something has changed in the last few days, I was literally just using an H100 on LambdaLabs without having to speak to a sales rep…
- neilv 3y agoThis page might like to be a table.
- tikkun 3y agoYes. What headers would you suggest? I'd like a table format that'll still look good on mobile without needing to horizontally scroll.
- nickstinemates 3y agoIt's surprisingly simple to run all of this at home. I just took a bunch of GPUs out of storage and racked up a machine yesterday. From 0 to Linux/Docker install to Stable Diffusion / llama in about 2 hours.
- psshank 3y agoConsumer GPUs are great for inference at scale. We're a distributed GPU cloud with 10k+ consumer GPUs on our network. We can crank out 4500+ stable diffusion images per dollar (highest in the market), all on consumer 3060s. The high-end GPUs definitely have a role but for a bulk of production/inference serving can be done on consumer GPUs at 80-90% of the cost.
- buildbot 3y agoMost people don’t have multiple GPUs in storage, a server chassis, or a place to rack up said chassis.
- nickstinemates 3y agoA lot of people have PC's with modern video cards in them, especially this crowd. It's really no different.
- 6gvONxR4sf7o 3y agoAn H100 or A100 have very different capabilities from the cards you're talking about.
- nickstinemates 3y agoThanks, I wasn't aware.
- neilv 3y agoMany old consumer gaming GPUs will run an implementation of Stable Diffusion. But this page seems to be about getting use of H100 and A100, such as one might want for running or training decent-sized LLMs.
- fpgaminer 3y agoNice compiled list of stats. I'm not sure what they mean by H100s requiring pre-approval on LambdaLabs? Maybe I was grandfathered in since I had an account prior to their rollout, but I never had to do anything special to rent H100s there. Somewhat related and hopefully helpful: My experience so far using the H100 PCIes to train ViTs: Pros: A little over 2x performance compared to A100 40GB. 80GB by default. fp8 support, which I haven't played with but supposedly is another 2x performance win for LLMs. For datacenters and local workstations it's twice as power efficient as the A100s. Supposedly they have better multi-GPU and multi-node bandwidth, but my workload doesn't stress that and I can only rent 1xH100s at the moment. Cons: I had trouble using them with anything but the most recent nVidia docker containers. The somewhat official PyTorch containers didn't work, nor did brewing my own. Luckily the nVidia containers have worked fine so far. I just don't like that they use nightly PyTorch. In addition to that, because of their increased performance, they really start to push the limits on feeding data fast enough to them with existing system configurations. I was CPU-limited on the LambdaLabs 1xH100 machines because of this. Overall the pricing has worked out equal for my use-case, but I'm sure fp8 would make them more affordable. If fp8 were available out-of-the-box on PyTorch I'd play with it, but it's only available right now from some nVidia codebase specific to LLMs. Even at equal pricing, having twice the power per GPU and per node is a big win. That increases experiment iteration across the board. Side note: If I recall correctly, nVidia is heavily differentiating the H100 products by their interface this go around, which I find quite odd. The SXM version of the cards are supposed to be something like twice as beefy as the PCIE version? Not sure why they're doing that; gonna make comparing rentable instances all the more difficult if you overlook that little detail. A100 had a little bit of this, but the difference was never much in practice except between specifically the A100 40 GB PCIe and the 80 GB SXM, which was something like 10% faster. Anyone else having fun with the new toy our overlords have allowed us to play with?
- akiselev 3y ago> The SXM version of the cards are supposed to be something like twice as beefy as the PCIE version? Not sure why they're doing that I can't help but feel it's another cloud/enterprise cash grab. The SXM baseboards are way more expensive than server motherboards and NVIDIA makes them.
- hughw 3y agoWhat a great resource. I've been hoping someday for AWS to expose A100s 1 GPU at a time, but you can only rent the 8-bangers. A couple of vendors on this list will rent me one at a time.
- 6gvONxR4sf7o 3y agoCrazy to see stuff like the H100 at $2.40 per GPU hour. It feels like not that long ago that K80s would cost more than that. Moore's law still seems alive in dollars per FLOP.
- andrewstuart 3y ago$21,000 per year ouch. That'll buy you about 13 RTX 4090's outright.
- dahart 3y agoWhy aren’t you comparing to the price of buying an H100?
- deleted 3y ago[deleted]
- selectodude 3y agoAn H100 is about $30,000.
- MacsHeadroom 3y agoPlus a year of electricity. $21k/year is a steal.
- raxxorraxor 3y agoOr 2-3 A100. The H100 are allegedly more budget friendly, but I din't find that true on the "black market". And I think it draws a bit less power, but I have absolutely no experience about the software suites NVidia offers. I think the price is quite high to be honest, although probably realistic. Wouldn't expect competitors to be able to offer something better and data centers need to calculate for maybe newer models with even better performance.
- andrewstuart 3y agoThe outcome of Nvidia's monopoly hold on AI/GPU computing is that consumer level devices that might otherwise be perfectly effective for this sort of stuff are prevented by Nvidia from being used for such purposes. If there was real competition -like two or more other suppliers on par in terms of capability - then artificially constraining devices just to plump up prices would not be a thing.
- psshank 3y agoThis is what we've done at Salad (www.salad.com). SaladCloud (officially launching on July 11th) has 10k+ Nvidia GPUs at the lowest prices in the market. Not conducive for training but for serving inferences at massive scale.
- MacsHeadroom 3y agoYour pricing is around 50-100% higher than vast.ai, a relatively mature competitor in this space who offers the same GPUs and more with beefier rigs. For example. An '8 core/16GB RAM/3080 Ti 12GB' is 50% more expensive on salad than on Vast. In fact, I can get more than double the resources for the same cost as Salad in the form of a '28core/32GB RAM/2x3080 Ti 12GB' on Vast. So, what is your differentiation? Why would I ever use Salad?
- smoldesu 3y ago> consumer level devices that might otherwise be perfectly effective for this sort of stuff are prevented by Nvidia from being used for such purposes. Citation? To the contrary, millions of consumer-level Nvidia customers have access to datacenter-grade HPC APIs because of their vertical integration. Nvidia's "monopoly hold" on GPGPU compute exists because the other competitors (eg. AMD and Apple) completely abandoned OpenCL. When the time came to build a successor, neither company ante'd up. So now we're here. CUDA is not a monopoly. If Apple or Microsoft wanted, they could start translating CUDA calls into native instructions for their own hardware. They don't though, because it would be an investment that doesn't make sense for their customers, costs tens of millions of dollars, and wouldn't meaningfully hurt Nvidia unless it was Open Source.
- babypuncher 3y agoAm I a bad person for wishing the AI bubble would burst already so Nvidia would go back to allocating their capacity towards affordable consumer GPUs? ChatGPT is cool, but not cool enough for me to be OK with the ridiculous prices Nvidia can get away with on GPUs nowadays. Consumer GPU sales are at the lowest point they've been in decades, but Nvidia doesn't care. Their profits are higher than ever because everyone and their dog is paying out the nose to build up their AI nonsense.
- verall 3y agoUsed 1080 ti's are going for less then $200 right now and can do solid 1080p gaming and light 1440p gaming, if you don't care about raytracing and DLSS and etc. If you do care about those things, then you can pay for them by buying a 3000 or 4000 series card. They're a bit more expensive.
- babypuncher 3y agoI have a 3080, I paid MSRP ($699) for it in 2020 because I got lucky. It will last me a while, but I'm dreading seeing the 5080 next year and having to pay an MSRP that is twice that for the same class of product. I don't think Pascal cards will be relevant for AAA games much longer now that AAA games are starting to skip last-gen consoles. Ubisoft showed of their Avatar and Star Wars games this week, and both are shipping with ray-traced global illumination.
- pmoriarty 3y agoStrong demand for faster and more capable GPU's is the best thing that can happen for GPU prices in the long run. It'll incentive GPU development, and the more new/more-capable GPU models come out, the cheaper older/less-capable models will get. We've seen this cycle again and again with all sorts of computer hardware, from hard drives to monitors, CPU's, to memory, to graphics cards themselves. With some notable exceptions, the power and capacity of most hardware consumers can afford today dwarfs what was affordable a decade or two ago. The same will happen to GPUs if demand remains high. So bring it on!
- technick 3y agoI'm glad my cloud provider isn't listed, spot instances are already becoming hard to come by.
- mmaunder 3y agoIt’s hard to have a serious conversation about black swan events because you’re a fool until they happen. Anyway, if China invades Taiwan, which appears increasingly likely, the availability of GPUs will plummet. If this occurs in about 2 years, it will hit at peak AI (IMO) before supply chains have adapted to new extreme levels of demand. In fact peak demand may be a catalyst for invasion. Hoarding GPUs at this point may seem laughable, but having capacity during and immediately after an invasion will be a competitive advantage of note.
- piyh 3y agoI've only thought about this from the perspective of TMSC fab getting hit with cruise missiles, but I've never gone down the long term impacts of the trade freeze that would happen with China. Decades of globalization having to be torn down and reconstructed in a very, very short time period.
- andrewstuart 3y agoYou can be certain the US would not leave TSMC as anything but a pile of rubble if the CCP took Taiwan.
- andrewstuart 3y ago>> if China invades Taiwan, which appears increasingly likely, the availability of GPUs will plummet If China invades Taiwan, GPUs will be the least of our concerns. >> which appears increasingly likely It's not likely. China gains from the sabre rattling and politicking. They've seen what happens when a bully country attacks a smaller country. The leaders of the CCP have everything they could possibly want in this world. They won't risk it all for an uncertain outcome attacking Taiwan.
- KeplerBoy 3y agoWhy did Putin attack Ukraine instead of retiring on a superyacht? Why is he risking it all for what would, in a best-case scenario, become a decade long guerilla war against the ukraine resistance, even if russia managed to capture all of ukraine (which probably was the assumed/likely outcome as the war broke out). Clearly the incentives and motivations of people running oppressive regimes are slightly different.
- gm 3y agoYour price quotes for AWS support are wrong: Business support is 10% of your usage, or $100 USD whichever is higher. Enterprise support is 10% of your usage or $15k USD, whichever is higher. It's actually more complicated if your spend is high. Better point people to the info page here: https://aws.amazon.com/premiumsupport/plans/ https://aws.amazon.com/premiumsupport/plans/
- sytelus 3y agoLooks like there is no dirth of H100s anymore. So, is GPU scarcity finally gone?
- villgax 3y agoOblivius has A100s at $2.41/hr(80GB VRAM) too & no talking to sales/pre-approval crap
- dna_polymerase 3y agoWho are the companies renting out GPUs at scale? There is obviously GAFAM (which have their clouds basically), OpenAI and other startups with big contracts for said cloud services, but aside from them, which companies actually rent out dozens of GPUs to train their networks? And what do they bring to the table that makes them think they can archive better than OpenAI and GAFAM? Serious question. I think most companies without the top talent would do better using an API product.
- Tepix 3y agoCloud prices are very competitive, you can get 2x RTX 3090 located in Canada for $0.31 or so - here in Germany the cost for the electricity would be higher than that (0.36€ * 0.9 kW = 0.315€).
- mejutoco 3y agoI wonder how much the amount of data transfer costs (for training or finetuning) might be a big on top of that. I am assuming ingress costs are relatively cheap in most services (as in AWS).
- guisalberto 3y agoAdding another provider with availability for the H100 here, https://www.latitude.sh/accelerate/pricing https://www.latitude.sh/accelerate/pricing Instances from 1 to 8 x H100 GPUs, paid by the hour