5 ms·
H100 GPUs Set Standard for Gen AI in Debut MLPerf Benchmark
- tikkun 3y agoIf you want to try one, some good options are: FluidStack or Lambda Labs If you need to rent a supercluster and you’re not tied to one of the big 3 clouds, then talk with FluidStack, Lambda, Oracle, maybe CoreWeave.
- rch 3y agoLambda seems like the easiest to work with at present.
- tikkun 3y agoI’ve used both and found both to work well for me. There are a variety of other gpu cloud options that aren’t good, though. I listed a bunch here, some are good, some seem to be worse on all dimensions (price, capacity, and UX) - https://gpus.llm-utils.org/alternative-gpu-clouds/ https://gpus.llm-utils.org/alternative-gpu-clouds/ Tl;dr of the good ones: FluidStack and Lambda for H100s (1x instances), Runpod for A100s.
- tiernano 3y agoI have heard these are 100k each... anyone know if that's correct? Guessing that's list, and no one pays list, but still...
- finexplained 3y agoI thought the H100s were ~30-40k each. But they're not widely available and you usually buy multiple boxes from vendors that also come with expensive CPUs/RAM etc.
- aiappreciator 3y agoH100s are intended to be retailed piecemeal. The big enterprise model is DGX GH200, at $10 mil each. Its just that current supply is extremely short, so the H100s end up only available to big buyers. But that will be resolved in time. Nvidia wants every university lab to have a H100 so no competitor sneaks in there.
- tikkun 3y agoI'm writing a cool mega post on this at the moment – not published yet but here's the excerpt on pricing: How much do these GPUs cost? H100s are around $30-33k at the IT hardware resellers CDW and SHI (https://www.cdw.com/product/nvidia-h100-gpu-computing-processor-nvidia-h100-tensor-core-80-gb/7367181 https://www.cdw.com/product/nvidia-h100-gpu-computing-proces..., https://www.shi.com/product/45671009/NVIDIA-H100-GPU-computing-processor https://www.shi.com/product/45671009/NVIDIA-H100-GPU-computi...) Supermicro’s HGX H100 8x GPU server is $297k at the reseller Dihuni (https://www.dihuni.com/product/supermicro-8125gs-tnhr-server/ https://www.dihuni.com/product/supermicro-8125gs-tnhr-server...) DGX H100 is $521k at the reseller Insight (https://www.insight.com/en_US/shop/product/DGXH-G640F+P2CMI60/NVIDIA/DGXH-G640F+P2CMI60/NVIDIA-DGX-H100-P4387-SYSTEM-640GB-FULL-STD-SUPPORT-5-YRS/ https://www.insight.com/en_US/shop/product/DGXH-G640F+P2CMI6...) The DGX GH200 might cost in the range of $10mm-20mm or more (A guesstimate ballpark from an exec at a cloud company I talked with) If anyone wants to pre-review the post and can offer thoughtful comments, my email's in my profile. And to clarify the difference between all of these product names, I put together this diagram - https://gpus.llm-utils.org/dgx-gh200-vs-gh200-vs-h100/ https://gpus.llm-utils.org/dgx-gh200-vs-gh200-vs-h100/. I don't have the HGX H100 or the DGX H100 on there though - but the HGX H100 is a reference platform for OEMs to design and make H100 based servers with either 4x H100s or 8x H100s (https://nvdam.widen.net/s/5kgbjq2v2t/hpc-hgx-h100-datasheet-nvidia-web https://nvdam.widen.net/s/5kgbjq2v2t/hpc-hgx-h100-datasheet-...) and the DGX H100 is the official Nvidia server with 8x H100s (https://resources.nvidia.com/en-us-dgx-systems/ai-enterprise-dgx https://resources.nvidia.com/en-us-dgx-systems/ai-enterprise...).
- huac 3y agofrom somebody who bought 2500 of them: https://news.ycombinator.com/item?id=36313960 https://news.ycombinator.com/item?id=36313960
- ed 3y agoThese could train GPT-3 in 46 hours. (Edited from "11 mins" per https://news.ycombinator.com/item?id=36500154 https://news.ycombinator.com/item?id=36500154 ) An equivalent number of V100's (GPT-3's original GPU) would've taken about 36 days [0]. 0 – https://www.reddit.com/r/GPT3/comments/p1xf10/comment/h8h3sl4/?utm_source=reddit&utm_medium=web2x&context=3 https://www.reddit.com/r/GPT3/comments/p1xf10/comment/h8h3sl...
- messe 3y ago3500 of them trained GPT-3 in 11 minutes. That number is still worth remarking.
- reaperman 3y agoIt’s remarkable that one of these could train GPT-3 in under a month. However, good to note that its the price is that of twenty 4090’s.
- ClassyJacket 3y agoCould it actually? Or do you need the memory size of thousands of them running in parallel?
- reaperman 3y agoYes I worded that extremely poorly.
- cavisne 3y agoOne interesting thing is if the model can’t fit into GPU memory (sharded across multiple chips) it would be much slower. So one chip would take more than a month. That’s why these clusters are so valuable, even with Nvidias margins they are still cheaper than using less compute for longer.
- visarga 3y agotitle says: "a massive GPT-3-based benchmark" that means they might pass a few batches through a GPT-3 sized random init network and time it
- hinkley 3y agoI am keeping my fingers crossed that there will be a 'Bloom Filter' moment for AI where we create an algorithm that can eliminate an expensive calculation in 99% of interactions and give us a 10x speedup in some larger problem domain. Or some other 'trust but verify' situation where the suggested action is still validated against business rules that have some notion of consumer protection.
- usaar333 3y agoDropping numerical precision seems to have done that to some degree?
- hinkley 3y agoYou have a deterministic AI program to show me?
- dsign 3y agoI think the parent is referring to the fact that LLMs and generative models are, for the time being, “creating new markets” and “creating new problems”, but not really solving any of the hard problems we already have. I see it this way: you have this incredible piece of technology. You can use it to take the scraps away from a bunch of starving artists, and convert that survival income in wealth for some entrepreneur. Or you can use it to cure cancer or aging or make fossil-free airplanes. At the moment, the first application seems to be the one moving the market, and that’s really sad. On the side of hope, if somebody were to demonstrate how to use a bunch of GPUs to make the e-coli bacteria live for a little longer using an approach that has a semblance of generality, I guarantee that a lot of old people would be willing to part with their fortunes in exchange for a sliver of hope for them or future generations.
- soultrees 3y agoThat’s only because the companies with marketing budgets and marketing teams are the ones you are implying. There is serious research going down right now, but you don’t see you typical medical research department at your university buying ads on Instagram.
- SushiHippie 3y agoDoes anyone know how much kWh was used (or could have probably been used) in these 11 minutes?
- aiappreciator 3y agoH100s consume like 350W. 3000 H100s = 350kW 350*0.2h = 70kWh, about $20 at $0.3/kWh (lets assume data centers have expensive electrical costs).
- SushiHippie 3y agoAh Thanks! Taking the data from here[0] for the USA, this would be about 25,69kg of CO2. [0] https://ourworldindata.org/grapher/carbon-intensity-electricity https://ourworldindata.org/grapher/carbon-intensity-electric...
- nl 3y agoIf you are on GCP you can choose low carbon data centers: https://cloud.google.com/sustainability/region-carbon https://cloud.google.com/sustainability/region-carbon
- alwa 3y agoWhich, per the EPA [0], is roughly the tailpipe CO2 you’d emit by burning a little under 3 gallons of gas in your car, right? Which is to say, driving 65ish miles in the US [1]? It amazes me that computational feats of this magnitude can be so energy efficient in the scheme of things. [0] https://www.epa.gov/greenvehicles/tailpipe-greenhouse-gas-emissions-typical-passenger-vehicle#burning https://www.epa.gov/greenvehicles/tailpipe-greenhouse-gas-em... [1] https://www.epa.gov/greenvehicles/tailpipe-greenhouse-gas-emissions-typical-passenger-vehicle#:~:text=This%20assumes%20the%20average%20gasoline,around%2011%2C500%20miles%20per%20year https://www.epa.gov/greenvehicles/tailpipe-greenhouse-gas-em....
- NhanH 3y agoIt’s not that surprising actually, computation moves nothing and all energy has to turn into heat. If computation would use as much energy as moving physical objects, the heat would burn everything down
- deviantbit 3y agoSkyNet awakens... then says, "I'll be back." Goes to sleep to be trained again. (ominous music follows)... Across the screen it says, "Sarah Connor is now in grade school." Then blinks away. A moment later, "John Connor has yet to be conceived." (More ominous music) Image of Jensen Huang in a terminator leather jacket standing hold the next generation GPU is shown. (More ominous music that peaks to a finale) Will you join the resistance? LOL, I welcome our new overload.
- Electricniko 3y agoIs our new overlord Jensen Huang or Skynet? I prefer one over the other.
- deviantbit 3y agoYes.
- Karupan 3y agoHas AMD completely missed the ML/AI train? I’m quite surprised that even for inference, there doesn’t seem to be a viable competitor to nvidia. Is there anything in AMD’s roadmap to suggest they are planning to even compete?
- cavisne 3y agoThe "even for inference" thing has turned into a bit of a trap imo. Data parallel models scaled up for training and then could run on individual chips, but these massive model parallel models require a couple of chips directly linked together even to do inference. So the idea that a competitor could come in with a simple, cheap inference chip doesn't really work.
- jiggawatts 3y agoTheir upcoming AMD MI300 APU looks very competitive with NVIDIA's GH200. The problem remains the lack of software support.
- rapsey 3y agoAMD is bad at software and that is not changing so they will always be behind.
- dsign 3y agoI agree with this. Even their GPU drivers for gaming are bad, they get stuck at “basic” things like MPEG encoding. But I don’t understand why software is a problem for them with their deep pockets. It can’t possibly be dearth of talent, or that it is expensive. Here in Europe a good software engineer earns half of what a mediocre software engineer earns in USA, to say nothing of India or China. They could just hire a bunch of teams and up their software game.
- rapsey 3y agoHardware development is very different to software development. AMD is a hardware company and they probably run their software development badly.
- mrcggl 3y agoA large H100 cluster (>10k GPUs) could likely train a LLM with 10x compute (FP8) of GPT-4, which was apparently trained on a mix of A100s and V100s.
- ollin 3y agoThere's a second blog post here with substantially more technical detail: https://developer.nvidia.com/blog/breaking-mlperf-training-records-with-nvidia-h100-gpus/ https://developer.nvidia.com/blog/breaking-mlperf-training-r... Additionally, code for the actual submission is available here https://github.com/mlcommons/training_results_v3.0/tree/main/NVIDIA_CoreWeave/benchmarks/gpt3/implementations/pytorch https://github.com/mlcommons/training_results_v3.0/tree/main...