25 ms·
Nvidia Hopper GPU Architecture and H100 Accelerator
- waynecochran 5y agoThis seems fast... TF32 ....... 1,000 TFLOPS (tensor core) FP64/FP32 ... 60 TFLOPS I am more interested in the 144-core Grace CPU Superchip. nVidia is getting into the CPU business...
- cjbgkagh 5y agoI think the 1PFLOPS figure for TF32 is with sparsity, which should be called out in the name. Maybe ‘TFS32’? I mainly use dense FP16 so the 1PFLOPS for that looks pretty good.
- lostmsu 5y agoAsked elsewhere, but why FP16 as opposed to BF16?
- cjbgkagh 5y agoI'm using older Turing GPUs BF16 would require Ampere. The weights in my models tend to be normalized so the fraction would be more important than the exponent so I would probably still use FP16. I would need to test it though.
- Melatonic 5y agoSame - plus its SUPER
- deleted 5y ago[deleted]
- macrolocal 5y ago50% sparsity and rated at 700W. The new DGX is 10kW!
- wmwmwm 5y agoI was recently researching how you'd host systems like this in a datacentre and was blown away to find out that you can cool 40kW in a single air cooled rack - this might be old news for many, but it was 2x or 3x what I expected! Glad I'm not paying the electricity bill :)
- jmole 5y agoHere's what a propane heater of similar output looks like: https://www.amazon.com/Dura-Heat-Propane-Forced-Heater/dp/B00427WQ36 https://www.amazon.com/Dura-Heat-Propane-Forced-Heater/dp/B0...
- HelloNurse 5y agoMost of the propane heater is a fan in a tube, the flame is probably quite smaller than a CPU package.
- baq 5y agoI've got an 8kW wood stove and that thing gets rather hot to touch - as in, you will get a blister... 40kW is a small city car worth of power.
- lIIIllllIIII 5y agoIf you think about it, cars can manage cooling 200+kW with a radiator - and a lot of airflow. That's still an amazing amount of power though. I can't help thinking about the kind of sear you could get on a steak with 40kW of power :D
- namibj 5y agoSupermicro will sell you 40 kW CPU TDP in a single air-cooled rack, not counting the rest of the servers.
- sliken 5y ago
- obeliskora 5y agoAnyone find details about the DPX instructions for dynamic programming?
- savant_penguin 5y ago1 petaflop on a chip?? What is the catch?
- dragontamer 5y agoTensor petaflops are useful in only very few circumstances. One of which is the highly lucrative deep learning community though.
- cjbgkagh 5y agoThe main tensor op is a matmul intrinsic which is useful for way more than just deep learning. Edit; many of these speeds are low precision which is less useful outside of deep learning, but the higher precision matmul ops in the tensor cores are still very fast and very useful for wide variety of tasks.
- dragontamer 5y ago> but the higher precision matmul ops in the tensor cores are still very fast and very useful for wide variety of tasks. The FP64 matrix-multiplication is only 60 TFlops, no where near the advertized 1000 TFlops. TF32 matrix-multiplication is a poorly named 16-bit operation.
- cjbgkagh 5y agoYou are indeed correct, I was (kinda) fooled by the marketing and I think that TF32 is deceptively named. I think the tensor cores are being used in this architecture for FP64 and 60 TFlops is still pretty decent. I'm on Turing architecture so I've never used TF32. I've only used FP32 and FP16 but FP32 isn't supported by these tensor cores.
- bcatanzaro 5y agoWell the addition is done in FP32, and it's a 32-bit storage format in memory, so calling it a 16-bit format isn't right either. It's really a hybrid format where everything is 32-bit except multiplication. Given that it's 32-bit in memory (so all your data structures are 32-bit) and also that in my experience using it is very transparent (I haven't run into any numerical issues compared to full FP32), I think calling it a 32-bit format is a reasonable compromise.
- nynx 5y agoSounds like we need some new training methods. If training could take place locally and asynchronously instead of globally through backpropagation, the amount of energy could probably be significantly reduced.
- zozbot234 5y agoThe principled way of doing this is via ensemble learning, combining the predictions of multiple separately-trained models. But perhaps there are ways of improving that by including "global" training as well, where the "separate" models are allowed to interact while limiting overall training costs.
- hwers 5y agoTrying to reduce energy consumption for ML like this is so silly.
- mlyle 5y agoTraining costs are growing exponentially bigger. The degree to which energy and capital costs can be optimized will determine how large they can go.
- thfuran 5y agoWhy?
- oblio 5y agoReducing energy consumption for computation is not silly. We're at a point we we're turning into a computation driven society and computation is becoming a globally relevant power consumption aspect. > global data centers likely consumed around 205 terawatt-hours (TWh) in 2018, or 1 percent of global electricity use And that's just data centers, if you add all client devices you probably double that. Plus that number will only continue to grow.
- jychang 5y agoThat's like a person driving the Model T in 1908 saying "trying to reduce gas efficiency is so silly". Why are people so dumb when it comes to planning for the future? Does it require a 1973 oil crisis to make people concerned about potential issues? Why can't people be preventative instead of reactive? Isn't the entire point of an engineer to optimize what they're building for the good of humanity?
- throw0101a 5y ago
- deleted 5y ago[deleted]
- komuher 5y ago1000 TFLOPS so i can run my GPT3 in under 100 ms locally :D If 1000 TFLOPS is possible to do in inference time then im speechless
- edf13 5y agoAt what costs I wonder?
- learndeeply 5y agoGPT-3 can't fit in 80GB of RAM.
- ml_hardware 5y agoAt inference time it will be possible to do 4000 TFLOPS using sparse FP8 :) But keep in mind the model won't fit on a single H100 (80GB) because it's 175B params, and ~90GB even with sparse FP8 model weights, and then more needed for live activation memory. So you'll still want atleast 2+ H100s to run inference, and more realistically you would rent a 8xH100 cloud instance. But yeah the latency will be insanely fast given how massive these models are!
- TOMDM 5y agoSo, we're about a 25-50% memory increase off of being able to run GPT3 on a single machine? Sounds doable in a generation or two.
- ml_hardware 5y ago
- algo_trader 5y agoFrom the nvidia page, > 80 billion transistors > Hopper H100 .. generational leap > 9x at-scale training performance over A100 > 30x LLM inference throughput > Transformer Engine .. speed .. 6x without losing accuracy So another monster chip - same size of the Apple M1-max thingy .. I guess it comes down to pricing. The A100 is already ridiculously expensive at $10K. They can this one at $50K and it would sell out?
- p1esk 5y agoYou can buy A100s in a server today, a number of integrators will happily sell it to you.
- chockchocschoir 5y agoAs someone who've tried for some weeks, it really seems like it's out-of-stock literally everywhere. The demand seems to be a lot higher than the supply at the moment, so much that I'm considering buying one myself instead of renting servers with it.
- p1esk 5y agoDid you check Lambda or Exxact?
- chockchocschoir 5y agoYes, nor Lambda Labs or Exxact Corporation have them available last time I checked (last week). Both citing high demand as the reason for it being unavailable.
- sabalaba 5y agoWe (Lambda) have all of the different NVIDIA GPUs in stock ---- can you send a message to sales@lambdalabs.com and check in again with your requirements? We're seeing a lot more stock these days as the supply chain crisis of 2021 comes to an end.
- minimaxir 5y agoSo the product naming for Nvidia's server-GPUs by compute power now goes: P100 -> V100 -> A100 -> H100 This is not confusing at all.
- gtirloni 5y agoIsn't it based on the architecture name? https://en.wikipedia.org/wiki/Category:Nvidia_microarchitectures https://en.wikipedia.org/wiki/Category:Nvidia_microarchitect...
- jsheard 5y agoYeah it is, but unless you've memorised the history of Nvidia architectures it doesn't tell you which is the newer one Fermi -> Kepler -> Maxwell -> Pascal -> Volta (HPC only) -> Turing -> Ampere -> Hopper (HPC only?) -> Lovelace?
- ksec 5y agoIsn't this the norm? Only AMD started the trend of naming the uArch with Numbers as Zen 4 or RDNA 3 fairly recently. With Intel it is Haswell > Broadwell > ..... Whatever Lake.
- jsheard 5y agoUsually the architecture name isn't the only distinguishing feature of the product name, you don't need to remember Intel codenames because a Core 12700 is obviously newer than a Core 11700 Nvidia's accelerators are just called <Architecture Letter>100 every time so if you don't remember the order of the letters it's not obvious They could have just named them P100, V200, A300 and H400 instead
- mywittyname 5y agoIntel was using Core i[3,5,7] names for multiple generations. A Core i7 could be faster or slower than a Core i5 depending on which generation each existed in. It is nice when products have a naming scheme where natural ordering of the name maps to performance.
- virtuallynathan 5y agoHow does a DGX Pod w/ the new 3.2Tbps per machine NVLINK switch compare to Tesla Dojo?
- TomVDB 5y agoWhen you take software support into account, probably very favorable. I don’t know anything about the state of Dojo, but Tesla was very hand wavy about their software stack during their presentation. And running AI algorithms efficiently on a piece of hardware is one of those things that many HW vendors have a hard time getting right.
- virtuallynathan 5y agoTesla Dojo Training tile (25x D1): 565 TF FP32 / 9 PF BF16/CFP8 / 11GB SRAM / 10kW NVIDIA DGX H100 (8x H100): 480 TF FP32 / 8 PF+ TF16 / 16 PF INT8 / 640GB HBM3 / 10kW Dojo off-chip BW: 16 TB/s / 36TB/s off-tile H100 off-chip BW: 3.9TB/s / 400GB/s off-DGX
- rafaelero 5y ago"Combined with the additional memory on H100 and the faster NVLink 4 I/O, and NVIDIA claims that a large cluster of GPUs can train a transformer up to 9x faster, which would bring down training times on today’s largest models down to a more reasonable period of time, and make even larger models more practical to tackle." Looking good.
- Melatonic 5y agoWe have needed wide use of NVlink or something like it for a long time now......heres to hoping mobo manufacturers actually widely implement it!
- learndeeply 5y agoThe open standard version of NVLink is CXL. They're available in latest gen CPUs.
- Melatonic 5y agoInteresting - I did not know that. Don't we also need motherboard manufacturers though to more widely implement the hardware required? It has been awhile since I have read about NVlink to be fair
- ml_hardware 5y agoThe 9x speedup is a bit inflated... it's measured at a reference point of ~8k GPUs, on a workload that the A100 cluster is particularly bad at. When measured at smaller #s of GPUs which are more realistic, the speedup is somewhere between 3.5x - 6x. See the GTC Keynote video at 38:50: https://youtu.be/39ubNuxnrK8?t=2330 https://youtu.be/39ubNuxnrK8?t=2330 Based on hardware specs alone, I think that training transformers with FP8 on H100 systems vs. FP16 on A100 systems should only be 3-4x faster. Definitely looking forward to external benchmarks over the coming months...
- deleted 5y ago[deleted]
- ortusdux 5y ago80 billion transistors boggles my mind. How many molecules are their per transistor?
- martini333 5y agowat
- Symmetry 5y agoIt's a crystal so just one molecule for all the transistors. In terms of atoms it's something on the order of the size of a 30 nm cube and with each silicon atom being .2nm in diameter something like 3 million, give or take an order of magnitude or two.
- ortusdux 5y agoThat makes sense. My mistake, I did mean atoms, not molecules. Wolfram alpha estimates 1.35 million Si atoms, so well within 1 order of magnitude. https://www.wolframalpha.com/input?i=30%5E3+cubic+nanometers+of+silicon+in+moles https://www.wolframalpha.com/input?i=30%5E3+cubic+nanometers...
- spoonjim 5y agoOff topic but I can't stand when corporations use actual people's names for their marketing who never gave them the permission to do so. For something like Shakespeare or Cicero I'm OK with it but Grace Hopper was alive in my lifetime, and even Tesla feels a little weird. What gives you the right to use that person's reputation to shill your product?
- eigenvalue 5y agoI generally agree with you, but in this case I suspect Grace Hopper would be honored by it and also impressed with the engineering here. It's not like they slapped her name on a soda can or something.
- spoonjim 5y agoThat’s not NVIDIA’s place to decide.
- paxys 5y ago> What gives you the right to use that person's reputation to shill your product? Practically speaking you have the right to do anything unless someone complains about it. A lot of popular figures, even those long dead, have estates and organizations that manage their likeliness and other related copyright and IP. IDK what the situation is in this case, but Nvidia may very well have paid for the name.
- cosmiccatnap 5y ago
- spoonjim 5y agoI don’t think my kids have any more right to use my name than a corporation, unless I specifically grant them that right (like Walt Disney did by naming it the Walt Disney company). Another sickening one is the Ed Lee Club in SF, who endorses political candidates under the name of a much-loved dead SF mayor.
- orangebeet 5y agoThis would be cool if they had decent drivers for Linux.
- paxys 5y agoI can assure you systems that take advantage of this chip for scientific/ML workloads aren't running Windows.
- flatiron 5y agothey may have edited their comment but they were commenting on the lack of quality of their Linux drivers (which I agree with but only on a consumer level, never used nvidia in a server)
- why_only_15 5y agoThey do have good drivers for Linux for the things this chips is intended to be used for (research, ML).
- johndough 5y agoIf you haven't had any issues with NVIDIA Linux drivers, you can count yourself extremely lucky. In the past, I had a 50/50 chance of boot failure after installing CUDA drivers over 12 different systems. Mainline Ubuntu drivers are somewhat stable, but installing a specific CUDA version from the official NVIDIA repos rarely works on the first try. Switching from Tensorflow to PyTorch has helped a lot though, as Tensorflow was much more picky about the installed CUDA version. Obligatory Linus Torvals on NVIDIA: https://www.youtube.com/watch?v=_36yNWw_07g https://www.youtube.com/watch?v=_36yNWw_07g
- captainbland 5y agoThose specs imply some pretty crazy architectural efficiency gains, massive theoretical compute performance per transistor compared to Ampere. It's all marketing numbers until the benchmarks are out, though. Edit: big TDP, though.
- ksec 5y agoAnd may be taking this opportunity to ask, what happen to Nvidia's leak? The hacker hasn't made any more news, and Nvidia hasn't provide an update either.
- quotemstr 5y agoThe simplest explanation is that Nvidia just paid up.
- deleted 5y ago[deleted]
- Melatonic 5y agoWhat was the leak?
- ksec 5y agohttps://news.ycombinator.com/item?id=30590752 https://news.ycombinator.com/item?id=30590752
- TomVDB 5y agoIn the keynote, Jensen made a sly remark about how they themselves could benefit a lot from one of their cyberthreat AI solutions.
- arvinsim 5y agoI wouldn't be surprised if they ended up "collaborating"
- Symmetry 5y agoNVidia and AMD datacenter GPUs continue to diverge between focusing on deep learning and traditional scientific computing respectively.
- shaklee3 5y agohow so? what is this bad at for scientific computing?
- ip26 5y agoFP16 performance is great, but the FP64 performance isn't terribly compelling. Scientific computing is generally FP64.
- shaklee3 5y agoIf you're comparing it to the MI250 you're comparing it to 2 separate chips on a single card. This is fundamentally different and unless you have an ideal workload or have optimized appropriately, it's not going to hit anywhere near the peak FLOPS if you have data movement between chips.
- YetAnotherNick 5y agoScientific computer scientists always prefer Nvidia because of CUDA and much better development experience on Nvidia tools.
- lmeyerov 5y agoSeeing the increased bandwidth is super exciting for a lot of business analytics cases we get into for IT/security/fraud/finance teams: imagine correlating across lots of event data from transactions, logs, ... . Every year, it just goes up! The big welcome surprise for us the secure virtualization. Outside of some limited 24/7 ML teams, we mostly see bursty multi-tenant scenarios for achieving cost-effective utilization. MiG etc static physical partitioning was interesting -- I can imagine cloud providers giving that -- but more dynamic & logical isolation, with more of a focus on namespace isolation, is more relevant to what we see. Once we get into federated learning, and further disintermediations around that, even more cool. Imagine bursting on 0.1-100 GPUs every 30s-20min. Amazing times!
- Rafuino 5y agoGood lord 700W TDP!
- trollied 5y agoLooking forward to seeing Doom run on this.
- deleted 5y ago[deleted]
- pixel_fcker 5y agoThe new block cluster shared memory and synchronisation stuff looks really really nice.
- gautamcgoel 5y agoThis chip is capable of 2000 INT8 Tensor TOPS, or 1000 F16 Tensor TFLOPS. In other words, it is capable of performing over a quadrillion operations per second. Absolutely insane... I still have fond memories of installing my first NVidia gaming GPU, with just 512MB of RAM, probably capable of much less than a single teraflop of compute.
- fancyfredbot 5y agoThe Tensor cores will be great for machine learning and the FP32/FP64 fantastic for HPC, but I'd be surprised if there were a lot of applications using both of these features at once. I wonder if there's room for a competitor to come in and sell another huge accelerator but with only one of these two features either at a lower price or with more performance? Perhaps the power density would be too high if everything was in use at once?
- TheGuyWhoCodes 5y agoJust look at Cerebras Wafer-Scale Engine it's basically a chip the size of a Wafer but it's not cheap...
- nine_k 5y agoCan different tenant VMs access different parts of H100 in parallel? If so, it may be a reasonable mix.
- fancyfredbot 5y agoRight now no - the unit of subdivision for MIG is too coarse. Everyone gets the same proportion of each function
- throwawaygh 5y ago> room for a competitor to come in and sell another huge accelerator but with only one of these two features either at a lower price or with more performance? They'd need fab capacity first. I wouldn't count on it any time soon, and chip gens have short lives.
- JonChesterfield 5y agoGraphcore's IPU is a machine learning variant on that. Power density seems to be ok. The CTO's talks (used to, I'm out of date) talk about dark silicon a lot. I share your suspicion that fp64 and ML workloads are distinct but can see each running on the same cluster at different times.
- patientplatypus 5y agoThe keynote is impressive (https://www.youtube.com/watch?v=39ubNuxnrK8 https://www.youtube.com/watch?v=39ubNuxnrK8). They can make an AI factory that will fit many times the current internet throughput in a small room. I'm super excited about what kinds of applications this could mean.