7 ms·
Nvidia H200 Tensor Core GPU
- deadballcretin 3y agoThe performance jumps that Nvidia has had in a fairly short amount of time is impressive, but I can't help but feel like there is a real need for another player in this space. Hopefully AMD can challenge this supremacy soon.
- 2OEH8eoCRo0 3y agoI'd prefer another player that doesn't rely on TSMC.
- 01100011 3y agoNot sure why you were downvoted. Taiwan is in a precarious position and diversifying manufacturing away from them makes sense.
- xethos 3y ago> diversifying manufacturing away from them Diversifying manufacturing away from Taiwan makes their position more precarious, not less.
- notact 3y agoBoth parent comments were likely referring to any entity other than Taiwan. If you are a fabless chip designer or one of their customers, it makes sense to diversify away from Taiwan, even if that comes at Taiwan's expense.
- 2OEH8eoCRo0 3y agoI worry that a lot of large cap companies either depend directly on TSMC (Nvidia, AMD, Apple) or depend on a company that depends on TSMC (Microsoft/OpenAI, Arm). It's TSMC all the way down and that scares me. I never thought I'd root for Intel.
- signatoremo 3y agoIt’s easy to forget that TSMC was not the chipmaking leader until 7 or 8 years ago. Intel was. Things can change quickly in tech. Intel is trying to reclaim their old glory. We’ll see if they succeed.
- ls612 3y agoHistorians might look back at Intel’s and GloFo’s stumbles in the mid 2010s as being a pivotal turning point. Chips are the new oil and that makes the mid term future very dangerous.
- sofixa 3y agoDoesn't that leave pretty much Samsung and Intel as the only options?
- brucethemoose2 3y agoAnd Nvidia has used Samsung before.
- astrodust 3y agoSo basically Samsung.
- signatoremo 3y agoNvidia has been test driving Intel’s foundry services - [0] Intel is on track with their node rollout roadmap, according to their CEO - [1] [0] - https://www.tomshardware.com/news/nvidia-ceo-intel-test-chip-results-for-next-gen-process-look-good https://www.tomshardware.com/news/nvidia-ceo-intel-test-chip... [1] - https://focustaiwan.tw/sci-tech/202311070017 https://focustaiwan.tw/sci-tech/202311070017
- evanjrowley 3y agoMaybe IBM, if the stars align: https://news.ycombinator.com/item?id=38256558 https://news.ycombinator.com/item?id=38256558
- brucethemoose2 3y agoOr even just offer an alternative, along with Intel: https://www.servethehome.com/intel-shows-gpu-max-1550-performance-and-gaudi3-ai-updates-at-sc23/ https://www.servethehome.com/intel-shows-gpu-max-1550-perfor... There aren't many many Gaudi/Instinct cloud offerings even though the market is accelerator starved.
- singhrac 3y agoYou can use Gaudi2s at the new Intel Developer Cloud[0]. Not sure why don't offer it on AWS though, seems a bit odd since they have the DL1 instances for the first-gen Gaudis. [0]: https://developer.habana.ai/intel-developer-cloud/ https://developer.habana.ai/intel-developer-cloud/
- brucethemoose2 3y agoInteresting, this looks like what I might want: https://eduand-alvarez.medium.com/llama2-fine-tuning-with-low-rank-adaptations-lora-on-gaudi-2-processors-52cf1ee6ce11 https://eduand-alvarez.medium.com/llama2-fine-tuning-with-lo...
- deleted 3y ago[deleted]
- meragrin_ 3y agoI'd rather Intel. People have been pleading with AMD for years to compete with Nvidia, but AMD really has not put in a proper effort. They still don't look like they are putting in a proper effort.
- JonChesterfield 3y agoAMD shipped Frontier. Compare and contrast with Intel's Aurora. Epyc took the performance crown from Intel. Games consoles have been AMD for ages. AMD are competing with Intel and Nvidia simultaneously with fewer resources than either, having come back from near bankruptcy in recent memory. There's been plenty of effort and execution from team red. It's commercially unfortunate that the crypto and now deep learning crowd don't particularly value the flexibility or control that comes from an open source toolchain. Regardless, I don't think the Cuda moat will hold out.
- sangnoir 3y agoAMD was fighting Intel for its life. After a number of flops, it only got a big breakthrough on the CPU-side with Zen just over half a decade ago - which is not that far back. Hopefully they now have a bit of money saved up in their war chest to help the GPU division.
- viewtransform 3y ago<They still don't look like they are putting in a proper effort.> Quite the contrary, they've turned around the company to focus on AI. Legacy software projects are on hold and software developers moved to work on AI under a new VP (former Xilinx exec). They have purchased some startups to get experienced AI developers. Here is Andrew Ng giving a positive evaluation of AMD's software efforts https://youtu.be/KDBq0GqKpqA?t=2359 https://youtu.be/KDBq0GqKpqA?t=2359
- brucethemoose2 3y agoThe H200 GPU die is the same as the H100, but its using a full set of faster 24GB memory stacks: https://www.anandtech.com/show/21136/nvidia-at-sc23-h200-accelerator-with-hbm3e-and-jupiter-supercomputer-for-2024 https://www.anandtech.com/show/21136/nvidia-at-sc23-h200-acc... This is an H100 141GB, not new silicon like the Nvidia page might lead one to believe.
- latchkey 3y agoWhat happened to the H100 NVL? https://www.anandtech.com/show/18780/nvidia-announces-h100-nvl-max-memory-server-card-for-large-language-models https://www.anandtech.com/show/18780/nvidia-announces-h100-n...
- brucethemoose2 3y agoI dunno. But thats a dual GPU product, so its not really 180GB.
- jauntywundrkind 3y agoThis is a single-chip H100 NVL. Both are GH100's with the same tweaked 20% wider 6144-bit HBM3e (versus 5120 bit on other H100's) running at a higher speed. The HBM3e loadout is slightly different than H100 NVL's was going to be, but this definitely seems like a higher bin H100. It's basically as-if AMD had shipped a 7900 XT, then latter started selling the 7900 XTX; same chip, but they brought up all the memory controllers on this one.
- sberens 3y agoWhere does the H200 fit in if the B100 is coming out the same year with 2x the performance? Is the H200 just cheaper than the B100?
- brucethemoose2 3y agoIts a different production line. They can keep producing both since they are both in demand anyway. And the B100 is farther away. Nvidia always doubles the memory of their cards like this mid generation.
- bluedino 3y agoDoes the L40S fit in a similar way? Most of the GPU's are backordered bigtime but our vendors are chomping at the bit to sell us these.
- Mistletoe 3y agoCan anyone explain to a layman what exactly I'm looking at in that picture? It looks like a neat little city or building from Bladerunner.
- brucethemoose2 3y agoIt's a server motherboard with 8 GPUs crammed on it, facing up. The tall towers are the GPU heatsinks. I believe the blade looking things on the side are CPU RAM, the heatsinks on the back are covering the CPUs, and the little heatsink in the middle must be the CPU VRMs. Fans are in the back, and they crammed some electrical components on the front where all the IO is.
- formerly_proven 3y agoLooks like an HGX drawer, so there’s only GPUs on this. The heatsinks towards the front are probably on NVLink switches.
- brucethemoose2 3y agoAh you are right.
- iszomer 3y agoJust fyi, if 1 of 8 of the GPU's fail, you will be replacing the entire assembly; they are not modular.
- NoMoreNicksLeft 3y agoAm I the only one that's annoyed by the non-alphabetical model numbers? Why not do B100 after the A100, then jump to H (supposing there won't be a C100 or D200 at some point)? Like, wtf Nvidia.
- robin_reala 3y agoAt least they haven’t tried to do a Tesla S, 3, X, Y progression.
- brucethemoose2 3y agoThey name their architectures after scientists (Maxwell, Pascal, Turing, Volta, Ampere, Lovelace, Hopper). Thats what the GPU initial stands for. As for the number, the die name counts down to 100 (with GA107, for instance, being a small GPU die and GA100 being the big one), and the big datacenter GPU as a product inherits the 100.
- semi 3y agoit'd be nice if they picked them in alphabetical order
- CooCooCaCha 3y ago
- wolframhempel 3y agoI'm curious: Do you think there is a realistic chance for another chip maker to catch up and overtake NVidia in the AI space in the next few years or is their lead and expertise insurmountable at this point?
- chaxor 3y agoI don't think that type of question or logic applies when predicting stock markets.
- edgyquant 3y agoLuckily no one is trying to predict a stock market here
- latchkey 3y agoAMD is trying. https://seekingalpha.com/article/4650521-amd-set-to-deliver-a-strong-product-to-ai-market-with-mi300 https://seekingalpha.com/article/4650521-amd-set-to-deliver-...
- dhruvdh 3y agoThis is launched in response to MI300X, and this should still not be enough to match AMD's product. This launches 2 quarters after MI300X, but B100 should arrive before AMD's MI400 generation.
- MikeKusold 3y agoI thought CUDA was NVIDIA’s moat. Is that no longer the case, or did AMD come up with a good alternative?
- zozbot234 3y agoCUDA code can be forward-ported to AMD's HIP, which can be used with the ROCm stack. For a more standards-focused alternative there's also SYCL, which has implementations targeting a variety of hardware backends (including HIP) and may also target Vulkan Compute in the future.
- bearjaws 3y ago"GPU" - zero video output capabilities built in.
- aceazzameen 3y agoAIPU?
- hencoappel 3y agoNPU is the term normally used
- zeusk 3y agoIt can still process graphics, you just need to do a cross-adapter scanout or encode it for transmission over network.
- brucethemoose2 3y agoCan it? I thought that capability ended with the A100. It still has a media encode/decode blocks. A big one, in fact.
- christkv 3y agoIs the limit on the speed on inference a memory bandwidth issue or compute?
- thatguysaguy 3y agoMemory bandwidth/latency, especially when you're at smaller batch sizes.
- huac 3y ago"it depends" https://kipp.ly/transformer-inference-arithmetic/ https://kipp.ly/transformer-inference-arithmetic/
- brucethemoose2 3y agoDepends. One might say its sometimes "cache size limited" too.
- gosub100 3y agoWhy do they still sell hardware now that practically every other business has moved to being a service provider? If we set aside the fact that it would be an awful move for end-users, what's to stop Nvidia from cornering the market by only renting them in their own data centers? Is it the logistics of moving the massive training sets?
- constantly 3y agoWhat do you think all the other service providers are running their services on?
- gosub100 3y agoI'm asking why Nvidia doesn't maximize their profits by retaining the hardware and selling compute. They could capture the market from those other providers if they sold more FLOPS/kilowatt (or whatever metric is used). Compared to manufacturing GPUs/TPUs, running a datacenter (especially one that specializes in Nvidia hw) would seem to be a trivial task.
- michaelt 3y agoGoogle Cloud Platform hasn't managed to make much of a dent in AWS's business, despite being the only place you can get 'TPUs' and 'bigquery'. Becoming a successful cloud provider is far from trivial, even if you can offer technology no-one else has.
- 10000truths 3y agoI surmise that such a strategy would essentially hand their market share over to AMD on a silver platter.
- jsnell 3y agoThat would be a highly risky bet on Nvidia becoming AWS faster than AWS can become Nvidia. What they're doing is instead trying to make sure that their GPUs continue to be seen as the best option in the short/medium term (by having them accessible everywhere), and trying to commoditize their complement by giving small cloud providers disproportionate GPU allocations, which they hope will drive customers from the big providers to the smaller ones that a) aren't trying to build their own ML hardware, b) will have less negotiating leverage with Nvidia in the long term.
- schrodingerscow 3y agoThis may be a naive question, but all the metrics seem to be for inference. Should we expect similar gains on training?
- p1esk 3y agoYes. Training would especially benefit from the increased memory size.
- schrodingerscow 3y agoInteresting thanks. I wonder why they aren’t marketing that more on this page that seems important
- mtw 3y agoI had a shock when I looked up prices for H100 gpus, wanting to use one just for personal experimentation and for an upcoming hackathon. How much this one costs? $300,000?
- nacs 3y agoThese are not for consumers -- these are datacenter-grade systems. If you want a consumer GPU, you can go for the RTX 4090 (24GB VRAM) or the A6000 Ada (48GB VRAM) if you are building a workstation. If you really need to "experiment" on an A/H100, then you can rent it by the hour through a cloud provider like Runpod.
- singhrac 3y agoTo elaborate: you can't really buy these except in specific configurations from Supermicro (usually 8x H100) or the like. So take whatever chip-specific cost you have in mind, and 8x it, and add on the cost of CPU/memory/storage. NVIDIA doesn't bother to sell these in a configuration that you can plug into your desktop.
- nojvek 3y agoWith cookies banner and ad banner, the page has barely 1/4th of screen space on mobile device.