8 ms·
AMD MI300 performance – Faster than H100, but how much?
- gigatexal 3y agoBut without CUDA it’s gotta be DOA no? CUDA is what makes Nvidia’s hardware valuable. AMD should finance a competitor. They’ve had strong hardware but it’s the software that’s hurting them from what I’ve seen.
- wmf 3y agoAMD is now starting to have decent software support.
- JonChesterfield 3y agoIt's somewhat sad that AMD bet on open source over proprietary and then approximately zero people stepped up to help. Easier to use cuda and complain I suppose. Fortunately they've stuck with the open source plan anyway and just written it themselves.
- wmf 3y agoOpen source people would rather spend their time reverse engineering Nvidia, Mali, and Apple GPUs than helping AMD.
- striking 3y agoLook, I love AMD, but there are different groups of open source people here. One group is trying to broaden the support for desktop Linux across all hardware, and the other group is trying to crunch numbers with GPUs and build support for that. There isn't a super strong overlap, and I think the former group is currently larger than the latter.
- ahartmetz 3y agoAMD's paid linux driver team is moving relatively quickly using some documentation and tools (simulators) under NDA. It is difficult to contribute usefully to the core of that effort without getting hired by AMD. AFAIK, that is why not many hobbyists are contributing.
- JonChesterfield 3y agoI would guess the documentation is something of a hindrance. I'm in compilers rather than drivers, where the public docs are essentially the pdfs at https://gpuopen.com/amd-isa-documentation/ https://gpuopen.com/amd-isa-documentation/. It's a lot better than the nvidia equivalent but it would also be rough going implementing a compiler based on those alone; too many instructions with names but unspecified semantics. The driver / firmware story is probably similar.
- ahartmetz 3y agoOh, I thought I was missing something, maybe another document with more explanations. Thanks for clearing that up. I was actually motivated to contribute to the AMD drivers at some point and did land a couple of patches, but wasn't looking to switch career paths. The drivers have become quite good without me anyway, so no regrets :)
- kcb 3y agoThe highest performing open source graphics driver available anywhere is RADV.
- deaddodo 3y agoThey have a competitor, it's just less popular than CUDA: https://www.amd.com/en/products/software/rocm.html https://www.amd.com/en/products/software/rocm.html Before that, they were pushing heavily for standard OpenCL, but that failed because the hardware wasn't as competitive and the ecosystem/tooling barren.
- atemerev 3y agoROCm is not just unpopular, it is so boilerplate-ridden, it is nearly unusable. It takes about 5x more code to do things there compared to CUDA. Yes, “just use libraries”. But as Andrej Karpathy used to say, “I don't need some library holding my hand and providing abstractions. Real men command GPUs with their own raw kernel code”. (And, incidentally, there are much less libraries for ROCm, for this reason. Somebody should write them, and why do that with something that takes 5x more code to do the same thing?)
- kcb 3y agoWhy? I understand the framework setup may not be as polished as CUDA, but I was under the impression HIP is the primary kernel language supported by ROCm and at first glance it's basically a 1:1 CUDA clone.
- atemerev 3y agoHm, apparently, in the last years, HIP became much integrated into ROCm as the principal way of doing things. I was referring to the situation as of 2-3 years ago, when plain ROCm was a chore to write. OK, I retract my statement.
- JonChesterfield 3y agoWhere do you get the 5x code idea from? Hip and cuda are almost regex-equivalent, openmp is the same language on each.
- atemerev 3y ago
- screye 3y agoRocm will only get adoption when Cuda code can be ported over with 1 click. At the very least, pytorch has to work effortlessly across GPUs, down to the less popular and odd workarounds people build into their pyotrch code. Even if RocM reaches parity, the first mover advantage for Cuda is too large. RocM porting has to be literally effortless. Everything else is DOA.
- screye 3y agoEdit- That being said, if you a cloud provider and need to scale up a bunch of basically similar transformers models, then it should be an easy sell for AMD.
- wmf 3y agoThe facts don't bear this out. Various companies have already ported to AMD even though it isn't effortless.
- com2kid 3y agoCounterpoint, at scale a month of training a GPU cost more than an engineer now days. And the large companies (Meta, Microsoft) buy these GPUs by the thousands upon thousands. Having a small team of engineers spend a couple months porting code over is well worth it for even modest cost reductions. These may not be useful to smaller companies working in the AI space, but odds are to sellout, all AMD really needs are the sales contracts that they've already announced.
- Der_Einzige 3y agoPeople here don't want to hear the truth, but this is the truth.
- trynumber9 3y agoDepends on your definition of DOA. For the big customers who can't get as many H100 as they want it'll be an option. They have the resources to make sure their workload does in fact work and then scale out hardware appropriately. And it might not even be a downgrade: MI300X has more bandwidth and larger shared memory space. And there's MI300A which has a combined CPU+GPU with shared memory space which already has supercomputer customers. You can say it's DOA relative to Hopper but MI300 will be AMD's most successful GPGPU yet.
- bratao 3y agoI don´t think so. There are few algorithms that cover the majority of use. For example in NLP a fast LLM (such as LLAMA) training/prediction and support for Hugging face transformers (BERT and such) would cover 95%+ of NLP use-cases that need a GPU.
- kemotep 3y ago> AMD should finance a competitor. If they can’t afford to out R&D Nvidia today, how could they afford to fund a competitor? In the past 20 years the only serious new entrant to the discrete GPU market has been Intel and they are also far behind Nvidia and CUDA.
- lanza 3y agoPyTorch and Triton don't use CUDA. An extraordinary volume of big tech compute only cares about the PTX “isa” for NVIDIA.
- nharada 3y agoOne interesting consequence of the model consolidation happening in the LLM world is that it's actually easier for AMD's hardware to be competitive. It used to be that you had to support a ton of ops since nobody is going to use your GPU for just a single model. Nowadays, most models are just transformers with only a few operations inside, albeit repeated many times over. On top of that, you can build a reasonable application or business using only one or two LLMs (i.e. LLaMA 2 + Whisper, etc). So as long as they can get those running fast, they can sell GPUs to people who don't need the flexibility and breadth of CUDA.
- KRAKRISMOTT 3y agoThe mighty dot product and matrix multiplication strikes again!
- oivey 3y agoI’m not sure this is true. Nvidia ships hardware today that is specialized for transformers, and presumably AMD needs something similar on their side. An arms race of specific hardware like that is probably not to their advantage. It makes the interoperability story even harder. They still need a usable GPGPU language to glue all that hardware together, too.
- dhruvdh 3y agoThere is nothing special about Nvidia hardware outside of them having significantly better lower precision FLOPs before anyone else. Their marketing chose to call this a "transformer engine" but there is no actual "engine" in silicon. Like AMD announced today, and is shown in the article we're commenting on, AMD now is 1.3 times faster at FP8, INT8 flops with and without sparsity, meaning their "transformer engine" is 1.3 times faster. The rest of your comment doesn't make sense. See the other comments on software in this thread.
- ilaksh 3y agoThe idea of model architecture making fast hardware design easier is what makes https://github.com/tinygrad/tinygrad https://github.com/tinygrad/tinygrad so interesting.
- latchkey 3y agoToday's MI300x announcement. Worth a watch before you comment about how their software sucks. https://www.youtube.com/watch?v=tfSZqjxsr0M https://www.youtube.com/watch?v=tfSZqjxsr0M
- VHRanger 3y agoKoduri left so they have a chance to make the software not suck anymore
- dhruvdh 3y agoI'll add some highlights for those who don't have the time - Flash Attention 2 kernels, Paged Attention kernels, PyTorch, vLLM, transformers, ONNX runtime support, OpenAI triton support in v3, tensorflow, JAX, OpenXLA. Also, while this wasn't in the presentation, CUDA can be translated to HIP using automated tools. and - CuPBoPAMD is a CUDA translator that translates CUDA programs at NVVM IR level to HIP-compatible IR that can run on AMD GPUs (https://dl.acm.org/doi/pdf/10.1145/3624062.3624185 https://dl.acm.org/doi/pdf/10.1145/3624062.3624185)
- latchkey 3y ago> CUDA can be translated to HIP using automated tools. Not just automated, but also open source. https://github.com/ROCm-Developer-Tools/HIPIFY https://github.com/ROCm-Developer-Tools/HIPIFY
- superkuh 3y agoOpen source, but only covering some AMD hardware for 4 years. Don't trust the support to last.
- slavik81 3y agoOne of my coworkers uses a Vega 56 in their local workstation. That card is six years old. It's not officially supported, but it still works. I have no idea if the AI frameworks work on hardware that old, but the math libraries still do.
- bastard_op 3y agoOk, so even if the AMD chips are equal to or faster, Nvidia still has a large advantage with their networking using their proprietary NVLink switching that was most certainly borne from the Mellanox Infiniband IP acquisition. AMD is going to have to rely on more traditional PCI-E based networking for all off-system traffic via Infiniband or Converged Ethernet vs the more optimized Nvidia GPU clustering that uses it's on proprietary super-low latency NVLink-based switching. https://www.nvidia.com/en-us/data-center/nvlink/ https://www.nvidia.com/en-us/data-center/nvlink/ I'm super curious to see how their networking results compare between transports when real testing results between large supercomputer-ish farms get out there.
- latchkey 3y agoYou obviously didn't watch the announcement. They are doubling down on ethernet and open standards.
- dhruvdh 3y agoAMD uses Infinity Fabric, which is their NVLink. They announced today how they're opening up Infinity Fabric to select partners, including Broadcom. Meaning partners can now develop their own switches. Also, as AMD and HPE power some of the most performant supercomputers in the world, they wouldn't have won those contracts if their networking was subpar. Those use slingshot. You may also be interested in reading about https://ultraethernet.org/ https://ultraethernet.org/ .
- saberience 3y agoNVLink is used for linking chips together, InfiniBand is used for linking systems together, i.e. into big HPC clusters. Infinity Fabric is used for inter chip communication, it's not a replacement for Infiniband and cannot be used to create clusters of machines.
- viewtransform 3y agoI queued for you the part where Forrest Norad explains all this in today's presentation including a discussion on ultrafast ethernet. https://youtu.be/tfSZqjxsr0M?t=5198 https://youtu.be/tfSZqjxsr0M?t=5198
- xnx 3y agoNo mention of price anywhere that I've seen. I assume it will only be a slight $/flop discount vs. Nvidia.
- latchkey 3y agoPricing at this point is almost irrelevant given that it is available in Q1/Q2 2024 and H100's are sold out until who knows when and you're still going to be at the bottom of a long line unless you have a giant order to place. Advantages, like IB, are 50+ week lead times as well, if you're lucky.
- zitterbewegung 3y agoIs this a drop in replacement for clouds (which is the actual question to ask because if you are buying this you have to make new infrastructure) the answer is No. Will Microsoft and Oracle allow you to use this to rent ? Yes. Does Microsoft and Oracle need more accelerators to do internal business processes? Absolutely (Microsoft announced it would rent AI accelerators because they can’t keep up with their own internal (and maybe external) demand). So cost has to not just be per card but infrastructure build out . Not sure when Microsoft will just have their own TPU accelerator but I wouldn’t be surprised if the announce it sometime.
- Der_Einzige 3y agoCan you cite your claims about Oracle wanting to deploy these? Oracle has a deep partnership with Nvidia to the point that in many cases, they have joint booths at conferences. I was not aware of any efforts to purchase or deploy MI300s at OCI.
- dhruvdh 3y agoOracle were present at the event today and said it themselves. Also, it has been widely reported for months now that this is happening.
- viewtransform 3y agoAt today's AMD presentation the SVP of Oracle Cloud Infrastructure reveals it. https://youtu.be/tfSZqjxsr0M?t=2006 https://youtu.be/tfSZqjxsr0M?t=2006
- metadat 3y agoThere are a lot of SVPs, but this was still helpful, even though Karan is just a small fry! Looking forward to trying out the MI300x.
- IceHegel 3y agoThey announced the Maia 100 AI accelerator last month. TSMC N5, 105b transistors. https://news.microsoft.com/source/features/ai/in-house-chips-silicon-to-service-to-meet-ai-demand/ https://news.microsoft.com/source/features/ai/in-house-chips...
- TradingPlaces 3y ago*if you ignore sparsity.
- shrubble 3y agoSo how much of the market does AMD need to be profitable and keep going? Supposedly the Nvidia H100 alone, has ~15 billion USD in order backlog. If AMD sells say $3 billion USD into that market, is that a big net positive for them?
- hajile 3y agoAMD's use of chiplets matters here. There is overhead, but they also not only get higher yields, but also have many more chips that can hit their highest target clockspeed meaning they can ship far more high-end chips. This is the same reason why they were so successful with EPYC. It's easier to find eight small 8-core chips with high clocks/low-power than to find one 56-core chip with that same high-clock/low-power set which is why Intel sold so many cut-down chips while AMD just sold their small amount of defective chips as 6 and 12-core consumer chips. Nvidia will have a harder time shipping defective because everyone wants the best chips with the best performance/watt to save money and do things fast. Outside of that, there's not a huge market for defective 800mm2 chips and the massive GPU the come on. AMD can sell their defective units a couple chiplets at a time to non-AI customers in laptops and workstation cards.
- shrubble 3y agoThank you for that detailed info - had not thought about that...
- shaklee3 3y agoIt also implies that it's harder to program efficiently and is slower on the interconnect.
- aurareturn 3y agoIt should be a transparent GPU like M1 Ultra.
- shaklee3 3y agoIt's transparent, but it has a performance cost. There will be low-level ways to improve locality for certain libraries, but it's nearly impossible for common users to claw that performance loss back.
- brucethemoose2 3y agoOne very exciting thing is the vram. 192GB!!! That seems like a minor bump compared to the new H100, but sometimes the quality of your training, or the speed of your inference, is hugely effected by small thresholds of VRAM capacity per card. The MI300X can do things the H100 cannot, and judging by the sheer size of the silicon, do them reasonably quickly.
- dharma1 3y agoI think vram is one thing they can beat Nvidia on. Both on the pro and consumer GPUs. Just put 48gb into a consumer GPU and everyone would flock to AMD and help build their software side
- brucethemoose2 3y agoThey already have 48GB Pro GPUs. If they were going to do it on the consumer side, they would have done it already. And I would own a 7900 48GB instead of a 3090, and probably have debugged ROCm on several projects by now :/
- dharma1 3y agoExactly - users would flock to it. They have to compete with Nvidia somehow and I don't see what other edge they could have - massive disadvantage on the software side that has to be compensated for in some way
- brucethemoose2 3y agoWell local LLM running with rocm isnt exactly huge business. You are not wrong, but I can envision why decision makers wouldn't want to spoil the pro line.
- dharma1 3y agoThe real pros will use data centre GPUs like the MI300. I think local ML will become huge in games
- yukIttEft 3y agoH100 GPU is a sibling of NVidia's customer RTX 40-series: https://en.wikipedia.org/wiki/Hopper_(microarchitecture) https://en.wikipedia.org/wiki/Hopper_(microarchitecture) Is MI300 also a variant of their customer GPUs?
- edward28 3y agoThis is a different architecture, CDNA3
- anticensor 3y agoWhich is a derivative of pre-RDNA customer GPU architecture, Vega.
- fransje26 3y agoDoes the memory controller also bottom out at about half the bandwidth in practice, like on the MI250? Or is the full memory bandwidth finally available? Theoretical numbers are nice, but they are just that, ethereal numbers on a piece of paper. And on a CUDA device, I know from practice that I can get at least 90% of the memory bandwidth.
- phdelightful 3y agoDOE HPC people are not having this problem with AMD GPUs: here’s a paper that reports 1.36 TB/s on one GCD of an MI250x (theoretical is 1.6TB/s). https://dl.acm.org/doi/pdf/10.1145/3624062.3624203 https://dl.acm.org/doi/pdf/10.1145/3624062.3624203 Table 6
- MichaelRazum 3y agoIt costs about 20k$ as far as I understand. Why is it so expensive?