8 ms·

CUDA Moat Still Alive

221 points by pella 2y ago

gigel82 2y ago
What I couldn't find is inference benchmarks for consumer hardware. Just pick a reasonable workload with llama.cpp or ollama and show us some numbers. I'm particularly interested in building a Home Assistant machine that can run the voice assistant locally (STT/TTS/LLM) while using the least amount of power / generating the least amount of heat and noise.
- GaggiX 2y ago
  It would be cool to see these benchmarks on the newly released Jetson Orin Nano Super, like faster-whisper.
- cherryteastain 2y ago
  Based on a machine we had bought at my university with 4 AMD W6800s (which are just RX 6800s with double the VRAM), it's bad _even if it works at all_.
- curt15 2y ago
  AMD's software for consumer GPUs demonstrates a lack of seriousness. ROCm only officially supports RDNA2 and RDNA3 GPUs (their last two generations of hardware), and for some reason most of them are supported on only Windows (https://rocm.docs.amd.com/projects/install-on-windows/en/latest/reference/system-requirements.html https://rocm.docs.amd.com/projects/install-on-windows/en/lat...) and not Linux (https://rocm.docs.amd.com/projects/install-on-linux/en/latest/reference/system-requirements.html https://rocm.docs.amd.com/projects/install-on-linux/en/lates...), where most AI training and inference occurs. In particular, Linux users can only start playing with ROCm with a top-of-the-line, power-guzzling unit whereas they can get started with CUDA using basically any Nvidia GPU on desktops or laptops.
  - cherryteastain 2y ago
    In practice, consumer Navi 21 based cards (RX 6900XT etc) and Navi 31 cards (RX 7900 XTX etc) are compatible with Pytorch on Linux. What they write about ROCm and Windows is equivocation. They target only one app: Blender. Pytorch+ROCm+Windows does not work. I had bought a 6900XT myself around launch time (the RTX3080 I ordered was not coming, it was the chip shortage times...) and it took around 2 years for Pytorch to become actually usable on it.
    - thousand_nights 2y ago
      in practice everyone who wants to do ML at home buys nvidia and pays the premium
      - tonetegeatinst 2y ago
        Sad but true. Years ago, pre2018 nvidia was the goto hardware supplier if you were doing anything with neural networks. I remember CUDA being much more buggy back then but it still worked pretty good. Back then AMD wasn't considered a real competition for ML/AI hardware. Glad as always to see more competition in the market to drive innovations. AMD seems to be letting larger VRAM onto consumer cards, which is nice to see, just hope the AI/ML experience can get better for their software ecosystem.
    - ryao 2y ago
      The obligatory link: https://xkcd.com/644/ https://xkcd.com/644/ That said, I would not expect it to stay working for long as long as ROCm is a dependency since AMD drops support for its older GPUs quickly while Nvidia continues to support older GPUs with less frequent legacy driver updates.
    - sliken 2y ago
      Maybe, but it's pretty lame when you buy a new $500 CPU (7800 XT) and the docs say it's unsupported, and makes you feel like even if you reported a bug they would just say "Sorry, not supported". Did make me wish I bought a Nvidia.
- sroussey 2y ago
  You might just check out the Home Assistant Voice: https://ameridroid.com/products/home-assistant-voice-preview-edition https://ameridroid.com/products/home-assistant-voice-preview...
  - gigel82 2y ago
    Yes, that's exactly what I was checking out. You need fast enough hardware to run the speech to text, text to speech and (most importantly) LLM locally: https://www.youtube.com/watch?v=XvbVePuP7NY https://www.youtube.com/watch?v=XvbVePuP7NY (he has dual 3090 GPUs but that's not a practical setup for most people - budget / power / noise).
the_king 2y ago
> It’s not just that it’s immature software, they need to change how they do development. I remember geohot saying something similar about a year ago
- delusional 2y ago
  "The software needs to be better" is (and was) an easy call to make for anyone paying attention. The problem is that "AMD just needs to do better" is not and will never be an implementable strategy. Engineering isn't just about money. It's also about the process of exploring all the edge cases. "We recommend that AMD to fix their GEMM libraries’ heuristic model such that it picks the correct algorithm out of the box instead of wasting the end user’s time doing tuning on their end." Is such a profoundly unhelpful thing to say unless you imagine AMDs engineers just sitting around wondering what to do all day. AMD needs to make their drivers better, and they have. Shit just takes time.
  - the_king 2y ago
    Sounds more like they were (and still are) being sloppy. “be better” is one thing. “runs without fatal crash” is what semi is talking about.
- cruunchMuncher 2y ago
  [dead]
- nabla9 2y ago
  In buggy numerical code many bugs go trough the software stack without any problems. No crash, no errors. For example,you might switch two double parameters to a function and if their value range is similar, everything works fine except it's all bullshit. If there are bugs in AMD code that prevent running tests, I bet there are even more bugs that don't manifest until you look at results.
- roenxi 2y ago
  I expect everyone has been saying it for a while, the calls are just getting more strident and public as it becomes clear that AMD's failures are strategic rather than tactical. And as people try to build business on their half-hearted attempts. I still think it is a mistake to say that CUDA is a moat. IMO the problem here is that AMD still doesn't seem to think that GPGPU compute is a thing. They don't seem to understand the idea that someone might want to use their graphics cards to multiply matricies independently of a graphics pipeline. All the features CUDA supports are irrelevant compared to the fact that AMD can't handle GEMM performantly out of the box. In my experience it just can't do it, back in the day my attempts to multiply matrices would crash drivers. That isn't a moat, but it certainly is something spectacular. If they could manage an engineering process that delivered good GEMM performance then the other stuff can probably get handled. But without it there really is a question of what these cards are for.
nabla9 2y ago
B100, B200 ramp-up is only 4-6 months away.
dboreham 2y ago
Sounds like a buy signal for AMD. If you run the right branch and set the right env cars, the thing flies.
- cherryteastain 2y ago
  Unfortunately, > Getting reasonable training performance out of AMD MI300X is an NP-Hard problem.
- rikafurude21 2y ago
  Would be a a buy signal if their actions (better drivers) show that they are seriously working on improving software. "This could be great _if_ you go through the trouble of doing it right!" is not persuasive, and any sane person would go with green if they know they have the choice between troubleshooting shitty software and things just working. Look at the george hotz archive youtube channel and watch the videos where he's debugging the amd drivers, it damn near ruins the man. And george is not the type to give up at the first roadblock, there are multiple 5-8 hour videos where he just tries to get the thing to work. The mad man ended up just writing his own driver lol.
  - ip26 2y ago
    It does seem like an improvement. Six or twelve months ago, I recall a lot of crashes and even more basic problems. “If you tune it right, it’s awesome” is a big step forward compared to that.
- incrudible 2y ago
  1. AMD always had a lot of hype already priced in, it is no different with AI. 2. AMD has always shipped a bad software stack, it is no different with AI.
- latchkey 2y ago
  Anthony has been doing training on 4 of our MI300x systems and has been getting great results... but of course, he is a genius, and writing his own code... https://x.com/HotAisle/status/1870984996171006035 https://x.com/HotAisle/status/1870984996171006035 https://x.com/zealandic1/status/1869857713280430349 https://x.com/zealandic1/status/1869857713280430349 https://x.com/zealandic1/status/1868810042168033623 https://x.com/zealandic1/status/1868810042168033623
stefan_ 2y ago
I made the mistake of clicking on one of the links to commits they mentioned only to end up at a MR changing multiple autogenerated yaml files with 10k line diffs and incomprehensible names. I guess this is where the whole "bad talent" thing comes in - a year later and you are thousands of YAML files deep but still no one can run a simple PyTorch compile ops and get the performance you sold, absolutely unhinged.
nrp 2y ago
> Give AMD Engineers more compute and engineering resources to fix and improve the AMD ecosystem, they have very few internal gpu boxes relative to what Nvidia provides to their engineers. This is real. We’ve found ourselves having to give hardware to engineers at AMD because they’re unable to get allocation of it internally.
- mountainriver 2y ago
  This seems so insane, is anyone actually doing the work to provide an alternative to CUDA? Maybe Google?
  - ryao 2y ago
    https://www.intel.com/content/www/us/en/developer/articles/technical/quick-guide-to-sycl-implementations.html https://www.intel.com/content/www/us/en/developer/articles/t...
  - anon291 2y ago
    Honestly, probably NVIDIA itself, since they contribute significantly to many open-source projects (MLIR), and also make their SoTA GEMM/Conv implementations open-source and available for study (Cutlass).
    - david-gpu 2y ago
      *> also make their SoTA GEMM/Conv implementations open-source and available for study (Cutlass)" Cutlass is a fine piece of engineering, but it is not quite as good as their closed source libraries in real world workloads. There is secret sauce that is not open sourced.
  - pjmlp 2y ago
    The problem is that everyone that tries, keeps missing the CUDA forest and focus only in a specific kind of tree.
- Archit3ch 2y ago
  Coming up next: "We bought AMD stock on the open market and used it to compensate AMD engineers".
cyberax 2y ago
> CUDA Moat Still Alive Wrong conclusion. AMD is slower than NVidia, but not _that_ much slower. They are actually pretty cost-competitive. The just need to do some improvements, and they'll be a very viable competitor.
- SideQuark 2y ago
  The amount of effort this team took, literally co-opting AMD engineers, and working for 5 months, to get closer but not yet usable, means they are not even close to usable. What team wanting to do ML training/inference can afford so much down time for zero benefit? How many except a few big ones can get AMD to devote so many resources simply for that team? And, if you’re training a model costing you millions, the last thing you need is a buggy, untested stack, breaking training or perhaps worse giving you noise that makes your models perform worse or increases training time. By the time AMD gets usable out of the box at this point, NVidia will have moved further ahead.
  - cyberax 2y ago
    Sure. But this work is done, and can be reused by others. Meanwhile, Nvidia hardware is expensive and still is in short supply. AMD might look quite tempting.
    - SideQuark 2y ago
      It sure doesn’t sound done. It’s a one off hacked set of scripts tied to incompatible chunks of a ton of libraries. What happens when you want or need other parts of the PyTorch/billion libs ecosystem? You’re gonna get more AMD engineers and waste 5 months getting those to work? Meanwhile those libs release running CUDA on NVidia’s old and newest releases out of the box. So no, it cannot be reused by others in production any more than my custom hacked car engine mod can be added by Ford to every car in existence. Have you done any deep professional production work on any of these stacks? I have, and would never, ever put stuff like the stuff in the article in production. It’s no where near ready for production use.
    - ryao 2y ago
dzdt 2y ago
[flagged]
- nrp 2y ago
  In this case it’s because Dylan Patel of Semianalysis interviews Lisa Su regularly and presumably has a direct line to her, and because Lisa and the rest of AMD leadership are absolutely reading the article. It’s unclear if Pat would have (e.g. I don’t think Pat ever sat down for a chat with Semianalysis like Lisa has).
- bloodyplonker22 2y ago
  This is the exact type of victim mentality that we don't need. There are absolute insane amounts of people calling out Gelsinger by name and blaming solely him for failures at Intel.
- hedgehog 2y ago
  It might be because Lisa has been so outstandingly effective at making AMD competitive across multiple product lines against bigger competitors and this feels to some people like an oversight that AMD could easily solve. I suspect the current situation with ML software at AMD is a consequence of a very focused company and not an easy fix without sacrificing something else. I don't think many people can keep track of who's running Intel let alone have hope that with a little work they can deliver reasonable substitutes for NVIDIA's products.
- redox99 2y ago
  People call out Gelsinger all the time. And yes, she is definitely responsible for this. Probably more than Gelsinger. At Intel it is not so obvious what they should have done to improve their fabs to better compete with TSMC, which is groundbreaking tech where you often have to make risky bets. At AMD it was pretty obvious what had to be done to better compete in AI, and it was basic software engineering, not lithography wizardry. Totally achievable by just spending money, increasing head count, hiring top talent, and firing underperformers. They have so much low hanging fruit that could have been solved just by hiring 5 or 10 software engineers.
- kaliqt 2y ago
9cb14c1ec0 2y ago
I once tried installing AMD ROCM to run a small llm on a consumer-grade AMD GPU. It was the most horrible software install experience I ever had. Never did manage to get it working.
Lerc 2y ago
That MatMul performance is fairly shocking. To be that much below theoretical maximum on what should be a fairly low overhead operation. I would at least hope that they know where the speed is going, but the issue of torch.matmul and F.Linear using different libraries with different performance suggests that they don't even know which code they are running, let alone where the slow bits in that code are.
- kevingadd 2y ago
  Low overhead in what sense? matmul is kinda complicated and there are varying, complex state-of-the-art algorithms for it, no? And then if you know things about the matrices in advance you can start optimizing for that, which adds another layer of complexity.
  - jiggawatts 2y ago
    Yes and no. Conceptually it's just three nested loops. The fiddly part is unrolling the inner loop and swizzling the data layouts in such a way that the cores can be kept "fed" efficiently. This usually means breaking things up into cache-sized chunks along some axis. It's easy enough that there's blog articles showing single developers getting within spitting distance of NVIDIA's highly optimised code. As in, 80-something-percent of the best available algorithms! All NVIDIA did was "put the effort in", where the effort isn't some super clever algorithm implemented by a unique genius, but they simply made hundreds of variants of the matmul algorithm optimised for various scenarios. It's a kind of algorithmic brute force for eking out every last percentage point for every shape and size of input matrices on every GPU model and even for various SLI configurations. From what I've seen, AMD has done... none of this.
    - cavisne 2y ago
      How do the cache sizes compare between AMD GPU’s and nvidia? I remember reading a while ago they were quite different (enough to make flash attention painful to implement)
    - Fronzie 2y ago
shrubble 2y ago
Back in the late 1990s I met the ATI guys, and they were slipshod then as well. That the ATI legacy of special-casing things lives on is sadly, not too surprising for me.
ryao 2y ago
> AMD is attempting to vertically integrate next year with their upcoming Pollara 400G NIC, which supports Ultra Ethernet, hopefully making AMD competitive with Nvidia. Infiniband is an industry standard. It is weird to see the industry invent yet another standard to do effectively the same thing just because Nvidia is using it. This “Nvidia does things this way so let’s do it differently” mentality is hurting AMD: * Nvidia has a unified architecture so let’s split ours into RDNA and CDNA. * Nvidia has a unified driver, so let’s make a different driver for every platform. * Nvidia made a virtual ISA (PTX) for backward compatibility. Let’s avoid that. * Nvidia is implementing tensor cores. Let’s avoid those on RDNA. Then implement them on CDNA and call them matrix cores. * Nvidia is using Infiniband like the rest of the HPC community. Let’s use Ethernet. I am sure people can find more examples. Also, they seem to have realized their mistake in splitting their architecture into RDNA and CDNA, since they are introducing UDNA in the future to unify them like Nvidia does.
- dralley 2y ago
  You're painting this like AMD is off to play in their own sandbox when it's more like the entire industry is trying to develop an alternative to Infiniband. Ultra Ethernet is a joint project between dozens of companies organized under the Linux Foundation. https://www.phoronix.com/news/Ultra-Ethernet-Consortium https://www.phoronix.com/news/Ultra-Ethernet-Consortium >> The Linux Foundation has established the Ultra Ethernet Consortium "UED" as an industry-wide effort founded by AMD, Arista, Broadcom, Cisco, Eviden, HPE, Intel, Meta, and Microsoft for designing a new Ethernet-based communication stack architecture for high performance networking. You probably can't call it "industry standard" yet but the goal is obviously for it to become one.
  - ryao 2y ago
    I wrote: > It is weird to see the industry invent yet another standard to do effectively the same thing just because Nvidia is using it. This is a misstep for all involved, AMD included. Even if AMD is following everyone else by jumping off a bridge, AMD is still jumping too.
- croes 2y ago
dhruvdh 2y ago
Disappointed that there wasn’t anything on inference performance in the article at all. That’s what the major customers have announced they use it for.
nromiun 2y ago
So basically Nvidia is like Windows and AMD is like Wine. I think trying to emulate CUDA and using forked Nvidia libraries is not the best strategy for AMD. They should have made a clean break and come out with a fresh API, like Apple's Metal.
benreesman 2y ago
AMD could spend their market cap in one year to get this done in three and it would be a coup for the shareholders. They could hire all of the best NVIDIA engineers at double their current comp, crush the next TSMC node on Apple levels, and just do it and if it got them a quarter of NVDA’s cap it would be a bargain. They don’t fucking want to! Believing this is anything like a market is fucking religion.
- robocat 2y ago
  Try to make sense... They can spend their market cap by either: 1: issuing new shares worth their market cap, diluting existing shareholders to 50%. 2: Or borrow their market cap and pay interest by decreasing profits. "AMD operating margin for the quarter ending September 30, 2024 was 5.64%" so profits would be extremely impacted by interest repayments. Either way your suggestion would be unlikely to be supported by shareholders. > crush the next TSMC node on Apple levels I would guess Apple is indirectly paying for the hardware (to avoid repatriating profits) or guaranteeing usage to get to the front of the line at TSMC. Good luck AMD competing with Apple: there's a reason AMD sold GlobalFoundries and there's a reason Intel is now struggling with their foundry costs. And it comes across as condescending to assume you know better than a successful company.
  - benreesman 2y ago
    When 4 trillion dollars are at stake, the financing is available or it fucking better be. What in God’s name do we pay these structured finance, bond-issue assholes 15% of GDP for if not to finance a sure thing like that? It sure as hell ain’t for their taste in Charvet and Hermes ties, because the ones they pick look like shit.
- atq2119 2y ago
  You make it sound like that's a sure thing, but I doubt it. A lot of this is about processes, team structures and incentives, all those fuzzy things between the people. Remember, most acquisitions fail. For the same reason, the likelihood of failure with your scenario seems high. Do you really think nobody at AMD is aware of all the points made in this thread? That seems too bizarre to be true. There are probably some issues in upper management which could perhaps be fixed with some targeted hiring decisions, but do you really believe some random person on here would have a chance making that call?
anon291 2y ago
My anecdata on AMD hiring: they just aren't moving fast enough. They still wanted to fly people out scheduling 3 weeks in advance for AI compiler work. That's just not going to work. Startups and companies like NVIDIA, OpenAI are hiring much faster with much less onerous interview processes, with higher compensation. This is not a mystery. People work for money and aren't going to hop through more hoops to be paid less.
cavisne 2y ago
I think it’s hardware not software. “Cuda moat” is a misnomer. The PTX spec is relatively short (600 page pdf). Triton directly writes PTX, skipping cuda. Flash attention was created by a non nvidia employee without access to any of the secret sauce within Cuda or its libraries. The hardware is just not as good, and no software can paper over its flaws.
- pjmlp 2y ago
  NVidia also refactored their hardware design to follow C++ memory model, I think this still isn't the case of others.
MortyWaves 2y ago
Wow they really botched the title. Why wasn’t it “Still Filled With Water”?
klelatti 2y ago
AMD doesn’t just have to fix these issues it has to build up a record of fixing issues like those discussed here. Otherwise who will bet their firm / cash / career on new hardware without a successful track record.
yaro330 2y ago
TL;DR: AMD still doesn't take software seriously?
pella 2y ago
Latest: Dylan Patel ( SemiAnalysis ) "Met with @LisaSu today for 1.5 hours as we went through everything She acknowledged the gaps in AMD software stack She took our specific recommendations seriously She asked her team and us a lot of questions Many changes are in flight already! Excited to see improvements coming" https://x.com/dylan522p/status/1871287937268383867 https://x.com/dylan522p/status/1871287937268383867
- pella 2y ago
  Lisa Su : https://x.com/LisaSu/status/1871362304194859511 https://x.com/LisaSu/status/1871362304194859511 "Thanks @dylan522p for the constructive conversation today. Feedback is a gift even when it’s critical. We have put a ton of work into customer and workload optimizations but there is lots more we can do to enable the broad ecosystem. I appreciate all the feedback and desire to engage with @AMD. We are committed to building a world-class open software stack. Lots planned for 2025. Happy holidays to all!"
varelse 2y ago
[dead]

221 points by pella 2y ago

What I couldn't find is inference benchmarks for consumer hardware. Just pick a reasonable workload with llama.cpp or ollama and show us some numbers. I'm particularly interested in building a Home Assistant machine that can run the voice assistant locally (STT/TTS/LLM) while using the least amount of power / generating the least amount of heat and noise.

GaggiX 2y ago
It would be cool to see these benchmarks on the newly released Jetson Orin Nano Super, like faster-whisper.
cherryteastain 2y ago
Based on a machine we had bought at my university with 4 AMD W6800s (which are just RX 6800s with double the VRAM), it's bad _even if it works at all_.
curt15 2y ago
AMD's software for consumer GPUs demonstrates a lack of seriousness. ROCm only officially supports RDNA2 and RDNA3 GPUs (their last two generations of hardware), and for some reason most of them are supported on only Windows (https://rocm.docs.amd.com/projects/install-on-windows/en/latest/reference/system-requirements.html https://rocm.docs.amd.com/projects/install-on-windows/en/lat...) and not Linux (https://rocm.docs.amd.com/projects/install-on-linux/en/latest/reference/system-requirements.html https://rocm.docs.amd.com/projects/install-on-linux/en/lates...), where most AI training and inference occurs. In particular, Linux users can only start playing with ROCm with a top-of-the-line, power-guzzling unit whereas they can get started with CUDA using basically any Nvidia GPU on desktops or laptops.
- cherryteastain 2y ago
  In practice, consumer Navi 21 based cards (RX 6900XT etc) and Navi 31 cards (RX 7900 XTX etc) are compatible with Pytorch on Linux. What they write about ROCm and Windows is equivocation. They target only one app: Blender. Pytorch+ROCm+Windows does not work. I had bought a 6900XT myself around launch time (the RTX3080 I ordered was not coming, it was the chip shortage times...) and it took around 2 years for Pytorch to become actually usable on it.
  - thousand_nights 2y ago
    in practice everyone who wants to do ML at home buys nvidia and pays the premium
    - tonetegeatinst 2y ago
      Sad but true. Years ago, pre2018 nvidia was the goto hardware supplier if you were doing anything with neural networks. I remember CUDA being much more buggy back then but it still worked pretty good. Back then AMD wasn't considered a real competition for ML/AI hardware. Glad as always to see more competition in the market to drive innovations. AMD seems to be letting larger VRAM onto consumer cards, which is nice to see, just hope the AI/ML experience can get better for their software ecosystem.
  - ryao 2y ago
    The obligatory link: https://xkcd.com/644/ https://xkcd.com/644/ That said, I would not expect it to stay working for long as long as ROCm is a dependency since AMD drops support for its older GPUs quickly while Nvidia continues to support older GPUs with less frequent legacy driver updates.
  - sliken 2y ago
    Maybe, but it's pretty lame when you buy a new $500 CPU (7800 XT) and the docs say it's unsupported, and makes you feel like even if you reported a bug they would just say "Sorry, not supported". Did make me wish I bought a Nvidia.
sroussey 2y ago
You might just check out the Home Assistant Voice: https://ameridroid.com/products/home-assistant-voice-preview-edition https://ameridroid.com/products/home-assistant-voice-preview...
- gigel82 2y ago
  Yes, that's exactly what I was checking out. You need fast enough hardware to run the speech to text, text to speech and (most importantly) LLM locally: https://www.youtube.com/watch?v=XvbVePuP7NY https://www.youtube.com/watch?v=XvbVePuP7NY (he has dual 3090 GPUs but that's not a practical setup for most people - budget / power / noise).

the_king 2y ago

> It’s not just that it’s immature software, they need to change how they do development. I remember geohot saying something similar about a year ago

delusional 2y ago
"The software needs to be better" is (and was) an easy call to make for anyone paying attention. The problem is that "AMD just needs to do better" is not and will never be an implementable strategy. Engineering isn't just about money. It's also about the process of exploring all the edge cases. "We recommend that AMD to fix their GEMM libraries’ heuristic model such that it picks the correct algorithm out of the box instead of wasting the end user’s time doing tuning on their end." Is such a profoundly unhelpful thing to say unless you imagine AMDs engineers just sitting around wondering what to do all day. AMD needs to make their drivers better, and they have. Shit just takes time.
- the_king 2y ago
  Sounds more like they were (and still are) being sloppy. “be better” is one thing. “runs without fatal crash” is what semi is talking about.
cruunchMuncher 2y ago
[dead]
nabla9 2y ago
In buggy numerical code many bugs go trough the software stack without any problems. No crash, no errors. For example,you might switch two double parameters to a function and if their value range is similar, everything works fine except it's all bullshit. If there are bugs in AMD code that prevent running tests, I bet there are even more bugs that don't manifest until you look at results.
roenxi 2y ago
I expect everyone has been saying it for a while, the calls are just getting more strident and public as it becomes clear that AMD's failures are strategic rather than tactical. And as people try to build business on their half-hearted attempts. I still think it is a mistake to say that CUDA is a moat. IMO the problem here is that AMD still doesn't seem to think that GPGPU compute is a thing. They don't seem to understand the idea that someone might want to use their graphics cards to multiply matricies independently of a graphics pipeline. All the features CUDA supports are irrelevant compared to the fact that AMD can't handle GEMM performantly out of the box. In my experience it just can't do it, back in the day my attempts to multiply matrices would crash drivers. That isn't a moat, but it certainly is something spectacular. If they could manage an engineering process that delivered good GEMM performance then the other stuff can probably get handled. But without it there really is a question of what these cards are for.

nabla9 2y ago

B100, B200 ramp-up is only 4-6 months away.

dboreham 2y ago

Sounds like a buy signal for AMD. If you run the right branch and set the right env cars, the thing flies.

cherryteastain 2y ago
Unfortunately, > Getting reasonable training performance out of AMD MI300X is an NP-Hard problem.
rikafurude21 2y ago
Would be a a buy signal if their actions (better drivers) show that they are seriously working on improving software. "This could be great _if_ you go through the trouble of doing it right!" is not persuasive, and any sane person would go with green if they know they have the choice between troubleshooting shitty software and things just working. Look at the george hotz archive youtube channel and watch the videos where he's debugging the amd drivers, it damn near ruins the man. And george is not the type to give up at the first roadblock, there are multiple 5-8 hour videos where he just tries to get the thing to work. The mad man ended up just writing his own driver lol.
- ip26 2y ago
  It does seem like an improvement. Six or twelve months ago, I recall a lot of crashes and even more basic problems. “If you tune it right, it’s awesome” is a big step forward compared to that.
incrudible 2y ago
1. AMD always had a lot of hype already priced in, it is no different with AI. 2. AMD has always shipped a bad software stack, it is no different with AI.
latchkey 2y ago
Anthony has been doing training on 4 of our MI300x systems and has been getting great results... but of course, he is a genius, and writing his own code... https://x.com/HotAisle/status/1870984996171006035 https://x.com/HotAisle/status/1870984996171006035 https://x.com/zealandic1/status/1869857713280430349 https://x.com/zealandic1/status/1869857713280430349 https://x.com/zealandic1/status/1868810042168033623 https://x.com/zealandic1/status/1868810042168033623

stefan_ 2y ago

I made the mistake of clicking on one of the links to commits they mentioned only to end up at a MR changing multiple autogenerated yaml files with 10k line diffs and incomprehensible names. I guess this is where the whole "bad talent" thing comes in - a year later and you are thousands of YAML files deep but still no one can run a simple PyTorch compile ops and get the performance you sold, absolutely unhinged.

nrp 2y ago

> Give AMD Engineers more compute and engineering resources to fix and improve the AMD ecosystem, they have very few internal gpu boxes relative to what Nvidia provides to their engineers. This is real. We’ve found ourselves having to give hardware to engineers at AMD because they’re unable to get allocation of it internally.

mountainriver 2y ago
This seems so insane, is anyone actually doing the work to provide an alternative to CUDA? Maybe Google?
- ryao 2y ago
  https://www.intel.com/content/www/us/en/developer/articles/technical/quick-guide-to-sycl-implementations.html https://www.intel.com/content/www/us/en/developer/articles/t...
- anon291 2y ago
  Honestly, probably NVIDIA itself, since they contribute significantly to many open-source projects (MLIR), and also make their SoTA GEMM/Conv implementations open-source and available for study (Cutlass).
  - david-gpu 2y ago
    *> also make their SoTA GEMM/Conv implementations open-source and available for study (Cutlass)" Cutlass is a fine piece of engineering, but it is not quite as good as their closed source libraries in real world workloads. There is secret sauce that is not open sourced.
- pjmlp 2y ago
  The problem is that everyone that tries, keeps missing the CUDA forest and focus only in a specific kind of tree.
Archit3ch 2y ago
Coming up next: "We bought AMD stock on the open market and used it to compensate AMD engineers".

cyberax 2y ago

> CUDA Moat Still Alive Wrong conclusion. AMD is slower than NVidia, but not _that_ much slower. They are actually pretty cost-competitive. The just need to do some improvements, and they'll be a very viable competitor.

SideQuark 2y ago
The amount of effort this team took, literally co-opting AMD engineers, and working for 5 months, to get closer but not yet usable, means they are not even close to usable. What team wanting to do ML training/inference can afford so much down time for zero benefit? How many except a few big ones can get AMD to devote so many resources simply for that team? And, if you’re training a model costing you millions, the last thing you need is a buggy, untested stack, breaking training or perhaps worse giving you noise that makes your models perform worse or increases training time. By the time AMD gets usable out of the box at this point, NVidia will have moved further ahead.
- cyberax 2y ago
  Sure. But this work is done, and can be reused by others. Meanwhile, Nvidia hardware is expensive and still is in short supply. AMD might look quite tempting.
  - SideQuark 2y ago
    It sure doesn’t sound done. It’s a one off hacked set of scripts tied to incompatible chunks of a ton of libraries. What happens when you want or need other parts of the PyTorch/billion libs ecosystem? You’re gonna get more AMD engineers and waste 5 months getting those to work? Meanwhile those libs release running CUDA on NVidia’s old and newest releases out of the box. So no, it cannot be reused by others in production any more than my custom hacked car engine mod can be added by Ford to every car in existence. Have you done any deep professional production work on any of these stacks? I have, and would never, ever put stuff like the stuff in the article in production. It’s no where near ready for production use.
  - ryao 2y ago

dzdt 2y ago

[flagged]

nrp 2y ago
In this case it’s because Dylan Patel of Semianalysis interviews Lisa Su regularly and presumably has a direct line to her, and because Lisa and the rest of AMD leadership are absolutely reading the article. It’s unclear if Pat would have (e.g. I don’t think Pat ever sat down for a chat with Semianalysis like Lisa has).
bloodyplonker22 2y ago
This is the exact type of victim mentality that we don't need. There are absolute insane amounts of people calling out Gelsinger by name and blaming solely him for failures at Intel.
hedgehog 2y ago
It might be because Lisa has been so outstandingly effective at making AMD competitive across multiple product lines against bigger competitors and this feels to some people like an oversight that AMD could easily solve. I suspect the current situation with ML software at AMD is a consequence of a very focused company and not an easy fix without sacrificing something else. I don't think many people can keep track of who's running Intel let alone have hope that with a little work they can deliver reasonable substitutes for NVIDIA's products.
redox99 2y ago
People call out Gelsinger all the time. And yes, she is definitely responsible for this. Probably more than Gelsinger. At Intel it is not so obvious what they should have done to improve their fabs to better compete with TSMC, which is groundbreaking tech where you often have to make risky bets. At AMD it was pretty obvious what had to be done to better compete in AI, and it was basic software engineering, not lithography wizardry. Totally achievable by just spending money, increasing head count, hiring top talent, and firing underperformers. They have so much low hanging fruit that could have been solved just by hiring 5 or 10 software engineers.
kaliqt 2y ago

9cb14c1ec0 2y ago

I once tried installing AMD ROCM to run a small llm on a consumer-grade AMD GPU. It was the most horrible software install experience I ever had. Never did manage to get it working.

Lerc 2y ago

That MatMul performance is fairly shocking. To be that much below theoretical maximum on what should be a fairly low overhead operation. I would at least hope that they know where the speed is going, but the issue of torch.matmul and F.Linear using different libraries with different performance suggests that they don't even know which code they are running, let alone where the slow bits in that code are.

kevingadd 2y ago
Low overhead in what sense? matmul is kinda complicated and there are varying, complex state-of-the-art algorithms for it, no? And then if you know things about the matrices in advance you can start optimizing for that, which adds another layer of complexity.
- jiggawatts 2y ago
  Yes and no. Conceptually it's just three nested loops. The fiddly part is unrolling the inner loop and swizzling the data layouts in such a way that the cores can be kept "fed" efficiently. This usually means breaking things up into cache-sized chunks along some axis. It's easy enough that there's blog articles showing single developers getting within spitting distance of NVIDIA's highly optimised code. As in, 80-something-percent of the best available algorithms! All NVIDIA did was "put the effort in", where the effort isn't some super clever algorithm implemented by a unique genius, but they simply made hundreds of variants of the matmul algorithm optimised for various scenarios. It's a kind of algorithmic brute force for eking out every last percentage point for every shape and size of input matrices on every GPU model and even for various SLI configurations. From what I've seen, AMD has done... none of this.
  - cavisne 2y ago
    How do the cache sizes compare between AMD GPU’s and nvidia? I remember reading a while ago they were quite different (enough to make flash attention painful to implement)
  - Fronzie 2y ago

shrubble 2y ago

Back in the late 1990s I met the ATI guys, and they were slipshod then as well. That the ATI legacy of special-casing things lives on is sadly, not too surprising for me.

ryao 2y ago

> AMD is attempting to vertically integrate next year with their upcoming Pollara 400G NIC, which supports Ultra Ethernet, hopefully making AMD competitive with Nvidia. Infiniband is an industry standard. It is weird to see the industry invent yet another standard to do effectively the same thing just because Nvidia is using it. This “Nvidia does things this way so let’s do it differently” mentality is hurting AMD: * Nvidia has a unified architecture so let’s split ours into RDNA and CDNA. * Nvidia has a unified driver, so let’s make a different driver for every platform. * Nvidia made a virtual ISA (PTX) for backward compatibility. Let’s avoid that. * Nvidia is implementing tensor cores. Let’s avoid those on RDNA. Then implement them on CDNA and call them matrix cores. * Nvidia is using Infiniband like the rest of the HPC community. Let’s use Ethernet. I am sure people can find more examples. Also, they seem to have realized their mistake in splitting their architecture into RDNA and CDNA, since they are introducing UDNA in the future to unify them like Nvidia does.

dralley 2y ago
You're painting this like AMD is off to play in their own sandbox when it's more like the entire industry is trying to develop an alternative to Infiniband. Ultra Ethernet is a joint project between dozens of companies organized under the Linux Foundation. https://www.phoronix.com/news/Ultra-Ethernet-Consortium https://www.phoronix.com/news/Ultra-Ethernet-Consortium >> The Linux Foundation has established the Ultra Ethernet Consortium "UED" as an industry-wide effort founded by AMD, Arista, Broadcom, Cisco, Eviden, HPE, Intel, Meta, and Microsoft for designing a new Ethernet-based communication stack architecture for high performance networking. You probably can't call it "industry standard" yet but the goal is obviously for it to become one.
- ryao 2y ago
  I wrote: > It is weird to see the industry invent yet another standard to do effectively the same thing just because Nvidia is using it. This is a misstep for all involved, AMD included. Even if AMD is following everyone else by jumping off a bridge, AMD is still jumping too.
croes 2y ago

dhruvdh 2y ago

Disappointed that there wasn’t anything on inference performance in the article at all. That’s what the major customers have announced they use it for.

nromiun 2y ago

So basically Nvidia is like Windows and AMD is like Wine. I think trying to emulate CUDA and using forked Nvidia libraries is not the best strategy for AMD. They should have made a clean break and come out with a fresh API, like Apple's Metal.

benreesman 2y ago

AMD could spend their market cap in one year to get this done in three and it would be a coup for the shareholders. They could hire all of the best NVIDIA engineers at double their current comp, crush the next TSMC node on Apple levels, and just do it and if it got them a quarter of NVDA’s cap it would be a bargain. They don’t fucking want to! Believing this is anything like a market is fucking religion.

robocat 2y ago
Try to make sense... They can spend their market cap by either: 1: issuing new shares worth their market cap, diluting existing shareholders to 50%. 2: Or borrow their market cap and pay interest by decreasing profits. "AMD operating margin for the quarter ending September 30, 2024 was 5.64%" so profits would be extremely impacted by interest repayments. Either way your suggestion would be unlikely to be supported by shareholders. > crush the next TSMC node on Apple levels I would guess Apple is indirectly paying for the hardware (to avoid repatriating profits) or guaranteeing usage to get to the front of the line at TSMC. Good luck AMD competing with Apple: there's a reason AMD sold GlobalFoundries and there's a reason Intel is now struggling with their foundry costs. And it comes across as condescending to assume you know better than a successful company.
- benreesman 2y ago
  When 4 trillion dollars are at stake, the financing is available or it fucking better be. What in God’s name do we pay these structured finance, bond-issue assholes 15% of GDP for if not to finance a sure thing like that? It sure as hell ain’t for their taste in Charvet and Hermes ties, because the ones they pick look like shit.
atq2119 2y ago
You make it sound like that's a sure thing, but I doubt it. A lot of this is about processes, team structures and incentives, all those fuzzy things between the people. Remember, most acquisitions fail. For the same reason, the likelihood of failure with your scenario seems high. Do you really think nobody at AMD is aware of all the points made in this thread? That seems too bizarre to be true. There are probably some issues in upper management which could perhaps be fixed with some targeted hiring decisions, but do you really believe some random person on here would have a chance making that call?

anon291 2y ago

My anecdata on AMD hiring: they just aren't moving fast enough. They still wanted to fly people out scheduling 3 weeks in advance for AI compiler work. That's just not going to work. Startups and companies like NVIDIA, OpenAI are hiring much faster with much less onerous interview processes, with higher compensation. This is not a mystery. People work for money and aren't going to hop through more hoops to be paid less.

cavisne 2y ago

I think it’s hardware not software. “Cuda moat” is a misnomer. The PTX spec is relatively short (600 page pdf). Triton directly writes PTX, skipping cuda. Flash attention was created by a non nvidia employee without access to any of the secret sauce within Cuda or its libraries. The hardware is just not as good, and no software can paper over its flaws.

pjmlp 2y ago
NVidia also refactored their hardware design to follow C++ memory model, I think this still isn't the case of others.

MortyWaves 2y ago

Wow they really botched the title. Why wasn’t it “Still Filled With Water”?

klelatti 2y ago

AMD doesn’t just have to fix these issues it has to build up a record of fixing issues like those discussed here. Otherwise who will bet their firm / cash / career on new hardware without a successful track record.

yaro330 2y ago

TL;DR: AMD still doesn't take software seriously?

pella 2y ago

Latest: Dylan Patel ( SemiAnalysis ) "Met with @LisaSu today for 1.5 hours as we went through everything She acknowledged the gaps in AMD software stack She took our specific recommendations seriously She asked her team and us a lot of questions Many changes are in flight already! Excited to see improvements coming" https://x.com/dylan522p/status/1871287937268383867 https://x.com/dylan522p/status/1871287937268383867

pella 2y ago
Lisa Su : https://x.com/LisaSu/status/1871362304194859511 https://x.com/LisaSu/status/1871362304194859511 "Thanks @dylan522p for the constructive conversation today. Feedback is a gift even when it’s critical. We have put a ton of work into customer and workload optimizations but there is lots more we can do to enable the broad ecosystem. I appreciate all the feedback and desire to engage with @AMD. We are committed to building a world-class open software stack. Lots planned for 2025. Happy holidays to all!"

varelse 2y ago

[dead]