25 ms·

I wish this were so, but I'm very, very sceptical. Everything I use is built to CUDA. Nothing actually works on non-NVidia hardware. "Actually works" is differe

by blagie 4y ago

I wish this were so, but I'm very, very sceptical. Everything I use is built to CUDA. Nothing actually works on non-NVidia hardware. "Actually works" is different from "theoretically works." A lot of what I use theoretically supports ROCm, but in practice, when you try to use it, you have enough bugs big and small, and things crash or don't work right. That means everything I build is locked to NVidia as well, and everything people build on top of my stuff will be locked to NVidia too. Trying to maintain compatibility is like making Wine run Windows apps natively. The lock-in is now very significant. Virtually no one else has shipped anything really stable and usable, and we're now a very long time into people coding and testing only on NVidia. I hear that higher-level APIs are being built, nominally intended to be cross-platform, but I suspect (1) those will go the same way (2) we'll want backwards-compatibility. I have code which uses old ML models, CuPy, or otherwise. That stuff is locked to NVidia. I'm buying NVidia. This means I'm only making patches to bugs impacting NVidia. Network effects. So long as most people buy NVidia, NVidia will have more engineering resources too. It's very hard for a smaller player to catch up or pass. See Intel for many decades. To break from NVidia would require either a gross failure in execution by NVidia or something fundamentally new. To break the monopoly, I could see new products which might deliver: - Very low latency - Support for conventional DDR in large quantity (so I can run GPT-3 models at home) - Much more general functionality (e.g. an open architecture, and integration with conventional Python to parallelize normal code) ... or other significant new features. However, I don't see a lot of stuff like that being worked on, let alone coming out.

fragmede 4y ago
Critically, there's also cost. The formula is $W Nvidia for N-flops vs $X ROCm for N-flops + $Y engineering cost of recoding. If $W > $X + $Y, then people will go with AMD.
- sokoloff 4y ago
  That's roughly right only if the people paying $W vs $X and $Y are the same people. The people paying $X and the people paying $Y are often not the same people across large swaths of the GPU-buying ecosystem. ($W and $X also change over time and with volume of units shipped and volume shipped varies with how useful and widely applicable the cards are.) I'd love to get an AMD [6650XT-ish] for my teen's gaming PC. But they also do some programming with me and I slightly worry about "oh, your PC can't do CUDA, therefore can't do XYZ that's interesting to you". I probably will still just go AMD and then have us use rented GPUs if we want to do any CUDA demos together.
  - Zach_the_Lizard 4y ago
    While the hardware and software budgets are not the same, some companies do have tricks to make the software team care about hardware at least in theory. I have never ordered any server hardware, but there's a dashboard where I can see some abstracted infrastructure CAPEX and OPEX numbers for my team, org, division, etc. These numbers get reported up the food chain and initiatives to reduce costs are rewarded if the savings are significant enough. Initiatives to support ARM servers or AMD's CUDA equivalents can take a long time and be expensive to achieve full compatibility, but experiments in areas where the effort is not terribly high can demonstrate enough value to fund teams to do so. I suspect that's most difficult in mid sized organizations where the hardware costs are high enough to get some savings but not high enough to fund a team to work on it, especially once the opportunity cost of new features is taken into account. Universities, mid sized companies, big companies without a software focus, etc. spring to mind as the ones with the biggest disconnect between hardware costs and ability to fund engineers to cut them down.
- blagie 4y ago
  For the long tail of ML uses, $Y dominates. That's actually most of the market. AMD will do fine for 3d gaming, stable diffusion, and a few similar uses. And price/performance wins for giant data centers doing something specialized (but if NVidia is about to lose a major data center deal because $W > $X + $Y, they can just cut a deal on $W).
abstractcontrol 4y ago
Another thing that could happen is a fundamental algorithmic advance to replace backprop. The brain doesn't use it, and backprop doesn't have answers for how to do long term credit assignment and continuous learning. It works poorly with RL as well. Better algorithms would require writing new ML libraries which would weaken the stranglehold GPUs have on ML. New niches opening would inevitably be bad for Nvidia, but good for upstarts.
- webmaven 4y ago
  Geoffrey Hinton recently came up with a new "forward forward" algorithm that seems to fit the bill: https://bdtechtalks.com/2022/12/19/forward-forward-algorithm-geoffrey-hinton/ https://bdtechtalks.com/2022/12/19/forward-forward-algorithm...
sharpneli 4y ago
AMD has a really weird view on how the programming world works internally. Some years ago they had some ARM server chips. Did they allow people access? No. They only allowed it for "Serious customers" and the whole thing failed as no software was available for it. Same applies to their compute stuff. They really really want you to purchase their Pro cards and whatnot, and if you have a consumer card well too bad. Go back to playing games. Whereas with my Nvidia card CUDA runs flawlessly which leads me to doing test projects at home and fixing a bug here and there. AMD doesn't understand how much of development happens on either home computers or university labs with small budgets which means consumer cards. They seem to think that everything happens using big teams and big money. Even though in reality it's the small developers doing stuff that then drives the purchase of the very lucrative large clusters. For all of it's downsides NVidia actually understand the grassroots approach. Which is why their stuff just works on consumer stuff too.
- ChuckNorris89 4y ago
  I feel there's a large disconnect at AMD between the CPU division and the RTG (Radeon Technologies Group) which feels more insular than the rest of the company. I feel Lisa Su had her hand deeper in propping up the wailing CPU division and kind of forgot about RTG that's been loosing market share to Nvidia for the last 10 years not just in gaming but in ML. Maybe she should take a break from Ryzen and look into leading the RTG.
- aseipp 4y ago
  My impression is that they largely seem to focus on Big Money HPC contracts and other very large deals where bespoke engineering on the software stack is worth their time and can be paid off. So if you're paying them a shitload of money in a massive supercomputing-style deal, they'll get you software for their otherwise-unobtainium accelerators. But normal "day to day" stuff is literally not on their radar. At best it's an afterthought. I think the timing of when they entered into the field is also relevant. When Nvidia first came on the scene swinging, they really needed to see what would stick; so you have to try fishing in every available pond. You don't know what will necessarily pan out or catch on. So you need the tech to be available to all your consumers at every level, consumer, enterprise, HPC, everything. And it turns out, they found that there are customers at every single one of those levels, and providing for all of them provides both a good on-ramp, and a path to bigger things. In contrast AMD is so late to the game, and there's so much money sloshing around in the field right now, and the trajectories of what big buyers want is so much more clear -- that they can just afford to mostly ignore the lower end. They know where the money is. The field and buyers and use cases are just much more well understood from the business "make money in the easiest way" POV. As for Intel, they seem to actually be dedicating tons of money to the software stack for oneAPI, to work everywhere, which is not surprising IMO. They have the money to dump into this (the most money intensive) task and know it's critical for their products to reach anything other than pork-belly HPC contracts.
danaris 4y ago
I'm only an interested observer of the space, but a bunch of work is being done getting at least some of this stuff to run on Apple Silicon in recent months. I've been running Stable Diffusion on my M1 MBP since...I dunno, November or something? and it's working great.
- kalium-xyz 4y ago
  Slow as heck tho compared to on a dedicated card. I was running it on my M1 pro till I bought some old miners to resell and tried out a 3070. That thing gets me a 512 x 512 DDIM 20 steps photorealistic image with protogen 5.3 in 2.85 seconds. Its insane to watch it churn out images like that. My on my MPB I had to downgrade to the older SD model that was trained only one 256 x 256 to get decent feedback on my prompts.
  - danaris 4y ago
    Really? Mine usually only takes about 30s per image for 50-step 512x512 images (which is perfectly reasonable for my purposes)...I wonder if it has to do with the RAM (I've got an M1 Max with 64GB)? Or maybe I'm just not doing it enough to care about the speed so much.
    - kalium-xyz 4y ago
      Compared to the 2.85 seconds on my dedicated card? Yes 30 seconds is slow as heck. Youre talking about 10 times longer and waiting this time to adjust your prompt. I dont find that acceptable for a workflow where I need to tweak prompts.
  - giovannibonetti 4y ago
    > Slow as heck tho compared to on a dedicated card. It's important to mention that there is a huge difference in performance between a new MacBook with 8 GB or 16 GB of RAM when it comes to this memory-intensive ML tasks.
    - kalium-xyz 4y ago
HansHamster 4y ago
> Nothing actually works on non-NVidia hardware. "Actually works" is different from "theoretically works." A lot of what I use theoretically supports ROCm, but in practice, when you try to use it, you have enough bugs big and small, and things crash or don't work right. I was surprised when I started playing around with Stable Diffusion a couple of weeks ago and it just kind of worked with ROCm. That being said, I did run into a couple of very annoying issues... almost running out of VRAM seems to trigger GPU timeouts but as far as I can tell the GPU itself is fine and the memory allocation crawled to a halt. This also seems to be much more likely when running at half precision, which is a shame given that the relatively large 16GB VRAM effectively gets cut in half when running at full precision... But this is already a big improvement over "ROCm doesn't support RDNA2" [0] which was still the case ~1.5 years ago. I just really hope that AMD keeps up the effort and fixes all the small and big issues that are still around... [0]: https://github.com/RadeonOpenCompute/ROCm/issues/1180 https://github.com/RadeonOpenCompute/ROCm/issues/1180
- blagie 4y ago
  > I was surprised when I started playing around with Stable Diffusion a couple of weeks ago and it just kind of worked with ROCm. Stable Diffusion is literally the single highest-profile application of machine learning. There has been extensive effort to make it work in all sorts of places. If it were the year 2000, someone would port it to a Beowulf cluster... I'm more concerned about SpaCy, CuPy, the hundreds (thousands) of models on Hugging Face, etc. I do use a lot of that stuff. Buying a GPU or accelerator which only runs the high-profile stuff is a bad buy.
  - HansHamster 4y ago
    The other models on Hugging Face I tried also worked fine... I'm not saying that everything is running perfectly or even that buying an AMD card for compute is a good idea at the moment, but there has been quite some progress even compared to 2-3 years ago so I'm cautiously optimistic... I still remember trying to train a model with TensorFlow on AMD maybe 2 years ago. The only option was OpenCL and it was disappointingly slow (and barely any load on the GPU) and in the end I switched back to my old 1060 which worked much better.
Aromasin 4y ago
I recently migrated a lot of my CUDA code to SYCL using Intel's compatibility tool [1]. I'm trying to get my applications to work irregardless of hardware, so it simply needed to be done despite the pain. It actually wasn't as hard as I was expecting. Don't get me wrong, it will definitely depend on the size of your code base as to how effective of a tool it is. However, I managed to port over most of mine in the course of the month, and it's showing dividends now I'm making back performance gains using FPGA hardware acceleration. I'm now vendor neutral, and can run my code on team Red, Blue, or Green. I'd argue that heterogeneous computing is a good enough reason to make the switch. [1]https://www.intel.com/content/www/us/en/develop/documentation/oneapi-programming-guide/top/software-development-process/migrating-code-to-dpcpp/migrating-from-cuda-to-dpcpp.html https://www.intel.com/content/www/us/en/develop/documentatio... [2]https://www.khronos.org/sycl/ https://www.khronos.org/sycl/

by blagie 4y ago

fragmede 4y ago

Critically, there's also cost. The formula is $W Nvidia for N-flops vs $X ROCm for N-flops + $Y engineering cost of recoding. If $W > $X + $Y, then people will go with AMD.

sokoloff 4y ago
That's roughly right only if the people paying $W vs $X and $Y are the same people. The people paying $X and the people paying $Y are often not the same people across large swaths of the GPU-buying ecosystem. ($W and $X also change over time and with volume of units shipped and volume shipped varies with how useful and widely applicable the cards are.) I'd love to get an AMD [6650XT-ish] for my teen's gaming PC. But they also do some programming with me and I slightly worry about "oh, your PC can't do CUDA, therefore can't do XYZ that's interesting to you". I probably will still just go AMD and then have us use rented GPUs if we want to do any CUDA demos together.
- Zach_the_Lizard 4y ago
  While the hardware and software budgets are not the same, some companies do have tricks to make the software team care about hardware at least in theory. I have never ordered any server hardware, but there's a dashboard where I can see some abstracted infrastructure CAPEX and OPEX numbers for my team, org, division, etc. These numbers get reported up the food chain and initiatives to reduce costs are rewarded if the savings are significant enough. Initiatives to support ARM servers or AMD's CUDA equivalents can take a long time and be expensive to achieve full compatibility, but experiments in areas where the effort is not terribly high can demonstrate enough value to fund teams to do so. I suspect that's most difficult in mid sized organizations where the hardware costs are high enough to get some savings but not high enough to fund a team to work on it, especially once the opportunity cost of new features is taken into account. Universities, mid sized companies, big companies without a software focus, etc. spring to mind as the ones with the biggest disconnect between hardware costs and ability to fund engineers to cut them down.
blagie 4y ago
For the long tail of ML uses, $Y dominates. That's actually most of the market. AMD will do fine for 3d gaming, stable diffusion, and a few similar uses. And price/performance wins for giant data centers doing something specialized (but if NVidia is about to lose a major data center deal because $W > $X + $Y, they can just cut a deal on $W).

abstractcontrol 4y ago

Another thing that could happen is a fundamental algorithmic advance to replace backprop. The brain doesn't use it, and backprop doesn't have answers for how to do long term credit assignment and continuous learning. It works poorly with RL as well. Better algorithms would require writing new ML libraries which would weaken the stranglehold GPUs have on ML. New niches opening would inevitably be bad for Nvidia, but good for upstarts.

webmaven 4y ago
Geoffrey Hinton recently came up with a new "forward forward" algorithm that seems to fit the bill: https://bdtechtalks.com/2022/12/19/forward-forward-algorithm-geoffrey-hinton/ https://bdtechtalks.com/2022/12/19/forward-forward-algorithm...

sharpneli 4y ago

AMD has a really weird view on how the programming world works internally. Some years ago they had some ARM server chips. Did they allow people access? No. They only allowed it for "Serious customers" and the whole thing failed as no software was available for it. Same applies to their compute stuff. They really really want you to purchase their Pro cards and whatnot, and if you have a consumer card well too bad. Go back to playing games. Whereas with my Nvidia card CUDA runs flawlessly which leads me to doing test projects at home and fixing a bug here and there. AMD doesn't understand how much of development happens on either home computers or university labs with small budgets which means consumer cards. They seem to think that everything happens using big teams and big money. Even though in reality it's the small developers doing stuff that then drives the purchase of the very lucrative large clusters. For all of it's downsides NVidia actually understand the grassroots approach. Which is why their stuff just works on consumer stuff too.

ChuckNorris89 4y ago
I feel there's a large disconnect at AMD between the CPU division and the RTG (Radeon Technologies Group) which feels more insular than the rest of the company. I feel Lisa Su had her hand deeper in propping up the wailing CPU division and kind of forgot about RTG that's been loosing market share to Nvidia for the last 10 years not just in gaming but in ML. Maybe she should take a break from Ryzen and look into leading the RTG.
aseipp 4y ago
My impression is that they largely seem to focus on Big Money HPC contracts and other very large deals where bespoke engineering on the software stack is worth their time and can be paid off. So if you're paying them a shitload of money in a massive supercomputing-style deal, they'll get you software for their otherwise-unobtainium accelerators. But normal "day to day" stuff is literally not on their radar. At best it's an afterthought. I think the timing of when they entered into the field is also relevant. When Nvidia first came on the scene swinging, they really needed to see what would stick; so you have to try fishing in every available pond. You don't know what will necessarily pan out or catch on. So you need the tech to be available to all your consumers at every level, consumer, enterprise, HPC, everything. And it turns out, they found that there are customers at every single one of those levels, and providing for all of them provides both a good on-ramp, and a path to bigger things. In contrast AMD is so late to the game, and there's so much money sloshing around in the field right now, and the trajectories of what big buyers want is so much more clear -- that they can just afford to mostly ignore the lower end. They know where the money is. The field and buyers and use cases are just much more well understood from the business "make money in the easiest way" POV. As for Intel, they seem to actually be dedicating tons of money to the software stack for oneAPI, to work everywhere, which is not surprising IMO. They have the money to dump into this (the most money intensive) task and know it's critical for their products to reach anything other than pork-belly HPC contracts.

danaris 4y ago

I'm only an interested observer of the space, but a bunch of work is being done getting at least some of this stuff to run on Apple Silicon in recent months. I've been running Stable Diffusion on my M1 MBP since...I dunno, November or something? and it's working great.

kalium-xyz 4y ago
Slow as heck tho compared to on a dedicated card. I was running it on my M1 pro till I bought some old miners to resell and tried out a 3070. That thing gets me a 512 x 512 DDIM 20 steps photorealistic image with protogen 5.3 in 2.85 seconds. Its insane to watch it churn out images like that. My on my MPB I had to downgrade to the older SD model that was trained only one 256 x 256 to get decent feedback on my prompts.
- danaris 4y ago
  Really? Mine usually only takes about 30s per image for 50-step 512x512 images (which is perfectly reasonable for my purposes)...I wonder if it has to do with the RAM (I've got an M1 Max with 64GB)? Or maybe I'm just not doing it enough to care about the speed so much.
  - kalium-xyz 4y ago
    Compared to the 2.85 seconds on my dedicated card? Yes 30 seconds is slow as heck. Youre talking about 10 times longer and waiting this time to adjust your prompt. I dont find that acceptable for a workflow where I need to tweak prompts.
- giovannibonetti 4y ago
  > Slow as heck tho compared to on a dedicated card. It's important to mention that there is a huge difference in performance between a new MacBook with 8 GB or 16 GB of RAM when it comes to this memory-intensive ML tasks.
  - kalium-xyz 4y ago

HansHamster 4y ago

> Nothing actually works on non-NVidia hardware. "Actually works" is different from "theoretically works." A lot of what I use theoretically supports ROCm, but in practice, when you try to use it, you have enough bugs big and small, and things crash or don't work right. I was surprised when I started playing around with Stable Diffusion a couple of weeks ago and it just kind of worked with ROCm. That being said, I did run into a couple of very annoying issues... almost running out of VRAM seems to trigger GPU timeouts but as far as I can tell the GPU itself is fine and the memory allocation crawled to a halt. This also seems to be much more likely when running at half precision, which is a shame given that the relatively large 16GB VRAM effectively gets cut in half when running at full precision... But this is already a big improvement over "ROCm doesn't support RDNA2" [0] which was still the case ~1.5 years ago. I just really hope that AMD keeps up the effort and fixes all the small and big issues that are still around... [0]: https://github.com/RadeonOpenCompute/ROCm/issues/1180 https://github.com/RadeonOpenCompute/ROCm/issues/1180

blagie 4y ago
> I was surprised when I started playing around with Stable Diffusion a couple of weeks ago and it just kind of worked with ROCm. Stable Diffusion is literally the single highest-profile application of machine learning. There has been extensive effort to make it work in all sorts of places. If it were the year 2000, someone would port it to a Beowulf cluster... I'm more concerned about SpaCy, CuPy, the hundreds (thousands) of models on Hugging Face, etc. I do use a lot of that stuff. Buying a GPU or accelerator which only runs the high-profile stuff is a bad buy.
- HansHamster 4y ago
  The other models on Hugging Face I tried also worked fine... I'm not saying that everything is running perfectly or even that buying an AMD card for compute is a good idea at the moment, but there has been quite some progress even compared to 2-3 years ago so I'm cautiously optimistic... I still remember trying to train a model with TensorFlow on AMD maybe 2 years ago. The only option was OpenCL and it was disappointingly slow (and barely any load on the GPU) and in the end I switched back to my old 1060 which worked much better.

Aromasin 4y ago

I recently migrated a lot of my CUDA code to SYCL using Intel's compatibility tool [1]. I'm trying to get my applications to work irregardless of hardware, so it simply needed to be done despite the pain. It actually wasn't as hard as I was expecting. Don't get me wrong, it will definitely depend on the size of your code base as to how effective of a tool it is. However, I managed to port over most of mine in the course of the month, and it's showing dividends now I'm making back performance gains using FPGA hardware acceleration. I'm now vendor neutral, and can run my code on team Red, Blue, or Green. I'd argue that heterogeneous computing is a good enough reason to make the switch. [1]https://www.intel.com/content/www/us/en/develop/documentation/oneapi-programming-guide/top/software-development-process/migrating-code-to-dpcpp/migrating-from-cuda-to-dpcpp.html https://www.intel.com/content/www/us/en/develop/documentatio... [2]https://www.khronos.org/sycl/ https://www.khronos.org/sycl/