10 ms·

If you want to go an run inference/train on one of these fresh and sweet LLMs using bitsandbytes/PEFT, which is really where the excitement is at, you gotta use

by zwaps 3y ago

If you want to go an run inference/train on one of these fresh and sweet LLMs using bitsandbytes/PEFT, which is really where the excitement is at, you gotta use CUDA pretty much. This is the story now. And the story for the innovation before was the same. Use CUDA, or wait for AMD to catch up a year late and with a worse version of everything. I mean, sure, you could compile your stuff to XLA or just, I dunno, set up 800 of these cards and train the whole thing on ROCM. But then would you really, really use AMD instead of some TPUs? AMD made Machine Learning either impossible, unsupported or a chore on most of their hardware for years. Their stack sucks and no one wanted to implement it, and in fact they rarely supported most of their own GPUs. Yes you have GPUs. But we need also drivers. And software. People have been yelling at AMD about this for years and years. Instead, AMD has made it clear that Machine Learning is not a priority for the company. Hence, they deserve the lack of traction. Investing in AMD hardware for ML has literally been a mistake at every point in recent history. Imagine if you bought a bunch of (insert last Gen card which is no longer supported by their stack) how dumb you'd look. Releasing a GPGPU card now? Honestly, why bother? No one is gonna buy it.

stevenhuang 3y ago
Been playing with Stable Diffusion on 6750XT 12GB for a while now with minimal issues (low memory issues sometimes on higher res, but it's less of a problem using grid upscaler). ROCm is getting there, slowly.
- MINIMAN10000 3y ago
  Also good news was that running olive and downloading the ONNX converter to run stable diffusion shows something like 120% speedup on Nvidia and 50% speedup on AMD from the numbers I see. Some UIs are beginning to support ONNX format. ONNX being the format that has Microsoft's support behind it for AI models to my understanding.
thereisnospork 3y ago
This is a naive question, but how hard would it be for AMD to make their cards/firmware CUDA compatible? Feels like that's what they would need to do to sell hardware in the space, other than banking on sufficiently severe shortages.
- JonChesterfield 3y ago
  Cuda runs in the layer above firmware. Compiling cuda to amdgpu isa could be done but might invite lawsuits. There's a language called HIP which is a fairly close approximation to cuda. You can probably convert one to the other with determination and regex. The GPUs themselves are fundamentally different in ways that hopefully don't matter to your application (warp synchronisation is the big one in my opinion, but I suspect cuda applications ignore it and just live with the race conditions).
mindcrime 3y ago
There's more to machine learning and AI than just training the latest LLM's though. Speaking for myself, I am very interested in supporting AMD and the ROCm ecosystem, whatever AMD's past sins may be. I'm building a machine now which will be based on a high-end consumer GPU from AMD. Not going for something like this as it's almost certainly way out of my budget, but perhaps in the future. So basically, I'm betting that AMD has had (or is having) a change of heart and is genuinely committed to AI/ML on their products. Time may ultimately prove me wrong, but so be it if that proves to be the case. And FWIW, one reason for my optimism is that, whatever you think about the state of ROCm today, they are clearly investing heavily into the platform and continually working on it. You can see that just from looking at the activity on their Github repos: https://github.com/orgs/ROCmSoftwarePlatform/repositories https://github.com/orgs/ROCmSoftwarePlatform/repositories There is constant activity and has been for some time, which I take as a good sign. Yes, it's just one signal among many that one could consider, but I think it's an important one.
- zwaps 3y ago
  Yeah but you see, we have been here before. "This time we are serious about ML/AI..." And if you went and bought an AMD card then, you'd have been wrong. My example about LLMs was just to show that AMD is simply not part of the conversation. Three month before you could have made the same point about another approach. And still, if you'd actually have to risk money, you and me both know you'd never invest in AMD hardware for AI or start developing on it something high stakes. I mean, look at geohot he tried and just gave up entirely and AMD.
  - mindcrime 3y ago
    And still, if you'd actually have to risk money, you and me both know you'd never invest in AMD hardware for AI or start developing on it something high stakes. I assume you meant "you" in the royal sense. Since I personally am, in point of fact, investing in AMD hardware for AI. Yeah, it's a gamble, but that's my style. And I have to admit, part of it is driven by ideological reasons (ROCm being open source) and a simple desire to support AMD since I want them to become a serious competitor to NVIDIA. I believe that's an outcome that would benefit everybody. I mean, look at geohot he tried and just gave up entirely and AMD. I have to admit, I don't find that particularly compelling in any regard. Nothing against geohot, he's clearly a smart dude, but... I'm not judging a company based on his interactions with them. shrug
jiggawatts 3y ago
Can someone who works at AMD please print this out, roll it up, and smack some senior managers on the nose with it? NVIDIA is about to walk off with a trillion dollars because nobody at AMD “gets it”. With no meaningful competition, NVIDIA will gouge as hard as they can. Such as charging $50K for a card that’s not too different to a 4090 but with more memory.
- mindcrime 3y ago
  NVIDIA is about to walk off with a trillion dollars because nobody at AMD “gets it”. I think they "get it" OK. Whether or not they can formulate a viable strategy and execute it is one question, but they get the idea that "AI is important" and they know where they stand vis-a-vis NVIDIA. This is another reason I'm willing to invest some time and money in working with AMD products for AI/ML. History has shown us their ability to go toe-to-toe with a seemingly unassailable industry titan before, and they came out in pretty good shape then. https://www.forbes.com/sites/iainmartin/2023/05/31/lisa-su-saved-amd-now-she-wants-nvidias-ai-crown/ https://www.forbes.com/sites/iainmartin/2023/05/31/lisa-su-s...
- JonChesterfield 3y ago
  AMD took aim at HPC and shipped Frontier. The ROCm stack is quite HPC themed because that was the driving project. Porting to run AI models is a work in progress but the back end is the same compiler stack that was written for graphics (largely games consoles iiuc) then upgraded for HPC, it'll get there.

by zwaps 3y ago

stevenhuang 3y ago

Been playing with Stable Diffusion on 6750XT 12GB for a while now with minimal issues (low memory issues sometimes on higher res, but it's less of a problem using grid upscaler). ROCm is getting there, slowly.

MINIMAN10000 3y ago
Also good news was that running olive and downloading the ONNX converter to run stable diffusion shows something like 120% speedup on Nvidia and 50% speedup on AMD from the numbers I see. Some UIs are beginning to support ONNX format. ONNX being the format that has Microsoft's support behind it for AI models to my understanding.

thereisnospork 3y ago

This is a naive question, but how hard would it be for AMD to make their cards/firmware CUDA compatible? Feels like that's what they would need to do to sell hardware in the space, other than banking on sufficiently severe shortages.

JonChesterfield 3y ago
Cuda runs in the layer above firmware. Compiling cuda to amdgpu isa could be done but might invite lawsuits. There's a language called HIP which is a fairly close approximation to cuda. You can probably convert one to the other with determination and regex. The GPUs themselves are fundamentally different in ways that hopefully don't matter to your application (warp synchronisation is the big one in my opinion, but I suspect cuda applications ignore it and just live with the race conditions).

mindcrime 3y ago

There's more to machine learning and AI than just training the latest LLM's though. Speaking for myself, I am very interested in supporting AMD and the ROCm ecosystem, whatever AMD's past sins may be. I'm building a machine now which will be based on a high-end consumer GPU from AMD. Not going for something like this as it's almost certainly way out of my budget, but perhaps in the future. So basically, I'm betting that AMD has had (or is having) a change of heart and is genuinely committed to AI/ML on their products. Time may ultimately prove me wrong, but so be it if that proves to be the case. And FWIW, one reason for my optimism is that, whatever you think about the state of ROCm today, they are clearly investing heavily into the platform and continually working on it. You can see that just from looking at the activity on their Github repos: https://github.com/orgs/ROCmSoftwarePlatform/repositories https://github.com/orgs/ROCmSoftwarePlatform/repositories There is constant activity and has been for some time, which I take as a good sign. Yes, it's just one signal among many that one could consider, but I think it's an important one.

zwaps 3y ago
Yeah but you see, we have been here before. "This time we are serious about ML/AI..." And if you went and bought an AMD card then, you'd have been wrong. My example about LLMs was just to show that AMD is simply not part of the conversation. Three month before you could have made the same point about another approach. And still, if you'd actually have to risk money, you and me both know you'd never invest in AMD hardware for AI or start developing on it something high stakes. I mean, look at geohot he tried and just gave up entirely and AMD.
- mindcrime 3y ago
  And still, if you'd actually have to risk money, you and me both know you'd never invest in AMD hardware for AI or start developing on it something high stakes. I assume you meant "you" in the royal sense. Since I personally am, in point of fact, investing in AMD hardware for AI. Yeah, it's a gamble, but that's my style. And I have to admit, part of it is driven by ideological reasons (ROCm being open source) and a simple desire to support AMD since I want them to become a serious competitor to NVIDIA. I believe that's an outcome that would benefit everybody. I mean, look at geohot he tried and just gave up entirely and AMD. I have to admit, I don't find that particularly compelling in any regard. Nothing against geohot, he's clearly a smart dude, but... I'm not judging a company based on his interactions with them. shrug

jiggawatts 3y ago

Can someone who works at AMD please print this out, roll it up, and smack some senior managers on the nose with it? NVIDIA is about to walk off with a trillion dollars because nobody at AMD “gets it”. With no meaningful competition, NVIDIA will gouge as hard as they can. Such as charging $50K for a card that’s not too different to a 4090 but with more memory.

mindcrime 3y ago
NVIDIA is about to walk off with a trillion dollars because nobody at AMD “gets it”. I think they "get it" OK. Whether or not they can formulate a viable strategy and execute it is one question, but they get the idea that "AI is important" and they know where they stand vis-a-vis NVIDIA. This is another reason I'm willing to invest some time and money in working with AMD products for AI/ML. History has shown us their ability to go toe-to-toe with a seemingly unassailable industry titan before, and they came out in pretty good shape then. https://www.forbes.com/sites/iainmartin/2023/05/31/lisa-su-saved-amd-now-she-wants-nvidias-ai-crown/ https://www.forbes.com/sites/iainmartin/2023/05/31/lisa-su-s...
JonChesterfield 3y ago
AMD took aim at HPC and shipped Frontier. The ROCm stack is quite HPC themed because that was the driving project. Porting to run AI models is a work in progress but the back end is the same compiler stack that was written for graphics (largely games consoles iiuc) then upgraded for HPC, it'll get there.