4 ms·

I don't quite understand. Most of LLM inference is MatMul and Softmax. It's not rocket science to implement matrix multiplication in any GPU. We don't have very

by smy20011 3y ago

I don't quite understand. Most of LLM inference is MatMul and Softmax. It's not rocket science to implement matrix multiplication in any GPU. We don't have very complex custom kernel for the LLM inference. If AMD can provide fast GPU with cheaper price, a lot of people don't mind to add support for them. GGML shows you can create a working LLM inference solution with one man's work.

whalesalad 3y ago
Everyone has been writing cuda code. Hopefully we continue the shift towards using higher level frameworks/abstractions that can determine the optimum machine code for the underlying hardware. I don’t really get it either. Nvidia is pouring $$$ into protecting their advantage as anemic as it might be. They give free compute and gpu resources to universities etc to cement the dependence on cuda. When you’re a beginner or someone who needs to just “get shit done” you’ll go nvidia at the expense of the greater good.
- dannyw 3y ago
  Also, the full consumer (RTX) stack is supported. Lots of people who are gamers, or want a hobbyist rig can go out to the shops and start with a RTX. AMD's ML / ROCm support for their consumer cards is still awful.
  - plasticchris 3y ago
    They did add official support for the latest gen rdna cards to rocm.
    - ekianjo 3y ago
      Why only the latest gen?
      - svnt 3y ago
        Because they want to recover their investment
        aryonoco 3y ago
        Actually because ROCm ships native machine code, as opposed to a bytecode, so supporting any additional architecture is a tonne of effort. I expect support will eventually be extended to a generation or two prior GPUs, but nothing beyond that.
    - dannyw 3y ago
      Not on Linux!
      - plasticchris 3y ago
        What? It was only for Linux. https://rocm.docs.amd.com/projects/radeon/en/latest/index.html https://rocm.docs.amd.com/projects/radeon/en/latest/index.ht...
- pavelstoev 3y ago
  Try Hidet - open source deep learning compiler written in Python, generating optimized CUDA https://pypi.org/project/hidet/ https://pypi.org/project/hidet/
- jakderrida 3y ago
  > They give free compute and gpu resources to universities etc to cement the dependence on cuda. It's actually quite brilliant. It never occurred to me before that the resources we had access to at university weren't the most popular because that's what the library wanted. They were the most popular because the provider made them cheap/free to widen their moat over the competition. It reminds me of Thomson Reuters launching Eikon, a superior alternative (imho) to Bloomberg terminals. Every business school has Bloomberg terminals. But few/none had Eikon, which is why nobody knows how to use one.
  - kimixa 3y ago
    It's the same reason that students get free MSDN access and unlimited windows keys. It's a pretty solid strategy, but naturally favors the current incumbent who has the $$$ to throw around.
- anvuong 3y ago
  What the hell is the greater good here? I'm a researcher in a R1 university, I can quickly try stupid things on my personal 3080, then if it works I can virtually git clone the codes onto my lab's server and bam it chops through the problems. After a few weeks I'll get (hopefully) some publisable results that (hopefully) advance my field a little. Do you really expect most researchers to use half of their time debugging the unfinished ROCM stack that is just not there, sacrificing their output, and maybe even their tenure prospect? We love Nvidia solutions, at least in our field, not because it's a monopoly, but because it's actually the best solution right now.
  - donor20 3y ago
    I really like that you can prototype on your own consumer level card - the rocm stack has not matched that historically and has over promised and underdelivered for years now.
    - cyanydeez 3y ago
      It's getting closer, and well, what matters is price points and what, billions of people crunching away on the various problems. There's huge momentum on the "user" side of the equation that just getting cheap hardware and passable performance means Nvidia losing dominance.
- nemothekid 3y ago
  >Nvidia is pouring $$$ into protecting their advantage as anemic as it might be. This description isn't fair. Nvidia was pouring money into CUDA for 16 years. AMD/Intel only woke up once ChatGPT launched and nvidia started printing money. It's not fair to call it "protecting their advantage" when nvidia invested into the ecosystem for years and AMD couldn't even bother to ship working examples with OpenCL. I'm not sure why choosing the company that decided to invest in the platform for years is at the expense of the "greater good." What "greater good" is there above actually working software?
  - bryanlarsen 3y ago
    Nvidia had boatloads of money to spend, and 6X the number of engineers. AMD was skating on the edge of bankruptcy. AMD bet the farm on Zen, and that bet paid off so now they have money to spend on other things. If they would have split resources off of Zen to spend on CUDA they would have failed at both and AMD would be a bankrupt husk.
  - cyanydeez 3y ago
    You can get a consumer laptop with AI: https://www.amd.com/en/products/apu/amd-ryzen-7-7840hs https://www.amd.com/en/products/apu/amd-ryzen-7-7840hs The thing AMD doesn't have is the software stack.
- cyanydeez 3y ago
  The obvious answer is to just let the LLM boostrap use better abstractions.
m_mueller 3y ago
Inference-only hardware like this could be a temporary cost saving solution to scale up AI infrastructure, but I think a chip the size of this should be able to support training, otherwise it’s just a waste of money and energy. I predict inference will move to edge computing based on mobile chips like the ones from Qualcomm in the midterm.
- wrigby 3y ago
  This. When I was still doing infra for a large recommendation engine, the training was what got the big GPUs. Inference was on CPU or (later) ASICs (may have been FPGAs?).
- mupuff1234 3y ago
  Unless you have a model that's good enough for your use case and it's mostly inference from here on.
godelski 3y ago
> It's not rocket science to implement matrix multiplication in any GPU. You're right, it's harder. Saying this as someone who's done more work on the former than the latter. (I have, with a team, built a rocket engine. And not your school or backyard project size, but nozzle bigger than your face kind. I've also written CUDA kernels and boy is there a big learning curve to the latter that you gotta fundamentally rethink how you view a problem. It's unquestionable why CUDA devs are paid so much. Really it's only questionable why they aren't paid more) I know it is easy to think this problem is easy, it really looks that way. But there's an incredible amount of optimization that goes into all of this and that's what's really hard. You aren't going to get away with just N for loops for a tensor rank N. You got to chop the data up, be intelligent about it, manage memory, how you load memory, handle many data types, take into consideration different results for different FMA operations, and a whole lot more. There's a whole lot of non-obvious things that result in high optimization (maybe obvious __after__ the fact, but that's not truthfully "obvious"). The thing is, the space is so well researched and implemented that you can't get away with naive implementations, you have to be on the bleeding edge. Then you have to do that and make it reasonably usable for the programmer too, abstracting away all of that. Cuda also has a huge head start and momentum is not a force to be reckoned with (pun intended). Look at TensorRT[0]. The software isn't even complete and it still isn't going to cover all neural networks on all GPUs. I've had stuff work on a V100 and H100 but not an A100, then later get fixed. They even have the "Apple Advantage" in that they have control of the hardware. I'm not certain AMD will have the same advantage. We talk a lot about the difficulties of being first mover, but I think we can also recognize that momentum is an advantage of being first mover. And it isn't one to scoff at. [0] https://github.com/NVIDIA/TensorRT https://github.com/NVIDIA/TensorRT
- doctorpangloss 3y ago
  This comment is not going to age well. In the interest of curiosity, in this indictment of yours, is there any specific reason, about the AMD hardware itself, that leads you to believe that such implementations don't already exist, but are merely just not open sourced? I have a specific reason to believe this is all already optimized: poor adoption of Infinity Fabric by unsophisticated people, whereas NVLink is on every 3090 and DGX/HGX box. But that doesn't mean DC users haven't optimized.
visarga 3y ago
> I don't quite understand. Most of LLM inference is MatMul and Softmax. Most operations are memory bound: slow because memory access takes longer than computation itself. The problem is we have to shove the whole model, billions of weights, in the SRAM once for every token generated. That creates slowness.

whalesalad 3y ago

Everyone has been writing cuda code. Hopefully we continue the shift towards using higher level frameworks/abstractions that can determine the optimum machine code for the underlying hardware. I don’t really get it either. Nvidia is pouring $$$ into protecting their advantage as anemic as it might be. They give free compute and gpu resources to universities etc to cement the dependence on cuda. When you’re a beginner or someone who needs to just “get shit done” you’ll go nvidia at the expense of the greater good.

dannyw 3y ago
Also, the full consumer (RTX) stack is supported. Lots of people who are gamers, or want a hobbyist rig can go out to the shops and start with a RTX. AMD's ML / ROCm support for their consumer cards is still awful.
- plasticchris 3y ago
  They did add official support for the latest gen rdna cards to rocm.
  - ekianjo 3y ago
    Why only the latest gen?
    - svnt 3y ago
      Because they want to recover their investment
      - aryonoco 3y ago
        Actually because ROCm ships native machine code, as opposed to a bytecode, so supporting any additional architecture is a tonne of effort. I expect support will eventually be extended to a generation or two prior GPUs, but nothing beyond that.
  - dannyw 3y ago
    Not on Linux!
    - plasticchris 3y ago
      What? It was only for Linux. https://rocm.docs.amd.com/projects/radeon/en/latest/index.html https://rocm.docs.amd.com/projects/radeon/en/latest/index.ht...
pavelstoev 3y ago
Try Hidet - open source deep learning compiler written in Python, generating optimized CUDA https://pypi.org/project/hidet/ https://pypi.org/project/hidet/
jakderrida 3y ago
> They give free compute and gpu resources to universities etc to cement the dependence on cuda. It's actually quite brilliant. It never occurred to me before that the resources we had access to at university weren't the most popular because that's what the library wanted. They were the most popular because the provider made them cheap/free to widen their moat over the competition. It reminds me of Thomson Reuters launching Eikon, a superior alternative (imho) to Bloomberg terminals. Every business school has Bloomberg terminals. But few/none had Eikon, which is why nobody knows how to use one.
- kimixa 3y ago
  It's the same reason that students get free MSDN access and unlimited windows keys. It's a pretty solid strategy, but naturally favors the current incumbent who has the $$$ to throw around.
anvuong 3y ago
What the hell is the greater good here? I'm a researcher in a R1 university, I can quickly try stupid things on my personal 3080, then if it works I can virtually git clone the codes onto my lab's server and bam it chops through the problems. After a few weeks I'll get (hopefully) some publisable results that (hopefully) advance my field a little. Do you really expect most researchers to use half of their time debugging the unfinished ROCM stack that is just not there, sacrificing their output, and maybe even their tenure prospect? We love Nvidia solutions, at least in our field, not because it's a monopoly, but because it's actually the best solution right now.
- donor20 3y ago
  I really like that you can prototype on your own consumer level card - the rocm stack has not matched that historically and has over promised and underdelivered for years now.
  - cyanydeez 3y ago
    It's getting closer, and well, what matters is price points and what, billions of people crunching away on the various problems. There's huge momentum on the "user" side of the equation that just getting cheap hardware and passable performance means Nvidia losing dominance.
nemothekid 3y ago
>Nvidia is pouring $$$ into protecting their advantage as anemic as it might be. This description isn't fair. Nvidia was pouring money into CUDA for 16 years. AMD/Intel only woke up once ChatGPT launched and nvidia started printing money. It's not fair to call it "protecting their advantage" when nvidia invested into the ecosystem for years and AMD couldn't even bother to ship working examples with OpenCL. I'm not sure why choosing the company that decided to invest in the platform for years is at the expense of the "greater good." What "greater good" is there above actually working software?
- bryanlarsen 3y ago
  Nvidia had boatloads of money to spend, and 6X the number of engineers. AMD was skating on the edge of bankruptcy. AMD bet the farm on Zen, and that bet paid off so now they have money to spend on other things. If they would have split resources off of Zen to spend on CUDA they would have failed at both and AMD would be a bankrupt husk.
- cyanydeez 3y ago
  You can get a consumer laptop with AI: https://www.amd.com/en/products/apu/amd-ryzen-7-7840hs https://www.amd.com/en/products/apu/amd-ryzen-7-7840hs The thing AMD doesn't have is the software stack.
cyanydeez 3y ago
The obvious answer is to just let the LLM boostrap use better abstractions.

m_mueller 3y ago

Inference-only hardware like this could be a temporary cost saving solution to scale up AI infrastructure, but I think a chip the size of this should be able to support training, otherwise it’s just a waste of money and energy. I predict inference will move to edge computing based on mobile chips like the ones from Qualcomm in the midterm.

wrigby 3y ago
This. When I was still doing infra for a large recommendation engine, the training was what got the big GPUs. Inference was on CPU or (later) ASICs (may have been FPGAs?).
mupuff1234 3y ago
Unless you have a model that's good enough for your use case and it's mostly inference from here on.

godelski 3y ago

> It's not rocket science to implement matrix multiplication in any GPU. You're right, it's harder. Saying this as someone who's done more work on the former than the latter. (I have, with a team, built a rocket engine. And not your school or backyard project size, but nozzle bigger than your face kind. I've also written CUDA kernels and boy is there a big learning curve to the latter that you gotta fundamentally rethink how you view a problem. It's unquestionable why CUDA devs are paid so much. Really it's only questionable why they aren't paid more) I know it is easy to think this problem is easy, it really looks that way. But there's an incredible amount of optimization that goes into all of this and that's what's really hard. You aren't going to get away with just N for loops for a tensor rank N. You got to chop the data up, be intelligent about it, manage memory, how you load memory, handle many data types, take into consideration different results for different FMA operations, and a whole lot more. There's a whole lot of non-obvious things that result in high optimization (maybe obvious __after__ the fact, but that's not truthfully "obvious"). The thing is, the space is so well researched and implemented that you can't get away with naive implementations, you have to be on the bleeding edge. Then you have to do that and make it reasonably usable for the programmer too, abstracting away all of that. Cuda also has a huge head start and momentum is not a force to be reckoned with (pun intended). Look at TensorRT[0]. The software isn't even complete and it still isn't going to cover all neural networks on all GPUs. I've had stuff work on a V100 and H100 but not an A100, then later get fixed. They even have the "Apple Advantage" in that they have control of the hardware. I'm not certain AMD will have the same advantage. We talk a lot about the difficulties of being first mover, but I think we can also recognize that momentum is an advantage of being first mover. And it isn't one to scoff at. [0] https://github.com/NVIDIA/TensorRT https://github.com/NVIDIA/TensorRT

doctorpangloss 3y ago
This comment is not going to age well. In the interest of curiosity, in this indictment of yours, is there any specific reason, about the AMD hardware itself, that leads you to believe that such implementations don't already exist, but are merely just not open sourced? I have a specific reason to believe this is all already optimized: poor adoption of Infinity Fabric by unsophisticated people, whereas NVLink is on every 3090 and DGX/HGX box. But that doesn't mean DC users haven't optimized.

visarga 3y ago

> I don't quite understand. Most of LLM inference is MatMul and Softmax. Most operations are memory bound: slow because memory access takes longer than computation itself. The problem is we have to shove the whole model, billions of weights, in the SRAM once for every token generated. That creates slowness.