5 ms·

nvidia is winning with cuda, not hardware. as soon as AMD can crack the rocm/cross compile nut they will be on top. dominating server market already. and in pur

nvidia is winning with cuda, not hardware. as soon as AMD can crack the rocm/cross compile nut they will be on top. dominating server market already. and in pure raster perf, they’re dominating the graphics space as well.

gumby 3y ago
I though usage of PyTorch >> cuda directly, which case cuda is less crucial.
- godelski 3y ago
  PyTorch is using cuda on the backend. You might not see it, but that's what's going on. So there's no "PyTorch vs CUDA" because "PyTorch __is__ CUDA". You can also say it's a lot of other things like ROCm. It's best to think of the python in PyTorch as more of an API.
  - anvuong 3y ago
    It's not that simple. Pytorch has its own computing model, calling it just APIs is really downplaying the phenomenal works behind the framework. Pytorch put Tensorflow to shame even though the latter had huge first mover advantage.
    - godelski 3y ago
      For sure it's not that simple, but the person I was responding to was at the stage of understanding that they were thinking PyTorch was better than CUDA. I'm trying to communicate to them, not someone with a more detailed understanding (I mean one of the top comments here thinks ML is "just matrix multiplies"...) Tensorflow had first mover advantage but they also had first mover disadvantage. TF isn't that easy to write nor to debug. PyTorch got to see everything that was wrong with TF and fix it. No doubt Meta has put in lots of amazing effort. It is an incredible piece of software. I'll also add that PyTorch does a lot of great stuff that people don't use or recognize. There are a lot of good distributional works out there for prob and stats people that give you cuda acceleration. Really a lot of numpy can be replaced with pytorch and its great. But be warned of FMA and other optimization differences.
  - hexaga 3y ago
    Their point seems sound. Restated: Using pytorch [is better than] using CUDA directly (i.e., ml devs interact w/ pytorch apis not CUDA apis). And if you're using a wrapper anyway, whether CUDA is used or some other backend is less crucial (than if directly writing to a GPGPU api) so long as the wrapper works as expected.
  - woadwarrior01 3y ago
    > So there's no "PyTorch vs CUDA" because "PyTorch __is__ CUDA". Not true. PyTorch has had an MPS backend[1] for Apple Silicon for quite some time now. And lately, they've also added a HIP backend[2] for AMD hardware. I've written CUDA kernels for PyTorch in the past, it'd have been a lot easier if PyTorch were a thin wrapper over CUDA, but it isn't. [1]: https://pytorch.org/docs/stable/notes/mps.html https://pytorch.org/docs/stable/notes/mps.html [2]: https://pytorch.org/docs/stable/notes/hip.html https://pytorch.org/docs/stable/notes/hip.html
    - godelski 3y ago
      Sure, I hope you can recognize that I was trying to convey that there were a lot of things. I mention ROCm because it is more than HIP and is what you'll see when you install pytorch. I hope you recognize that I was intending to convey that it was more than a wrapper. I'll admit that I could have phrased my intent much better, but I was trying to write to an audience of one with my best interpretation of their comment.
      - weebull 3y ago
        You don't mention whether PyTorch runs with ROCm though, and it does. RX 6xxx and 7xxx series GPUs do pretty well with it.
        godelski 3y ago
        > So there's no "PyTorch vs CUDA" because "PyTorch __is__ CUDA". ==> You can also say it's a lot of other things like ROCm. <== I mean my intent was just to communicate that PyTorch runs CUDA in the backend, which was what I meant by API, because it seemed the person I was responding to wasn't aware of this backend code you're interfacing with. I thought adding the same phrasing about it also "being" ROCm and using "such as" and "other things," that others would have allowed anyone with more knowledge to infer that I'm aware of the generalization but I guess I was deeply mistaken given the replies that are certain that I can't read the pytorch homepage. Though I'm not quite sure how you came to the conclusion that I wasn't aware of ROCm support since I did explicitly mention it. Do you have feedback for how I can better communicate? I seem to be running into failures like this a lot lately where I feel like I can point to where I clearly stated something but that doesn't matter because communication is about getting the person on the other side to receive the message. I've been scratching my head about this and could definitely use the outside perspective.
  - gumby 3y ago
    I wrote that the usage of PyTorch is much higher than bare cuda, not making a claim that one is better than the other. Obviously you can use cuda to write more than ML algos, but that’s its overwhelming use case. Thus my point is if AMD ported PyTorch to their hardware (and PT is not cuda-only) it would create a more level playing field on which they could compete (in a cousin vs cousin battle) by eliminating the switching cost for the majority of developers. This is the same reason cpu mfrs have compiler ports but generally don’t have to worry about instruction set level compatibility.
    - godelski 3y ago
      But you also suggested that cuda is less critical. But there's so much that is dependent on cuda, even if it is directly visible. You're right, that if AMD does fully integrate into Pytorch, something they've been working on btw, then yes this will be a big step up for AMD as ML is one of the biggest sectors. But it's important to realize that this is not as easy of a task as it sounds. They have been working on this for years and every step of the way Nvidia is still improving too.
    - weebull 3y ago
      > Thus my point is if AMD ported PyTorch to their hardware (and PT is not cuda-only) it would create a more level playing field on which they could compete It is ported (I think by AMD). I use it daily. Biggest problem is that the people maintaining PyTorch aren't very interested as they don't use it themselves. Hence there are silly bugs in it. Users contribute fixes and they get closed down as "unsupported" or get plain misunderstood.
girvo 3y ago
> and in pure raster perf, they’re dominating the graphics space as well. Wait, is that actually true? The 4090 is superior in raster to the 7900XTX I thought? Happy to be corrected!
- klausa 3y ago
  My understanding is that they're roughly on-par in raster performance, but Nvidia runs circles around AMD when it comes to RT and DLSS stuff.
  - whalesalad 3y ago
    yeah they do, but simply because they were first to market and have game studios using their stuff. ultimately dlss/frame gen is a hack that no one on a high end card would ever use. ray tracing nvidia is king but they got the jump. that will normalize over the next generation or two.
    - nodja 3y ago
      > ultimately dlss/frame gen is a hack that no one on a high end card would ever use. The irony of those technologies is that they shine best on high end cards. If you're on a low end card and say pushing the limit of a game and are upscaling 720p to 1080p, then dlss can't do miracles, there will be tons of defects due how little information about the scene the upscaler has. But if you're on a high end system trying to upscale 1440p to 4k then even not only is the upscaler less likely to make a mistake, it's also much less noticeable. Same with framegen, on a low end system that's getting sub 60fps, the framegen not only adds noticeable input lag, but often makes noticeable defective frames in between, if your system was already pushing above 60 then not only is the input lag much more bearable, the possible defective frames stay on screen for so little that you'd have to be paying attention to it to notice them most of the time. They're still useful for low end systems, they can prolong their lifespan quite a bit for current gen games, but as this tech becomes the norm, developers rely on them to hit performance targets and your card will be outdated just as fast as older cards used to.
    - 3y ago
Eridrus 3y ago
I've seen multiple people mention training LLMs successfuly on AMD hardware, I think the software stack is close enough on these data center chips that you can productively use them and the bigger question, is if you can get your hands on them more easily than H100s.
- _3u10 3y ago
  It’s difficult to productively game on AMD cards I don’t even want to know how annoying it is to start training over because of a driver crash.
smy20011 3y ago
I don't quite understand. Most of LLM inference is MatMul and Softmax. It's not rocket science to implement matrix multiplication in any GPU. We don't have very complex custom kernel for the LLM inference. If AMD can provide fast GPU with cheaper price, a lot of people don't mind to add support for them. GGML shows you can create a working LLM inference solution with one man's work.
- whalesalad 3y ago
  Everyone has been writing cuda code. Hopefully we continue the shift towards using higher level frameworks/abstractions that can determine the optimum machine code for the underlying hardware. I don’t really get it either. Nvidia is pouring $$$ into protecting their advantage as anemic as it might be. They give free compute and gpu resources to universities etc to cement the dependence on cuda. When you’re a beginner or someone who needs to just “get shit done” you’ll go nvidia at the expense of the greater good.
  - dannyw 3y ago
    Also, the full consumer (RTX) stack is supported. Lots of people who are gamers, or want a hobbyist rig can go out to the shops and start with a RTX. AMD's ML / ROCm support for their consumer cards is still awful.
    - plasticchris 3y ago
      They did add official support for the latest gen rdna cards to rocm.
      - ekianjo 3y ago
        Why only the latest gen?
        svnt 3y ago
        Because they want to recover their investment
stefan_ 3y ago
You are telling me what is supposedly a multi trillion dollar business being pursued by the brightest minds at Google and others would rather join the queue for a hardware drop from NVIDIA than enjoy a substantial competitive advantage being able to run on AMD hardware? For something that supposedly breaks down into such simple steps everyone is also building hardware to do it? Something doesn’t quite add up for this dependence on CUDA. Sure, I’m installing some dependency hellscape Python research project from GitHub I reasonably expect idiosyncratic hardware dependencies, but surely the companies in this space are beyond that on the maturity timeline.
- weebull 3y ago
  > Something doesn’t quite add up for this dependence on CUDA. Sure, I’m installing some dependency hellscape Python research project from GitHub I reasonably expect idiosyncratic hardware dependencies, but surely the companies in this space are beyond that on the maturity timeline. I think part of the problem is something you're touching on here. The ML software space is horrifically put together. It's a bunch of research projects cobbled together. Hence the dependency hell, and very few people who actually know how to port it. Cuda siits at the bottom of the stack with its design assumptions baked into everything above it (see PyTorch for a good example of that). Nobody wrote a hardware agnostic layer over the top, and now the thought of pulling out a Jenga block at the bottom scares everyone. Cuda is an intentionally leaky abstraction layer over the hardware, and its worked.
varelse 3y ago
[dead]

gumby 3y ago

I though usage of PyTorch >> cuda directly, which case cuda is less crucial.

godelski 3y ago
PyTorch is using cuda on the backend. You might not see it, but that's what's going on. So there's no "PyTorch vs CUDA" because "PyTorch __is__ CUDA". You can also say it's a lot of other things like ROCm. It's best to think of the python in PyTorch as more of an API.
- anvuong 3y ago
  It's not that simple. Pytorch has its own computing model, calling it just APIs is really downplaying the phenomenal works behind the framework. Pytorch put Tensorflow to shame even though the latter had huge first mover advantage.
  - godelski 3y ago
    For sure it's not that simple, but the person I was responding to was at the stage of understanding that they were thinking PyTorch was better than CUDA. I'm trying to communicate to them, not someone with a more detailed understanding (I mean one of the top comments here thinks ML is "just matrix multiplies"...) Tensorflow had first mover advantage but they also had first mover disadvantage. TF isn't that easy to write nor to debug. PyTorch got to see everything that was wrong with TF and fix it. No doubt Meta has put in lots of amazing effort. It is an incredible piece of software. I'll also add that PyTorch does a lot of great stuff that people don't use or recognize. There are a lot of good distributional works out there for prob and stats people that give you cuda acceleration. Really a lot of numpy can be replaced with pytorch and its great. But be warned of FMA and other optimization differences.
- hexaga 3y ago
  Their point seems sound. Restated: Using pytorch [is better than] using CUDA directly (i.e., ml devs interact w/ pytorch apis not CUDA apis). And if you're using a wrapper anyway, whether CUDA is used or some other backend is less crucial (than if directly writing to a GPGPU api) so long as the wrapper works as expected.
- woadwarrior01 3y ago
  > So there's no "PyTorch vs CUDA" because "PyTorch __is__ CUDA". Not true. PyTorch has had an MPS backend[1] for Apple Silicon for quite some time now. And lately, they've also added a HIP backend[2] for AMD hardware. I've written CUDA kernels for PyTorch in the past, it'd have been a lot easier if PyTorch were a thin wrapper over CUDA, but it isn't. [1]: https://pytorch.org/docs/stable/notes/mps.html https://pytorch.org/docs/stable/notes/mps.html [2]: https://pytorch.org/docs/stable/notes/hip.html https://pytorch.org/docs/stable/notes/hip.html
  - godelski 3y ago
    Sure, I hope you can recognize that I was trying to convey that there were a lot of things. I mention ROCm because it is more than HIP and is what you'll see when you install pytorch. I hope you recognize that I was intending to convey that it was more than a wrapper. I'll admit that I could have phrased my intent much better, but I was trying to write to an audience of one with my best interpretation of their comment.
    - weebull 3y ago
      You don't mention whether PyTorch runs with ROCm though, and it does. RX 6xxx and 7xxx series GPUs do pretty well with it.
      - godelski 3y ago
        > So there's no "PyTorch vs CUDA" because "PyTorch __is__ CUDA". ==> You can also say it's a lot of other things like ROCm. <== I mean my intent was just to communicate that PyTorch runs CUDA in the backend, which was what I meant by API, because it seemed the person I was responding to wasn't aware of this backend code you're interfacing with. I thought adding the same phrasing about it also "being" ROCm and using "such as" and "other things," that others would have allowed anyone with more knowledge to infer that I'm aware of the generalization but I guess I was deeply mistaken given the replies that are certain that I can't read the pytorch homepage. Though I'm not quite sure how you came to the conclusion that I wasn't aware of ROCm support since I did explicitly mention it. Do you have feedback for how I can better communicate? I seem to be running into failures like this a lot lately where I feel like I can point to where I clearly stated something but that doesn't matter because communication is about getting the person on the other side to receive the message. I've been scratching my head about this and could definitely use the outside perspective.
- gumby 3y ago
  I wrote that the usage of PyTorch is much higher than bare cuda, not making a claim that one is better than the other. Obviously you can use cuda to write more than ML algos, but that’s its overwhelming use case. Thus my point is if AMD ported PyTorch to their hardware (and PT is not cuda-only) it would create a more level playing field on which they could compete (in a cousin vs cousin battle) by eliminating the switching cost for the majority of developers. This is the same reason cpu mfrs have compiler ports but generally don’t have to worry about instruction set level compatibility.
  - godelski 3y ago
    But you also suggested that cuda is less critical. But there's so much that is dependent on cuda, even if it is directly visible. You're right, that if AMD does fully integrate into Pytorch, something they've been working on btw, then yes this will be a big step up for AMD as ML is one of the biggest sectors. But it's important to realize that this is not as easy of a task as it sounds. They have been working on this for years and every step of the way Nvidia is still improving too.
  - weebull 3y ago
    > Thus my point is if AMD ported PyTorch to their hardware (and PT is not cuda-only) it would create a more level playing field on which they could compete It is ported (I think by AMD). I use it daily. Biggest problem is that the people maintaining PyTorch aren't very interested as they don't use it themselves. Hence there are silly bugs in it. Users contribute fixes and they get closed down as "unsupported" or get plain misunderstood.

girvo 3y ago

> and in pure raster perf, they’re dominating the graphics space as well. Wait, is that actually true? The 4090 is superior in raster to the 7900XTX I thought? Happy to be corrected!

klausa 3y ago
My understanding is that they're roughly on-par in raster performance, but Nvidia runs circles around AMD when it comes to RT and DLSS stuff.
- whalesalad 3y ago
  yeah they do, but simply because they were first to market and have game studios using their stuff. ultimately dlss/frame gen is a hack that no one on a high end card would ever use. ray tracing nvidia is king but they got the jump. that will normalize over the next generation or two.
  - nodja 3y ago
    > ultimately dlss/frame gen is a hack that no one on a high end card would ever use. The irony of those technologies is that they shine best on high end cards. If you're on a low end card and say pushing the limit of a game and are upscaling 720p to 1080p, then dlss can't do miracles, there will be tons of defects due how little information about the scene the upscaler has. But if you're on a high end system trying to upscale 1440p to 4k then even not only is the upscaler less likely to make a mistake, it's also much less noticeable. Same with framegen, on a low end system that's getting sub 60fps, the framegen not only adds noticeable input lag, but often makes noticeable defective frames in between, if your system was already pushing above 60 then not only is the input lag much more bearable, the possible defective frames stay on screen for so little that you'd have to be paying attention to it to notice them most of the time. They're still useful for low end systems, they can prolong their lifespan quite a bit for current gen games, but as this tech becomes the norm, developers rely on them to hit performance targets and your card will be outdated just as fast as older cards used to.
  - 3y ago

Eridrus 3y ago

I've seen multiple people mention training LLMs successfuly on AMD hardware, I think the software stack is close enough on these data center chips that you can productively use them and the bigger question, is if you can get your hands on them more easily than H100s.

_3u10 3y ago
It’s difficult to productively game on AMD cards I don’t even want to know how annoying it is to start training over because of a driver crash.

smy20011 3y ago

I don't quite understand. Most of LLM inference is MatMul and Softmax. It's not rocket science to implement matrix multiplication in any GPU. We don't have very complex custom kernel for the LLM inference. If AMD can provide fast GPU with cheaper price, a lot of people don't mind to add support for them. GGML shows you can create a working LLM inference solution with one man's work.

whalesalad 3y ago
Everyone has been writing cuda code. Hopefully we continue the shift towards using higher level frameworks/abstractions that can determine the optimum machine code for the underlying hardware. I don’t really get it either. Nvidia is pouring $$$ into protecting their advantage as anemic as it might be. They give free compute and gpu resources to universities etc to cement the dependence on cuda. When you’re a beginner or someone who needs to just “get shit done” you’ll go nvidia at the expense of the greater good.
- dannyw 3y ago
  Also, the full consumer (RTX) stack is supported. Lots of people who are gamers, or want a hobbyist rig can go out to the shops and start with a RTX. AMD's ML / ROCm support for their consumer cards is still awful.
  - plasticchris 3y ago
    They did add official support for the latest gen rdna cards to rocm.
    - ekianjo 3y ago
      Why only the latest gen?
      - svnt 3y ago
        Because they want to recover their investment

stefan_ 3y ago

You are telling me what is supposedly a multi trillion dollar business being pursued by the brightest minds at Google and others would rather join the queue for a hardware drop from NVIDIA than enjoy a substantial competitive advantage being able to run on AMD hardware? For something that supposedly breaks down into such simple steps everyone is also building hardware to do it? Something doesn’t quite add up for this dependence on CUDA. Sure, I’m installing some dependency hellscape Python research project from GitHub I reasonably expect idiosyncratic hardware dependencies, but surely the companies in this space are beyond that on the maturity timeline.

weebull 3y ago
> Something doesn’t quite add up for this dependence on CUDA. Sure, I’m installing some dependency hellscape Python research project from GitHub I reasonably expect idiosyncratic hardware dependencies, but surely the companies in this space are beyond that on the maturity timeline. I think part of the problem is something you're touching on here. The ML software space is horrifically put together. It's a bunch of research projects cobbled together. Hence the dependency hell, and very few people who actually know how to port it. Cuda siits at the bottom of the stack with its design assumptions baked into everything above it (see PyTorch for a good example of that). Nobody wrote a hardware agnostic layer over the top, and now the thought of pulling out a Jenga block at the bottom scares everyone. Cuda is an intentionally leaky abstraction layer over the hardware, and its worked.

varelse 3y ago

[dead]