5 ms·

I though usage of PyTorch >> cuda directly, which case cuda is less crucial.

by gumby 3y ago

godelski 3y ago
PyTorch is using cuda on the backend. You might not see it, but that's what's going on. So there's no "PyTorch vs CUDA" because "PyTorch __is__ CUDA". You can also say it's a lot of other things like ROCm. It's best to think of the python in PyTorch as more of an API.
- anvuong 3y ago
  It's not that simple. Pytorch has its own computing model, calling it just APIs is really downplaying the phenomenal works behind the framework. Pytorch put Tensorflow to shame even though the latter had huge first mover advantage.
  - godelski 3y ago
    For sure it's not that simple, but the person I was responding to was at the stage of understanding that they were thinking PyTorch was better than CUDA. I'm trying to communicate to them, not someone with a more detailed understanding (I mean one of the top comments here thinks ML is "just matrix multiplies"...) Tensorflow had first mover advantage but they also had first mover disadvantage. TF isn't that easy to write nor to debug. PyTorch got to see everything that was wrong with TF and fix it. No doubt Meta has put in lots of amazing effort. It is an incredible piece of software. I'll also add that PyTorch does a lot of great stuff that people don't use or recognize. There are a lot of good distributional works out there for prob and stats people that give you cuda acceleration. Really a lot of numpy can be replaced with pytorch and its great. But be warned of FMA and other optimization differences.
- hexaga 3y ago
  Their point seems sound. Restated: Using pytorch [is better than] using CUDA directly (i.e., ml devs interact w/ pytorch apis not CUDA apis). And if you're using a wrapper anyway, whether CUDA is used or some other backend is less crucial (than if directly writing to a GPGPU api) so long as the wrapper works as expected.
- woadwarrior01 3y ago
  > So there's no "PyTorch vs CUDA" because "PyTorch __is__ CUDA". Not true. PyTorch has had an MPS backend[1] for Apple Silicon for quite some time now. And lately, they've also added a HIP backend[2] for AMD hardware. I've written CUDA kernels for PyTorch in the past, it'd have been a lot easier if PyTorch were a thin wrapper over CUDA, but it isn't. [1]: https://pytorch.org/docs/stable/notes/mps.html https://pytorch.org/docs/stable/notes/mps.html [2]: https://pytorch.org/docs/stable/notes/hip.html https://pytorch.org/docs/stable/notes/hip.html
  - godelski 3y ago
    Sure, I hope you can recognize that I was trying to convey that there were a lot of things. I mention ROCm because it is more than HIP and is what you'll see when you install pytorch. I hope you recognize that I was intending to convey that it was more than a wrapper. I'll admit that I could have phrased my intent much better, but I was trying to write to an audience of one with my best interpretation of their comment.
    - weebull 3y ago
      You don't mention whether PyTorch runs with ROCm though, and it does. RX 6xxx and 7xxx series GPUs do pretty well with it.
      - godelski 3y ago
        > So there's no "PyTorch vs CUDA" because "PyTorch __is__ CUDA". ==> You can also say it's a lot of other things like ROCm. <== I mean my intent was just to communicate that PyTorch runs CUDA in the backend, which was what I meant by API, because it seemed the person I was responding to wasn't aware of this backend code you're interfacing with. I thought adding the same phrasing about it also "being" ROCm and using "such as" and "other things," that others would have allowed anyone with more knowledge to infer that I'm aware of the generalization but I guess I was deeply mistaken given the replies that are certain that I can't read the pytorch homepage. Though I'm not quite sure how you came to the conclusion that I wasn't aware of ROCm support since I did explicitly mention it. Do you have feedback for how I can better communicate? I seem to be running into failures like this a lot lately where I feel like I can point to where I clearly stated something but that doesn't matter because communication is about getting the person on the other side to receive the message. I've been scratching my head about this and could definitely use the outside perspective.
- gumby 3y ago
  I wrote that the usage of PyTorch is much higher than bare cuda, not making a claim that one is better than the other. Obviously you can use cuda to write more than ML algos, but that’s its overwhelming use case. Thus my point is if AMD ported PyTorch to their hardware (and PT is not cuda-only) it would create a more level playing field on which they could compete (in a cousin vs cousin battle) by eliminating the switching cost for the majority of developers. This is the same reason cpu mfrs have compiler ports but generally don’t have to worry about instruction set level compatibility.
  - godelski 3y ago
    But you also suggested that cuda is less critical. But there's so much that is dependent on cuda, even if it is directly visible. You're right, that if AMD does fully integrate into Pytorch, something they've been working on btw, then yes this will be a big step up for AMD as ML is one of the biggest sectors. But it's important to realize that this is not as easy of a task as it sounds. They have been working on this for years and every step of the way Nvidia is still improving too.
  - weebull 3y ago
    > Thus my point is if AMD ported PyTorch to their hardware (and PT is not cuda-only) it would create a more level playing field on which they could compete It is ported (I think by AMD). I use it daily. Biggest problem is that the people maintaining PyTorch aren't very interested as they don't use it themselves. Hence there are silly bugs in it. Users contribute fixes and they get closed down as "unsupported" or get plain misunderstood.

I though usage of PyTorch >> cuda directly, which case cuda is less crucial.

by gumby 3y ago

I though usage of PyTorch >> cuda directly, which case cuda is less crucial.

PyTorch is using cuda on the backend. You might not see it, but that's what's going on. So there's no "PyTorch vs CUDA" because "PyTorch __is__ CUDA". You can also say it's a lot of other things like ROCm. It's best to think of the python in PyTorch as more of an API.

anvuong 3y ago
It's not that simple. Pytorch has its own computing model, calling it just APIs is really downplaying the phenomenal works behind the framework. Pytorch put Tensorflow to shame even though the latter had huge first mover advantage.
- godelski 3y ago
  For sure it's not that simple, but the person I was responding to was at the stage of understanding that they were thinking PyTorch was better than CUDA. I'm trying to communicate to them, not someone with a more detailed understanding (I mean one of the top comments here thinks ML is "just matrix multiplies"...) Tensorflow had first mover advantage but they also had first mover disadvantage. TF isn't that easy to write nor to debug. PyTorch got to see everything that was wrong with TF and fix it. No doubt Meta has put in lots of amazing effort. It is an incredible piece of software. I'll also add that PyTorch does a lot of great stuff that people don't use or recognize. There are a lot of good distributional works out there for prob and stats people that give you cuda acceleration. Really a lot of numpy can be replaced with pytorch and its great. But be warned of FMA and other optimization differences.
hexaga 3y ago
Their point seems sound. Restated: Using pytorch [is better than] using CUDA directly (i.e., ml devs interact w/ pytorch apis not CUDA apis). And if you're using a wrapper anyway, whether CUDA is used or some other backend is less crucial (than if directly writing to a GPGPU api) so long as the wrapper works as expected.
woadwarrior01 3y ago
> So there's no "PyTorch vs CUDA" because "PyTorch __is__ CUDA". Not true. PyTorch has had an MPS backend[1] for Apple Silicon for quite some time now. And lately, they've also added a HIP backend[2] for AMD hardware. I've written CUDA kernels for PyTorch in the past, it'd have been a lot easier if PyTorch were a thin wrapper over CUDA, but it isn't. [1]: https://pytorch.org/docs/stable/notes/mps.html https://pytorch.org/docs/stable/notes/mps.html [2]: https://pytorch.org/docs/stable/notes/hip.html https://pytorch.org/docs/stable/notes/hip.html
- godelski 3y ago
  Sure, I hope you can recognize that I was trying to convey that there were a lot of things. I mention ROCm because it is more than HIP and is what you'll see when you install pytorch. I hope you recognize that I was intending to convey that it was more than a wrapper. I'll admit that I could have phrased my intent much better, but I was trying to write to an audience of one with my best interpretation of their comment.
  - weebull 3y ago
    You don't mention whether PyTorch runs with ROCm though, and it does. RX 6xxx and 7xxx series GPUs do pretty well with it.
    - godelski 3y ago
      > So there's no "PyTorch vs CUDA" because "PyTorch __is__ CUDA". ==> You can also say it's a lot of other things like ROCm. <== I mean my intent was just to communicate that PyTorch runs CUDA in the backend, which was what I meant by API, because it seemed the person I was responding to wasn't aware of this backend code you're interfacing with. I thought adding the same phrasing about it also "being" ROCm and using "such as" and "other things," that others would have allowed anyone with more knowledge to infer that I'm aware of the generalization but I guess I was deeply mistaken given the replies that are certain that I can't read the pytorch homepage. Though I'm not quite sure how you came to the conclusion that I wasn't aware of ROCm support since I did explicitly mention it. Do you have feedback for how I can better communicate? I seem to be running into failures like this a lot lately where I feel like I can point to where I clearly stated something but that doesn't matter because communication is about getting the person on the other side to receive the message. I've been scratching my head about this and could definitely use the outside perspective.
gumby 3y ago
I wrote that the usage of PyTorch is much higher than bare cuda, not making a claim that one is better than the other. Obviously you can use cuda to write more than ML algos, but that’s its overwhelming use case. Thus my point is if AMD ported PyTorch to their hardware (and PT is not cuda-only) it would create a more level playing field on which they could compete (in a cousin vs cousin battle) by eliminating the switching cost for the majority of developers. This is the same reason cpu mfrs have compiler ports but generally don’t have to worry about instruction set level compatibility.
- godelski 3y ago
  But you also suggested that cuda is less critical. But there's so much that is dependent on cuda, even if it is directly visible. You're right, that if AMD does fully integrate into Pytorch, something they've been working on btw, then yes this will be a big step up for AMD as ML is one of the biggest sectors. But it's important to realize that this is not as easy of a task as it sounds. They have been working on this for years and every step of the way Nvidia is still improving too.
- weebull 3y ago
  > Thus my point is if AMD ported PyTorch to their hardware (and PT is not cuda-only) it would create a more level playing field on which they could compete It is ported (I think by AMD). I use it daily. Biggest problem is that the people maintaining PyTorch aren't very interested as they don't use it themselves. Hence there are silly bugs in it. Users contribute fixes and they get closed down as "unsupported" or get plain misunderstood.