4 ms·

Almost the entirety of the post is not related to the title. The summary is of this is given in last: > OpenAI Triton only officially supports Nvidia GPUs toda

Almost the entirety of the post is not related to the title. The summary is of this is given in last: > OpenAI Triton only officially supports Nvidia GPUs today, but that is changing in the near future. Multiple other hardware vendors will be supported in the future, and this open-source project is gaining incredible steam. It's not even the first tool for allowing ML workloads in non NVidia cards. ROCm already exists. TF already supports M1/M2 GPU already. The entire point is that no cards come close to Nvidia on raw GPGPU performance, and this is not going to change in near future.

atemerev 4y ago
Have you actually tried to write something with ROCm compared to CUDA? For me, ROCm is very low-level, boilerplate-ridden and nearly unusable.
- JonChesterfield 4y ago
  HIP is very similar to cuda. There's a perl string substitution thing that will try to convert one to the other. There is lower level code below cuda and below hip. In cuda, it's closed source and hidden. In rocm, you can follow the control flow all the way into the amdkfd kernel module. I like having the option to program at a lower level than cuda. When I need to write in hip, I like having all the source from the stack trace.
  - pjmlp 4y ago
    Except the small detail that CUDA is polyglot and quite a few runtimes target PTX. Those Perl scripts won't do nothing for them.
    - JonChesterfield 4y ago
      In which sense is cuda polyglot? It's a dialect of C++. Maybe you mean cuda the toolchain (driver libs, ptxas etc) as opposed to cuda the language. Shame they have the same name. Anything targeting PTX via LLVM should be able to target GCN by changing flags, though that may not be totally seamless. Things writing ptx assembly directly would be more awkward to retarget, there's no direct equivalent there.
      - pjmlp 4y ago
        In the sense that it officially supports C, C++ and Fortran on NVidia/PGI toolchain, plus anything else that targets PTX and there are plenty of those, like .NET, Java, Haskell, Julia. It is only a dialect of C++ for those that don't bother to read about what CUDA actually is. https://developer.nvidia.com/language-solutions https://developer.nvidia.com/language-solutions https://developer.nvidia.com/hpc-sdk https://developer.nvidia.com/hpc-sdk
- YetAnotherNick 4y ago
  I don't use CUDA or ROCm. I use tensorflow/pytorch which supports both of them. But the performance difference between them is huge, as AMD cards have much lower GPGPU/Tensor flops compared to top NVidia cards.
  - lostmsu 4y ago
    Any good benchmarks on large transformers?

atemerev 4y ago

Have you actually tried to write something with ROCm compared to CUDA? For me, ROCm is very low-level, boilerplate-ridden and nearly unusable.

JonChesterfield 4y ago
HIP is very similar to cuda. There's a perl string substitution thing that will try to convert one to the other. There is lower level code below cuda and below hip. In cuda, it's closed source and hidden. In rocm, you can follow the control flow all the way into the amdkfd kernel module. I like having the option to program at a lower level than cuda. When I need to write in hip, I like having all the source from the stack trace.
- pjmlp 4y ago
  Except the small detail that CUDA is polyglot and quite a few runtimes target PTX. Those Perl scripts won't do nothing for them.
  - JonChesterfield 4y ago
    In which sense is cuda polyglot? It's a dialect of C++. Maybe you mean cuda the toolchain (driver libs, ptxas etc) as opposed to cuda the language. Shame they have the same name. Anything targeting PTX via LLVM should be able to target GCN by changing flags, though that may not be totally seamless. Things writing ptx assembly directly would be more awkward to retarget, there's no direct equivalent there.
    - pjmlp 4y ago
      In the sense that it officially supports C, C++ and Fortran on NVidia/PGI toolchain, plus anything else that targets PTX and there are plenty of those, like .NET, Java, Haskell, Julia. It is only a dialect of C++ for those that don't bother to read about what CUDA actually is. https://developer.nvidia.com/language-solutions https://developer.nvidia.com/language-solutions https://developer.nvidia.com/hpc-sdk https://developer.nvidia.com/hpc-sdk
YetAnotherNick 4y ago
I don't use CUDA or ROCm. I use tensorflow/pytorch which supports both of them. But the performance difference between them is huge, as AMD cards have much lower GPGPU/Tensor flops compared to top NVidia cards.
- lostmsu 4y ago
  Any good benchmarks on large transformers?