3 ms·

Have you actually tried to write something with ROCm compared to CUDA? For me, ROCm is very low-level, boilerplate-ridden and nearly unusable.

JonChesterfield 4y ago
HIP is very similar to cuda. There's a perl string substitution thing that will try to convert one to the other. There is lower level code below cuda and below hip. In cuda, it's closed source and hidden. In rocm, you can follow the control flow all the way into the amdkfd kernel module. I like having the option to program at a lower level than cuda. When I need to write in hip, I like having all the source from the stack trace.
- pjmlp 4y ago
  Except the small detail that CUDA is polyglot and quite a few runtimes target PTX. Those Perl scripts won't do nothing for them.
  - JonChesterfield 4y ago
    In which sense is cuda polyglot? It's a dialect of C++. Maybe you mean cuda the toolchain (driver libs, ptxas etc) as opposed to cuda the language. Shame they have the same name. Anything targeting PTX via LLVM should be able to target GCN by changing flags, though that may not be totally seamless. Things writing ptx assembly directly would be more awkward to retarget, there's no direct equivalent there.
    - pjmlp 4y ago
      In the sense that it officially supports C, C++ and Fortran on NVidia/PGI toolchain, plus anything else that targets PTX and there are plenty of those, like .NET, Java, Haskell, Julia. It is only a dialect of C++ for those that don't bother to read about what CUDA actually is. https://developer.nvidia.com/language-solutions https://developer.nvidia.com/language-solutions https://developer.nvidia.com/hpc-sdk https://developer.nvidia.com/hpc-sdk
YetAnotherNick 4y ago
I don't use CUDA or ROCm. I use tensorflow/pytorch which supports both of them. But the performance difference between them is huge, as AMD cards have much lower GPGPU/Tensor flops compared to top NVidia cards.
- lostmsu 4y ago
  Any good benchmarks on large transformers?

JonChesterfield 4y ago

HIP is very similar to cuda. There's a perl string substitution thing that will try to convert one to the other. There is lower level code below cuda and below hip. In cuda, it's closed source and hidden. In rocm, you can follow the control flow all the way into the amdkfd kernel module. I like having the option to program at a lower level than cuda. When I need to write in hip, I like having all the source from the stack trace.

pjmlp 4y ago
Except the small detail that CUDA is polyglot and quite a few runtimes target PTX. Those Perl scripts won't do nothing for them.
- JonChesterfield 4y ago
  In which sense is cuda polyglot? It's a dialect of C++. Maybe you mean cuda the toolchain (driver libs, ptxas etc) as opposed to cuda the language. Shame they have the same name. Anything targeting PTX via LLVM should be able to target GCN by changing flags, though that may not be totally seamless. Things writing ptx assembly directly would be more awkward to retarget, there's no direct equivalent there.
  - pjmlp 4y ago
    In the sense that it officially supports C, C++ and Fortran on NVidia/PGI toolchain, plus anything else that targets PTX and there are plenty of those, like .NET, Java, Haskell, Julia. It is only a dialect of C++ for those that don't bother to read about what CUDA actually is. https://developer.nvidia.com/language-solutions https://developer.nvidia.com/language-solutions https://developer.nvidia.com/hpc-sdk https://developer.nvidia.com/hpc-sdk

YetAnotherNick 4y ago

I don't use CUDA or ROCm. I use tensorflow/pytorch which supports both of them. But the performance difference between them is huge, as AMD cards have much lower GPGPU/Tensor flops compared to top NVidia cards.

lostmsu 4y ago
Any good benchmarks on large transformers?