5 ms·

Why this infrastructure is so tightly coupled with CUDA? CUDA is very specific and closed APIs for NVidia hardware only. Programming languages should focus on m

Why this infrastructure is so tightly coupled with CUDA? CUDA is very specific and closed APIs for NVidia hardware only. Programming languages should focus on more general primitives that might work on NVidia or TPUs or something else. PyTorch also has CUDA all over in its APIs and its frustrating to see such tight binding with closed one company API. Also take a look at OpenCL.

darknoon 7y ago
It's because CUDA performs better. It's not nice, but it's the situation we're living in. Particularly AMD support and performance are lot.
- sytelus 7y ago
  Sure, I don't mind CUDA backend as first class citizen. I'm talking about having my code sprinkled with word "cuda" all over. Why can't I write my code that is bit more abstract and potentially compilable to different backends? That is, think about the primitives instead of tightly getting married to cuda forever. AMD performance might not be good today but how about 10 years later? How about using TPUs instead? or FPGAs (if someone creates backend for it)?
  - KenoFischer 7y ago
    Well, one problem is that you're reading an NVIDIA marketing post on an NVIDIA blog talking in particular about the lowest levels of the stack targeting NVIDIA hardware. Higher level abstractions can and do just work across different hardware backends (not as well as we'd like, but we have some thoughts on how to improve that).
- shmerl 7y ago
  It doesn't perform better than what you can do in Vulkan. It's simply more entrenched.
  - deleted 7y ago
    [deleted]
  - backdoorsgalore 7y ago
    WebGPU version of TensorFlow is in the works. Hopefully, that will dethrone NVIDIA. AFAIK, it's significantly more performant than WebGL, and it works on top of Vulkan, D3D and Metal. The heck with NVIDIA. Such a bunch of control freaks.
- Athas 7y ago
  Are you certain that the story is as simple as "CUDA performs better"? It's common folklore, but I have seen little evidence. The only situations I know of when CUDA performs better is when CUDA-specific features are used (if they are relevant for whatever problem is at hand). Also, CUDA libraries (like cuBLAS or cuFFT) tend to be more efficient than their OpenCL equivalent, which is likely because much more work has gone into them. I have also noted that the CUDA compiler is willing to use less accurate (but faster) floating-point instructions by default (for things like e.g. inverse square root), where you need to pass options to the OpenCL compiler for it to do the same. This will matter for some programs. In fact, I have run tens of thousands of lines of essentially equivalent CUDA and OpenCL code (automatically generated) on the same hardware, and performance was in all cases very similar[0]. If anything, CUDA was actually slower than average (but in the cases I investigated, this was down to arbitrary differences like the CUDA compiler not unrolling some loops as aggressively and such). [0]: https://futhark-lang.org/blog/2019-02-08-futhark-0.9.1-released.html#the-cuda-backend https://futhark-lang.org/blog/2019-02-08-futhark-0.9.1-relea...
  - llukas 7y ago
    Did you compare the performance of nvrtc vs offline nvcc compiler?
    - Athas 7y ago
      No; the code we would need to generate would be rather different. Would you expect a significant difference? When we did research on nvrtc before implementing this, we couldn't find any concrete information that nvrtc should generate slower code.
dlphn___xyz 7y ago
there are specific benefits of cuda over opencl: see https://arxiv.org/vc/arxiv/papers/1005/1005.2581v1.pdf https://arxiv.org/vc/arxiv/papers/1005/1005.2581v1.pdf
- sytelus 7y ago
  Yes, but is cuda going to keep its edge 10 years down the line? Do I want to hardcode my algorithms so tightly with today's cuda APIs? Can there be better more generic primitives that are agnostic of propitiatory cuda APIs but would support it as backend without too much perf hit?
  - xiaodai 7y ago
    If you want performance then yeah. If you are after hypothetical performance in future which may not even materialise, then the choice is yours. Everyone knows where the sensible ground is. Which, unfortunately, is CUDA only
  - shaklee3 7y ago
    AMD has a search and replace library that's API compatible with many cuda functions now. It hasn't caught on yet, but if they release decent hardware soon, it might.
  - refresh-creds 7y ago
    That paper is already about 10 years old so I think you are being trolled.
idnefju 7y ago
OpenCL isn't in a good place. CUDA has become the industry standard.
shmerl 7y ago
Because Nvidia likes lock-in. It totally doesn't have to. Today we have Vulkan for general purpose GPU programming.
- llukas 7y ago
  OpenCL, hip, Vulkan, what tomorrow? Or alternatively, there were cl* libraries, roc* libraries and hip* libraries for AMD? Which ones are supported? CUDA doesn't require rewriting code with ${OSS} framework of the year, every year. They need to earn that lock-in with future compatibility guarantees which none of OSS projects has.
  - shmerl 7y ago
    Whatever it is, as long as it's not tied to one GPU only, it could be promising. Something that's tied to Nvidia or anyone else exclusively is not good, and surely isn't democratizing anything. > CUDA doesn't require rewriting code with ${OSS} framework of the year, every year. How so? Change the GPU from Nvidia, and you are forced to rewrite code. That's the whole point of lock-in, it's a tax on developers. CUDA doens't guarantee you anything, if you don't stick with their GPUs. Vulkan on the the other hand has conformance requirements.
jlebar 7y ago
> Why this infrastructure is so tightly coupled with CUDA? It's not. It uses LLVM, which can easily target AMD GPUs. (Whether the Julia folks have invested in making this work, I dunno, but it's not Extremely Hard.) Understandably nvidia gives you the wrong impression.
- wallnuss 7y ago
  We are indeed interested in targeting AMD GPUs. There is a prototype backend available at https://github.com/JuliaGPU/AMDGPUnative.jl https://github.com/JuliaGPU/AMDGPUnative.jl and we are closely following the status of SPIR-V and Intel GPUs in LLVM. The focus on CUDA comes from the fact that most HPC systems for scientific computing are using Nvidia GPUs. That is finally slowly changing.
maleadt 7y ago
Our view is that to get performance out of a system (here CUDA), it's better not to start abstracting it right away. So we have CUDAnative.jl and CUDAdrv.jl for fairly low-level CUDA programming, albeit in a high-level language. However, with CuArrays.jl we implement the Julia array interface for CUDA GPUs. That means you can write array code for one platform (CPU using Base.Array) and start using hardware accelerators by just switching the array type (CUDA GPU using CuArray). Of course, real-life applications might still need to use CUDA specific functionality for one reason or another, but at least you can get most of the way without platform-specific programming.
pjmlp 7y ago
Because Khronos up to a little while lived on a bubble that we have to use C, write our own compiler and linking logic to use GPGPUs and collect debugging toolchains from each OEM. Only when they started getting a beating of PTX bytecode and multi-language deployment on CUDA did they woke up and came up with SPIR (later SPIR-V) and SYCL, which still isn't widely deployed.

darknoon 7y ago

It's because CUDA performs better. It's not nice, but it's the situation we're living in. Particularly AMD support and performance are lot.

sytelus 7y ago
Sure, I don't mind CUDA backend as first class citizen. I'm talking about having my code sprinkled with word "cuda" all over. Why can't I write my code that is bit more abstract and potentially compilable to different backends? That is, think about the primitives instead of tightly getting married to cuda forever. AMD performance might not be good today but how about 10 years later? How about using TPUs instead? or FPGAs (if someone creates backend for it)?
- KenoFischer 7y ago
  Well, one problem is that you're reading an NVIDIA marketing post on an NVIDIA blog talking in particular about the lowest levels of the stack targeting NVIDIA hardware. Higher level abstractions can and do just work across different hardware backends (not as well as we'd like, but we have some thoughts on how to improve that).
shmerl 7y ago
It doesn't perform better than what you can do in Vulkan. It's simply more entrenched.
- deleted 7y ago
  [deleted]
- backdoorsgalore 7y ago
  WebGPU version of TensorFlow is in the works. Hopefully, that will dethrone NVIDIA. AFAIK, it's significantly more performant than WebGL, and it works on top of Vulkan, D3D and Metal. The heck with NVIDIA. Such a bunch of control freaks.
Athas 7y ago
Are you certain that the story is as simple as "CUDA performs better"? It's common folklore, but I have seen little evidence. The only situations I know of when CUDA performs better is when CUDA-specific features are used (if they are relevant for whatever problem is at hand). Also, CUDA libraries (like cuBLAS or cuFFT) tend to be more efficient than their OpenCL equivalent, which is likely because much more work has gone into them. I have also noted that the CUDA compiler is willing to use less accurate (but faster) floating-point instructions by default (for things like e.g. inverse square root), where you need to pass options to the OpenCL compiler for it to do the same. This will matter for some programs. In fact, I have run tens of thousands of lines of essentially equivalent CUDA and OpenCL code (automatically generated) on the same hardware, and performance was in all cases very similar[0]. If anything, CUDA was actually slower than average (but in the cases I investigated, this was down to arbitrary differences like the CUDA compiler not unrolling some loops as aggressively and such). [0]: https://futhark-lang.org/blog/2019-02-08-futhark-0.9.1-released.html#the-cuda-backend https://futhark-lang.org/blog/2019-02-08-futhark-0.9.1-relea...
- llukas 7y ago
  Did you compare the performance of nvrtc vs offline nvcc compiler?
  - Athas 7y ago
    No; the code we would need to generate would be rather different. Would you expect a significant difference? When we did research on nvrtc before implementing this, we couldn't find any concrete information that nvrtc should generate slower code.

dlphn___xyz 7y ago

there are specific benefits of cuda over opencl: see https://arxiv.org/vc/arxiv/papers/1005/1005.2581v1.pdf https://arxiv.org/vc/arxiv/papers/1005/1005.2581v1.pdf

sytelus 7y ago
Yes, but is cuda going to keep its edge 10 years down the line? Do I want to hardcode my algorithms so tightly with today's cuda APIs? Can there be better more generic primitives that are agnostic of propitiatory cuda APIs but would support it as backend without too much perf hit?
- xiaodai 7y ago
  If you want performance then yeah. If you are after hypothetical performance in future which may not even materialise, then the choice is yours. Everyone knows where the sensible ground is. Which, unfortunately, is CUDA only
- shaklee3 7y ago
  AMD has a search and replace library that's API compatible with many cuda functions now. It hasn't caught on yet, but if they release decent hardware soon, it might.
- refresh-creds 7y ago
  That paper is already about 10 years old so I think you are being trolled.

idnefju 7y ago

OpenCL isn't in a good place. CUDA has become the industry standard.

shmerl 7y ago

Because Nvidia likes lock-in. It totally doesn't have to. Today we have Vulkan for general purpose GPU programming.

llukas 7y ago
OpenCL, hip, Vulkan, what tomorrow? Or alternatively, there were cl* libraries, roc* libraries and hip* libraries for AMD? Which ones are supported? CUDA doesn't require rewriting code with ${OSS} framework of the year, every year. They need to earn that lock-in with future compatibility guarantees which none of OSS projects has.
- shmerl 7y ago
  Whatever it is, as long as it's not tied to one GPU only, it could be promising. Something that's tied to Nvidia or anyone else exclusively is not good, and surely isn't democratizing anything. > CUDA doesn't require rewriting code with ${OSS} framework of the year, every year. How so? Change the GPU from Nvidia, and you are forced to rewrite code. That's the whole point of lock-in, it's a tax on developers. CUDA doens't guarantee you anything, if you don't stick with their GPUs. Vulkan on the the other hand has conformance requirements.

jlebar 7y ago

> Why this infrastructure is so tightly coupled with CUDA? It's not. It uses LLVM, which can easily target AMD GPUs. (Whether the Julia folks have invested in making this work, I dunno, but it's not Extremely Hard.) Understandably nvidia gives you the wrong impression.

wallnuss 7y ago
We are indeed interested in targeting AMD GPUs. There is a prototype backend available at https://github.com/JuliaGPU/AMDGPUnative.jl https://github.com/JuliaGPU/AMDGPUnative.jl and we are closely following the status of SPIR-V and Intel GPUs in LLVM. The focus on CUDA comes from the fact that most HPC systems for scientific computing are using Nvidia GPUs. That is finally slowly changing.

maleadt 7y ago

Our view is that to get performance out of a system (here CUDA), it's better not to start abstracting it right away. So we have CUDAnative.jl and CUDAdrv.jl for fairly low-level CUDA programming, albeit in a high-level language. However, with CuArrays.jl we implement the Julia array interface for CUDA GPUs. That means you can write array code for one platform (CPU using Base.Array) and start using hardware accelerators by just switching the array type (CUDA GPU using CuArray). Of course, real-life applications might still need to use CUDA specific functionality for one reason or another, but at least you can get most of the way without platform-specific programming.

pjmlp 7y ago

Because Khronos up to a little while lived on a bubble that we have to use C, write our own compiler and linking logic to use GPGPUs and collect debugging toolchains from each OEM. Only when they started getting a beating of PTX bytecode and multi-language deployment on CUDA did they woke up and came up with SPIR (later SPIR-V) and SYCL, which still isn't widely deployed.