6 ms·

Kompute – Vulkan Alternative to CUDA

204 points by coffeeaddict1 2y ago

Remnant44 2y ago
This looks great - I've been looking for a sustainable, cross-platform-and-vendor GPU compute solution, and the alternatives are not really great. CUDA is nvidia only, Metal is apple only, etc etc. OpenCL has been the closest match but it seems like it's on the way out. Does anyone have real world experience using Vulkan compute shaders versus, say, OpenCL? Does Kompute make things as straightforward as it seems?
- cherryteastain 2y ago
  Check out SYCL
- exDM69 2y ago
  I've done some compute shaders with Vulkan (and a lot of graphics shaders). It's quite nice and simple, after you have gone through the initial hurdle of setting it up (or used a helper library to do it for you). For compute shaders in particular you should enable buffer device addresses, shader subgroups and timeline semaphores (need to be explicitly enabled at init). The downside of compute shaders, regardless of graphics API (gl, dx, vk), is that they have an unspecified time limit after which the OS will kill your process. There are ways to disable this in your OS/GPU configuration but there isn't a portable way of doing this programmatically from your code. Another issue is that if you use the same GPU for compute (or heavy graphics tasks) as your display output, your desktop responsiveness may go down. I had some issue where my desktop got really sluggish when I was drawing some graphics that took 350ms per frame (laptop integrated graphics).
- zozbot234 2y ago
  > OpenCL has been the closest match but it seems like it's on the way out. SYCL is the unofficial successor to OpenCL - in that SYCL implementations like OpenCL are based on SPIR-V compute 'kernels'. (Note that these are not directly compatible with SPIR-V compute 'shaders' as found in Vulkan, so implementing OpenCL or SYCL on top of the Vulkan compute facilities comes with some challenges.)
Conscat 2y ago
Vulkan has some advantages to OpenCL. You gain lower level control over memory allocation and resource synchronization. Rocm has an infamous synchronization pessimization which doesn't exist for Vulkan. You can even explicitly allocate Vulkan resources at specific memory addresses, which means Vulkan can easily be used for embedded devices. But some of the caveats for compute applications are currently: - No bfloat16 in shaders - No shader work graphs (GPU-driven shader control flow) - No inline PTX (inline GCN/RDNA/GEN is available) These may or may not be important to you. Vulkan recently gained an ability to seamlessly dispatch CUDA kernels if you need these in some places, but there aren't currently similar Vulkan extensions for HIP.
- Remnant44 2y ago
  Those are totally reasonable caveats to deal with currently. Thanks!
- tormeh 2y ago
  The biggest advantage of Vulkan is that there are games relying on Vulkan, which means GPU drivers have to play nice with it. My impression is that OpenCL is borderline unsupported. In fact, there is an effort to reimplement OpenCL on top of Vulkan because the drivers are just that shit.
  - chii 2y ago
    > GPU drivers have to play nice with it. i would imagine nvidia to not want their moat with CUDA be uprooted, and so their vulkan support might get gimped in some ways (that does not affect games). At least, that is what the cynic in me would predict.
    - pjmlp 2y ago
      They don't have much to worry, Vulkan SPIR-V doesn't cover everything that PTX has, nor the ecosystem of compilers with PTX backend.
  - 20k 2y ago
    Nvidia have actually been putting a fair amount of work into their OpenCL drivers in the past few years and have a pretty decent implementation, I have a feeling that opencl is more widely used in the embedded space and they were kicked into gear AMD, very recently have also been putting a bit more work into it, though their drivers are still the worst, and still way worse than their pre-rocm drivers
cowmix 2y ago
Pytorch alreadh has Vulkan support -- and Kompute does not support pytorch yet. That's is going to show adaptation of this project.
- juliangoldsmith 2y ago
  A quick Googling shows that Vulkan had an experimental, Android-only Vulkan backend around version 1.7. It doesn't appear to still exist in current versions.
kcb 2y ago
A key component of CUDA is that the kernels are written in C/C++ and not some shader language you would only be familiar with if you were into graphics.
- mepian 2y ago
  CUDA C++ is not exactly standard either.
  - josefx 2y ago
    It probably extends the language less than g++ or msvc do out of the box.
  - pjmlp 2y ago
    Nowadays one can write pretty standard C++20 (minus modules), alongside CUDA frameworks that hide the non standard stuff on their internals.
- darby_nine 2y ago
  I mean it's technically c/c++ but the runtime is so different it might as well be a completely different ecosystem. Also, there's enough tooling popping up around the concept of avoiding using this ecosystem that it's pretty clear people want more than that.
- pjmlp 2y ago
  A key component of CUDA is that kernels can be written in C, C++, Fortran, and any language with PTX compiler backends. The fact many think CUDA is C/C++, is already the first error trying to replace it.
JackYoustra 2y ago
Anyone have a comparison to something like wgsl's compute shader mode over stuff like wgpu? I've never seriously written in either.
- exDM69 2y ago
  They are very similar, but wgsl is its own language. Vulkan can run SPIR-V shaders written in glsl, hlsl or slang. Other experimental languages exist (like rust-gpu). Wgpu is a bit behind on GPU features, for example they've added support for shader subgroups (aka warps or waves) in 2024, where as this feature was available in Vulkan 1.1 released in 2018 or Direct3d 12 shader model 6.0 (same timeframe). Wgpu still does not support buffer device address ("GPU pointers") which I consider quite a game changer. Many popular tools, e.g. RenderDoc, don't have wgpu support. If you are not targeting the web platform, wgpu isn't really bringing anything to the table.
pjmlp 2y ago
Alternatives can only become one, if they support the same set of C, C++, Fortran, and PTX compiler backends, with similar level of IDE integration, grapical GPGPU debugging, and frameworks. Until then they are wannabe alternatives, for a subset of use cases, with lesser tooling. It always feels like those proposing CUDA alternatives don't understand what they are trying to replace, and that is already the first error.
- aniviacat 2y ago
  Are you proposing that they should be doing stuff like frameworks and ide integration all by themselves or are you saying they should magically make a community appear to make them? The product comes before the community (unless you have insane marketing money)
  - pjmlp 2y ago
    I am saying that without them, it isn't an real alternative. Alternatives are supposed to cover all uses cases, otherwise they aren't alternatives. Not even AMD and Intel are able to make it happen, so it remains to be seen how much small communities are able to achieve.
    - PcChip 2y ago
      >Alternatives are supposed to cover all uses cases, otherwise they aren't alternatives. A bicycle is an alternative to a car, however it doesn’t cover all the same use cases
      - pjmlp 2y ago
        With Kompute being the bicycle.
    - indymike 2y ago
      > Alternatives are supposed to cover all uses case I can't think of a single tech product where feature parity was a driver for growth. If all you have is parity, then all you compete on is lower price. Usually, some advantage (better/safer/faster/easier) makes a difference for a few important use cases (good examples were early, feature-incomplete no-sql databases that excelled in one use case that existing SQL servers did not). That advantage hasn't emerged yet, so no community has developed. We'll see if it ever does...
einpoklum 2y ago
This is _not_ an alternative to CUDA nor to OpenCL. It has some high-level and opinionated API [1], which covers a part (rather small part) of the API of each of those. It may, _in principle_, have been developed - with much more work than has gone into it - into such an alternative; but I am actually not sure of that since I have poor command of Vulcan. I got suspicious being someone who maintains C++ API wrappers for CUDA myself [2], and know that just doing that is a lot more code and a lot more work. [1] - I assume it is opinionated to cater to CNN simulation for large language models, and basically not much more. [2] - https://github.com/eyalroz/cuda-api-wrappers/ https://github.com/eyalroz/cuda-api-wrappers/
EVa5I7bHFq9mnYK 2y ago
Can't we make a chip that only does one thing: multiply and add a lot of 32x32 matrices in parallel? I think that would be enough for all AI needs and easy to program.
- jsolson 2y ago
  You're sorta describing TPUs, especially early ones: https://cloud.google.com/blog/products/ai-machine-learning/an-in-depth-look-at-googles-first-tensor-processing-unit-tpu https://cloud.google.com/blog/products/ai-machine-learning/a... That first generation also included the ability to apply a handful of fixed activation functions, but really that's about it. The array is bigger than 32x32, also.
- 0-_-0 2y ago
  Believe it or not, to be able to do that at a high throughput you need half of a GPU anyway, like a cache hierarchy.
  - EVa5I7bHFq9mnYK 2y ago
    Great, eliminating half of transistors is a lot.
    - deleted 2y ago
      [deleted]
    - marshray 2y ago
      You'll need large economies of scale to do better than GPUs, and you'll still have all the same problems of memory IO, packaging, and heat removal. Transistors that aren't in use cost almost nothing. This is a big part of why RISC didn't win and today the largest server chips in use in datacenters are still mostly compatible with an 8 bit part from the early 1980's.
ein0p 2y ago
All you really need form these in Transfofmers-dominated 2024 are GEMM and GEMV, plus fused RMS norm and some element wise primitives to apply RoPE and residuals. And all of that must be brain dead easy to install and access, and it should be cross platform. And yet no such thing exists as far as I can tell.
axsaucedo 2y ago
Kompute author here - thank you very much for sharing our work! If you are interested to learn more, do join the community through our discord here: https://discord.gg/MaH5Jv5zwv https://discord.gg/MaH5Jv5zwv For some background, this project started after seeing various renowned machine learning frameworks like Pytorch and Tensorflow integrating Vulkan as a backend. The Vulkan SDK offers a great low level interface that enables for highly specialized optimizations - however it comes at a cost of highly verbose code which requires 800-2000 lines of code to even begin writing application code. This has resulted in each of these projects having to implement the same baseline to abstract the non-compute related features of the Vulkan SDK. This large amount of non-standardised boiler-plate can result in limited knowledge transfer, higher chance of unique framework implementation bugs being introduced, etc. We are aiming to address this with Kompute. As of today, we are now part of the Linux Foundation, and slowly contributing to the cross-vendor GPGPU revolution. Some of the key features / highlights of Kompute: * C++ SDK with Flexible Python Package * BYOV: Bring-your-own-Vulkan design to play nice with existing Vulkan applications * Asynchronous & parallel processing support through GPU family queues * Explicit relationships for GPU and host memory ownership and memory management: https://kompute.cc/overview/memory-management.html https://kompute.cc/overview/memory-management.html * Robust codebase with 90% unit test code coverage: https://kompute.cc/codecov/ https://kompute.cc/codecov/ * Mobile enabled via Android NDK across several architectures Relevant blog posts: Machine Learning: https://towardsdatascience.com/machine-learning-and-data-processing-in-the-gpu-with-vulkan-kompute-c9350e5e5d3a https://towardsdatascience.com/machine-learning-and-data-pro... Mobile development: https://towardsdatascience.com/gpu-accelerated-machine-learning-in-your-mobile-applications-using-the-android-ndk-vulkan-kompute-1e9da37b7617 https://towardsdatascience.com/gpu-accelerated-machine-learn... Game development (we need to update to Godot4): https://towardsdatascience.com/supercharging-game-development-with-gpu-accelerated-ml-using-vulkan-kompute-the-godot-game-engine-4e75a84ea9f0 https://towardsdatascience.com/supercharging-game-developmen...

204 points by coffeeaddict1 2y ago

This looks great - I've been looking for a sustainable, cross-platform-and-vendor GPU compute solution, and the alternatives are not really great. CUDA is nvidia only, Metal is apple only, etc etc. OpenCL has been the closest match but it seems like it's on the way out. Does anyone have real world experience using Vulkan compute shaders versus, say, OpenCL? Does Kompute make things as straightforward as it seems?

cherryteastain 2y ago
Check out SYCL
exDM69 2y ago
I've done some compute shaders with Vulkan (and a lot of graphics shaders). It's quite nice and simple, after you have gone through the initial hurdle of setting it up (or used a helper library to do it for you). For compute shaders in particular you should enable buffer device addresses, shader subgroups and timeline semaphores (need to be explicitly enabled at init). The downside of compute shaders, regardless of graphics API (gl, dx, vk), is that they have an unspecified time limit after which the OS will kill your process. There are ways to disable this in your OS/GPU configuration but there isn't a portable way of doing this programmatically from your code. Another issue is that if you use the same GPU for compute (or heavy graphics tasks) as your display output, your desktop responsiveness may go down. I had some issue where my desktop got really sluggish when I was drawing some graphics that took 350ms per frame (laptop integrated graphics).
zozbot234 2y ago
> OpenCL has been the closest match but it seems like it's on the way out. SYCL is the unofficial successor to OpenCL - in that SYCL implementations like OpenCL are based on SPIR-V compute 'kernels'. (Note that these are not directly compatible with SPIR-V compute 'shaders' as found in Vulkan, so implementing OpenCL or SYCL on top of the Vulkan compute facilities comes with some challenges.)

Conscat 2y ago

Vulkan has some advantages to OpenCL. You gain lower level control over memory allocation and resource synchronization. Rocm has an infamous synchronization pessimization which doesn't exist for Vulkan. You can even explicitly allocate Vulkan resources at specific memory addresses, which means Vulkan can easily be used for embedded devices. But some of the caveats for compute applications are currently: - No bfloat16 in shaders - No shader work graphs (GPU-driven shader control flow) - No inline PTX (inline GCN/RDNA/GEN is available) These may or may not be important to you. Vulkan recently gained an ability to seamlessly dispatch CUDA kernels if you need these in some places, but there aren't currently similar Vulkan extensions for HIP.

Remnant44 2y ago
Those are totally reasonable caveats to deal with currently. Thanks!
tormeh 2y ago
The biggest advantage of Vulkan is that there are games relying on Vulkan, which means GPU drivers have to play nice with it. My impression is that OpenCL is borderline unsupported. In fact, there is an effort to reimplement OpenCL on top of Vulkan because the drivers are just that shit.
- chii 2y ago
  > GPU drivers have to play nice with it. i would imagine nvidia to not want their moat with CUDA be uprooted, and so their vulkan support might get gimped in some ways (that does not affect games). At least, that is what the cynic in me would predict.
  - pjmlp 2y ago
    They don't have much to worry, Vulkan SPIR-V doesn't cover everything that PTX has, nor the ecosystem of compilers with PTX backend.
- 20k 2y ago
  Nvidia have actually been putting a fair amount of work into their OpenCL drivers in the past few years and have a pretty decent implementation, I have a feeling that opencl is more widely used in the embedded space and they were kicked into gear AMD, very recently have also been putting a bit more work into it, though their drivers are still the worst, and still way worse than their pre-rocm drivers

cowmix 2y ago

Pytorch alreadh has Vulkan support -- and Kompute does not support pytorch yet. That's is going to show adaptation of this project.

juliangoldsmith 2y ago
A quick Googling shows that Vulkan had an experimental, Android-only Vulkan backend around version 1.7. It doesn't appear to still exist in current versions.

kcb 2y ago

A key component of CUDA is that the kernels are written in C/C++ and not some shader language you would only be familiar with if you were into graphics.

mepian 2y ago
CUDA C++ is not exactly standard either.
- josefx 2y ago
  It probably extends the language less than g++ or msvc do out of the box.
- pjmlp 2y ago
  Nowadays one can write pretty standard C++20 (minus modules), alongside CUDA frameworks that hide the non standard stuff on their internals.
darby_nine 2y ago
I mean it's technically c/c++ but the runtime is so different it might as well be a completely different ecosystem. Also, there's enough tooling popping up around the concept of avoiding using this ecosystem that it's pretty clear people want more than that.
pjmlp 2y ago
A key component of CUDA is that kernels can be written in C, C++, Fortran, and any language with PTX compiler backends. The fact many think CUDA is C/C++, is already the first error trying to replace it.

JackYoustra 2y ago

Anyone have a comparison to something like wgsl's compute shader mode over stuff like wgpu? I've never seriously written in either.

exDM69 2y ago
They are very similar, but wgsl is its own language. Vulkan can run SPIR-V shaders written in glsl, hlsl or slang. Other experimental languages exist (like rust-gpu). Wgpu is a bit behind on GPU features, for example they've added support for shader subgroups (aka warps or waves) in 2024, where as this feature was available in Vulkan 1.1 released in 2018 or Direct3d 12 shader model 6.0 (same timeframe). Wgpu still does not support buffer device address ("GPU pointers") which I consider quite a game changer. Many popular tools, e.g. RenderDoc, don't have wgpu support. If you are not targeting the web platform, wgpu isn't really bringing anything to the table.

pjmlp 2y ago

Alternatives can only become one, if they support the same set of C, C++, Fortran, and PTX compiler backends, with similar level of IDE integration, grapical GPGPU debugging, and frameworks. Until then they are wannabe alternatives, for a subset of use cases, with lesser tooling. It always feels like those proposing CUDA alternatives don't understand what they are trying to replace, and that is already the first error.

aniviacat 2y ago
Are you proposing that they should be doing stuff like frameworks and ide integration all by themselves or are you saying they should magically make a community appear to make them? The product comes before the community (unless you have insane marketing money)
- pjmlp 2y ago
  I am saying that without them, it isn't an real alternative. Alternatives are supposed to cover all uses cases, otherwise they aren't alternatives. Not even AMD and Intel are able to make it happen, so it remains to be seen how much small communities are able to achieve.
  - PcChip 2y ago
    >Alternatives are supposed to cover all uses cases, otherwise they aren't alternatives. A bicycle is an alternative to a car, however it doesn’t cover all the same use cases
    - pjmlp 2y ago
      With Kompute being the bicycle.
  - indymike 2y ago
    > Alternatives are supposed to cover all uses case I can't think of a single tech product where feature parity was a driver for growth. If all you have is parity, then all you compete on is lower price. Usually, some advantage (better/safer/faster/easier) makes a difference for a few important use cases (good examples were early, feature-incomplete no-sql databases that excelled in one use case that existing SQL servers did not). That advantage hasn't emerged yet, so no community has developed. We'll see if it ever does...

einpoklum 2y ago

This is _not_ an alternative to CUDA nor to OpenCL. It has some high-level and opinionated API [1], which covers a part (rather small part) of the API of each of those. It may, _in principle_, have been developed - with much more work than has gone into it - into such an alternative; but I am actually not sure of that since I have poor command of Vulcan. I got suspicious being someone who maintains C++ API wrappers for CUDA myself [2], and know that just doing that is a lot more code and a lot more work. [1] - I assume it is opinionated to cater to CNN simulation for large language models, and basically not much more. [2] - https://github.com/eyalroz/cuda-api-wrappers/ https://github.com/eyalroz/cuda-api-wrappers/

EVa5I7bHFq9mnYK 2y ago

Can't we make a chip that only does one thing: multiply and add a lot of 32x32 matrices in parallel? I think that would be enough for all AI needs and easy to program.

jsolson 2y ago
You're sorta describing TPUs, especially early ones: https://cloud.google.com/blog/products/ai-machine-learning/an-in-depth-look-at-googles-first-tensor-processing-unit-tpu https://cloud.google.com/blog/products/ai-machine-learning/a... That first generation also included the ability to apply a handful of fixed activation functions, but really that's about it. The array is bigger than 32x32, also.
0-_-0 2y ago
Believe it or not, to be able to do that at a high throughput you need half of a GPU anyway, like a cache hierarchy.
- EVa5I7bHFq9mnYK 2y ago
  Great, eliminating half of transistors is a lot.
  - deleted 2y ago
    [deleted]
  - marshray 2y ago
    You'll need large economies of scale to do better than GPUs, and you'll still have all the same problems of memory IO, packaging, and heat removal. Transistors that aren't in use cost almost nothing. This is a big part of why RISC didn't win and today the largest server chips in use in datacenters are still mostly compatible with an 8 bit part from the early 1980's.

ein0p 2y ago

All you really need form these in Transfofmers-dominated 2024 are GEMM and GEMV, plus fused RMS norm and some element wise primitives to apply RoPE and residuals. And all of that must be brain dead easy to install and access, and it should be cross platform. And yet no such thing exists as far as I can tell.

axsaucedo 2y ago

Kompute author here - thank you very much for sharing our work! If you are interested to learn more, do join the community through our discord here: https://discord.gg/MaH5Jv5zwv https://discord.gg/MaH5Jv5zwv For some background, this project started after seeing various renowned machine learning frameworks like Pytorch and Tensorflow integrating Vulkan as a backend. The Vulkan SDK offers a great low level interface that enables for highly specialized optimizations - however it comes at a cost of highly verbose code which requires 800-2000 lines of code to even begin writing application code. This has resulted in each of these projects having to implement the same baseline to abstract the non-compute related features of the Vulkan SDK. This large amount of non-standardised boiler-plate can result in limited knowledge transfer, higher chance of unique framework implementation bugs being introduced, etc. We are aiming to address this with Kompute. As of today, we are now part of the Linux Foundation, and slowly contributing to the cross-vendor GPGPU revolution. Some of the key features / highlights of Kompute: * C++ SDK with Flexible Python Package * BYOV: Bring-your-own-Vulkan design to play nice with existing Vulkan applications * Asynchronous & parallel processing support through GPU family queues * Explicit relationships for GPU and host memory ownership and memory management: https://kompute.cc/overview/memory-management.html https://kompute.cc/overview/memory-management.html * Robust codebase with 90% unit test code coverage: https://kompute.cc/codecov/ https://kompute.cc/codecov/ * Mobile enabled via Android NDK across several architectures Relevant blog posts: Machine Learning: https://towardsdatascience.com/machine-learning-and-data-processing-in-the-gpu-with-vulkan-kompute-c9350e5e5d3a https://towardsdatascience.com/machine-learning-and-data-pro... Mobile development: https://towardsdatascience.com/gpu-accelerated-machine-learning-in-your-mobile-applications-using-the-android-ndk-vulkan-kompute-1e9da37b7617 https://towardsdatascience.com/gpu-accelerated-machine-learn... Game development (we need to update to Godot4): https://towardsdatascience.com/supercharging-game-development-with-gpu-accelerated-ml-using-vulkan-kompute-the-godot-game-engine-4e75a84ea9f0 https://towardsdatascience.com/supercharging-game-developmen...