4 ms·

This is at the core of the matter for me, and my knowledge is too weak to understand why this is the case. I don't enjoy the idea of relying on Nvidia's stack f

This is at the core of the matter for me, and my knowledge is too weak to understand why this is the case. I don't enjoy the idea of relying on Nvidia's stack for GPU compute, but the alternatives I've tried (e.g. Vulkan compute) are higher friction to use. I am trying to reconcile why; Nvidia shouldn't have this moat. My software is labeled "CPU only unless using an nVidia GPU". I would prefer to strikethrough "nVidia". Incidentally, this means no more Mac support.

swerner 19d ago
Vulkan compute is not really designed or intended to be a CUDA competitor, its feature set is much more restricted, and Vulkan host side code is much more verbose than CUDA. OpenCL or SYCL are much closer in features to CUDA. I found that when using SYCL on Nvidia, debugging symbols etc can be passed through and you can use tools like NSight Compute to profile it as if it were CUDA.
- throwdbaaway 19d ago
  I tried getting LLMs to add proper Vulkan support to ik_llama.cpp, which have very good support for CUDA and CPU. The models do an admirable job; they don't care much about poor DX. Few problems I noticed: * coopmat2 from nvidia is the classic embrace, extend, extinguish. No point to ask the models to translate from CUDA to coopmat2. Instead, the models can understand the existing CUDA and CPU kernels, and adapt them accordingly to non-nvidia devices. * However, the standard API is also lacking. The models struggled to make prompt processing compute-bound on strix halo when the graph is complex. Upfront standard API might just be an evolution dead end.
  - my123 19d ago
    coopmat2 can be implemented by anybody non-nvidia. It's not EEE when the regular coopmat extension is not good enough to get good performance
    - throwdbaaway 19d ago
      From what I can tell, coopmat2 can get to about 75~90% of cuda performance on a single device, and there is no good way to do direct communication across devices. It is fair to say that nobody would replace cuda with coopmat2? That looks like a EEE project that can assigned to a couple of nvidia engineers, to fragment the ecosystem.
      - my123 19d ago
        For reference: https://vulkan.org/user/pages/09.events/vulkanised-2025/T47-Jeff-Bolz-NVIDIA.pdf https://vulkan.org/user/pages/09.events/vulkanised-2025/T47-... coopmat2 features will eventually be rolled elsewhere. coopmat also started as an NVIDIA extension. The client use cases that coopmat was intended for are customer machines, not multi-GPU, which is broadly seen as a datacenter feature instead. That said coopmat orthogonal to this.
        throwdbaaway 19d ago
        So when I said "a couple of nvidia engineers", I indeed meant Jeff. VK_KHR_cooperative_matrix - embrace? VK_NV_cooperative_matrix2 - extend? I am pretty sure VkImportSemaphoreFdInfoKHR, mentioned in https://github.com/ggml-org/llama.cpp/issues/22648 https://github.com/ggml-org/llama.cpp/issues/22648, works across multiple AMD devices, but somehow doesn't work across multiple nvidia devices.
        my123 19d ago
        > I am pretty sure VkImportSemaphoreFdInfoKHR, mentioned in https://github.com/ggml-org/llama.cpp/issues/22648 https://github.com/ggml-org/llama.cpp/issues/22648, works across multiple AMD devices, but somehow doesn't work across multiple nvidia devices. p2p is disabled on nvidia customer cards, vulkan device groups are shipped for the RTX 6000s > Added support for creating Vulkan logical devices from multiple physical devices on select cards via VK_KHR_device_group_creation. This feature can be enabled by setting the environment variable __VK_ENABLE_DEVICE_GROUPS=1.
        19d ago
- mschuetz 19d ago
  It could easily be a competitor to Cuda, if it just made things easier. Like, why does it take 50 lines of code to allocate memory in vulkan, and just one single line in cuda? Vulkan should just provide a single-line gpuMalloc convenience function. And not just for allocation, for all the other nonsense as well.
pjmlp 19d ago
All commercial APIs have this moat, which is why other than Carmack famously advocating for OpenGL, the large majority of other devs never cared, they implement an API agnostic engine, use the best API for each platform, and move on with the rest of the game. Even Carmack later admitted that DirectX had become much better than OpenGL, but they weren't going to change given the studio culture. See https://www.bit-tech.net/news/gaming/pc/carmack-directx-better-opengl/1 https://www.bit-tech.net/news/gaming/pc/carmack-directx-bett...

swerner 19d ago

Vulkan compute is not really designed or intended to be a CUDA competitor, its feature set is much more restricted, and Vulkan host side code is much more verbose than CUDA. OpenCL or SYCL are much closer in features to CUDA. I found that when using SYCL on Nvidia, debugging symbols etc can be passed through and you can use tools like NSight Compute to profile it as if it were CUDA.

throwdbaaway 19d ago
I tried getting LLMs to add proper Vulkan support to ik_llama.cpp, which have very good support for CUDA and CPU. The models do an admirable job; they don't care much about poor DX. Few problems I noticed: * coopmat2 from nvidia is the classic embrace, extend, extinguish. No point to ask the models to translate from CUDA to coopmat2. Instead, the models can understand the existing CUDA and CPU kernels, and adapt them accordingly to non-nvidia devices. * However, the standard API is also lacking. The models struggled to make prompt processing compute-bound on strix halo when the graph is complex. Upfront standard API might just be an evolution dead end.
- my123 19d ago
  coopmat2 can be implemented by anybody non-nvidia. It's not EEE when the regular coopmat extension is not good enough to get good performance
  - throwdbaaway 19d ago
    From what I can tell, coopmat2 can get to about 75~90% of cuda performance on a single device, and there is no good way to do direct communication across devices. It is fair to say that nobody would replace cuda with coopmat2? That looks like a EEE project that can assigned to a couple of nvidia engineers, to fragment the ecosystem.
    - my123 19d ago
      For reference: https://vulkan.org/user/pages/09.events/vulkanised-2025/T47-Jeff-Bolz-NVIDIA.pdf https://vulkan.org/user/pages/09.events/vulkanised-2025/T47-... coopmat2 features will eventually be rolled elsewhere. coopmat also started as an NVIDIA extension. The client use cases that coopmat was intended for are customer machines, not multi-GPU, which is broadly seen as a datacenter feature instead. That said coopmat orthogonal to this.
      - throwdbaaway 19d ago
        So when I said "a couple of nvidia engineers", I indeed meant Jeff. VK_KHR_cooperative_matrix - embrace? VK_NV_cooperative_matrix2 - extend? I am pretty sure VkImportSemaphoreFdInfoKHR, mentioned in https://github.com/ggml-org/llama.cpp/issues/22648 https://github.com/ggml-org/llama.cpp/issues/22648, works across multiple AMD devices, but somehow doesn't work across multiple nvidia devices.
        my123 19d ago
        > I am pretty sure VkImportSemaphoreFdInfoKHR, mentioned in https://github.com/ggml-org/llama.cpp/issues/22648 https://github.com/ggml-org/llama.cpp/issues/22648, works across multiple AMD devices, but somehow doesn't work across multiple nvidia devices. p2p is disabled on nvidia customer cards, vulkan device groups are shipped for the RTX 6000s > Added support for creating Vulkan logical devices from multiple physical devices on select cards via VK_KHR_device_group_creation. This feature can be enabled by setting the environment variable __VK_ENABLE_DEVICE_GROUPS=1.
      - 19d ago
mschuetz 19d ago
It could easily be a competitor to Cuda, if it just made things easier. Like, why does it take 50 lines of code to allocate memory in vulkan, and just one single line in cuda? Vulkan should just provide a single-line gpuMalloc convenience function. And not just for allocation, for all the other nonsense as well.

pjmlp 19d ago

All commercial APIs have this moat, which is why other than Carmack famously advocating for OpenGL, the large majority of other devs never cared, they implement an API agnostic engine, use the best API for each platform, and move on with the rest of the game. Even Carmack later admitted that DirectX had become much better than OpenGL, but they weren't going to change given the studio culture. See https://www.bit-tech.net/news/gaming/pc/carmack-directx-better-opengl/1 https://www.bit-tech.net/news/gaming/pc/carmack-directx-bett...