5 ms·

From what I can tell, coopmat2 can get to about 75~90% of cuda performance on a single device, and there is no good way to do direct communication across device

From what I can tell, coopmat2 can get to about 75~90% of cuda performance on a single device, and there is no good way to do direct communication across devices. It is fair to say that nobody would replace cuda with coopmat2? That looks like a EEE project that can assigned to a couple of nvidia engineers, to fragment the ecosystem.

my123 12d ago
For reference: https://vulkan.org/user/pages/09.events/vulkanised-2025/T47-Jeff-Bolz-NVIDIA.pdf https://vulkan.org/user/pages/09.events/vulkanised-2025/T47-... coopmat2 features will eventually be rolled elsewhere. coopmat also started as an NVIDIA extension. The client use cases that coopmat was intended for are customer machines, not multi-GPU, which is broadly seen as a datacenter feature instead. That said coopmat orthogonal to this.
- throwdbaaway 12d ago
  So when I said "a couple of nvidia engineers", I indeed meant Jeff. VK_KHR_cooperative_matrix - embrace? VK_NV_cooperative_matrix2 - extend? I am pretty sure VkImportSemaphoreFdInfoKHR, mentioned in https://github.com/ggml-org/llama.cpp/issues/22648 https://github.com/ggml-org/llama.cpp/issues/22648, works across multiple AMD devices, but somehow doesn't work across multiple nvidia devices.
  - my123 12d ago
    > I am pretty sure VkImportSemaphoreFdInfoKHR, mentioned in https://github.com/ggml-org/llama.cpp/issues/22648 https://github.com/ggml-org/llama.cpp/issues/22648, works across multiple AMD devices, but somehow doesn't work across multiple nvidia devices. p2p is disabled on nvidia customer cards, vulkan device groups are shipped for the RTX 6000s > Added support for creating Vulkan logical devices from multiple physical devices on select cards via VK_KHR_device_group_creation. This feature can be enabled by setting the environment variable __VK_ENABLE_DEVICE_GROUPS=1.
- throwdbaaway 12d ago
  Back to the topic about cuda moat, in the slide with title "Problems with Coopmat1", the current frontier open models have absolutely no issue with: * manual pipelining * shared memory staging * tiling * bounds checking
  - my123 12d ago
    A big problem there is ensuring performance portability between different GPUs

my123 12d ago

For reference: https://vulkan.org/user/pages/09.events/vulkanised-2025/T47-Jeff-Bolz-NVIDIA.pdf https://vulkan.org/user/pages/09.events/vulkanised-2025/T47-... coopmat2 features will eventually be rolled elsewhere. coopmat also started as an NVIDIA extension. The client use cases that coopmat was intended for are customer machines, not multi-GPU, which is broadly seen as a datacenter feature instead. That said coopmat orthogonal to this.

throwdbaaway 12d ago
So when I said "a couple of nvidia engineers", I indeed meant Jeff. VK_KHR_cooperative_matrix - embrace? VK_NV_cooperative_matrix2 - extend? I am pretty sure VkImportSemaphoreFdInfoKHR, mentioned in https://github.com/ggml-org/llama.cpp/issues/22648 https://github.com/ggml-org/llama.cpp/issues/22648, works across multiple AMD devices, but somehow doesn't work across multiple nvidia devices.
- my123 12d ago
  > I am pretty sure VkImportSemaphoreFdInfoKHR, mentioned in https://github.com/ggml-org/llama.cpp/issues/22648 https://github.com/ggml-org/llama.cpp/issues/22648, works across multiple AMD devices, but somehow doesn't work across multiple nvidia devices. p2p is disabled on nvidia customer cards, vulkan device groups are shipped for the RTX 6000s > Added support for creating Vulkan logical devices from multiple physical devices on select cards via VK_KHR_device_group_creation. This feature can be enabled by setting the environment variable __VK_ENABLE_DEVICE_GROUPS=1.
throwdbaaway 12d ago
Back to the topic about cuda moat, in the slide with title "Problems with Coopmat1", the current frontier open models have absolutely no issue with: * manual pipelining * shared memory staging * tiling * bounds checking
- my123 12d ago
  A big problem there is ensuring performance portability between different GPUs