4 ms·
I tried getting LLMs to add proper Vulkan support to ik_llama.cpp, which have very good support for CUDA and CPU. The models do an admirable job; they don't car
by throwdbaaway 19d ago
I tried getting LLMs to add proper Vulkan support to ik_llama.cpp, which have very good support for CUDA and CPU. The models do an admirable job; they don't care much about poor DX.
Few problems I noticed:
* coopmat2 from nvidia is the classic embrace, extend, extinguish. No point to ask the models to translate from CUDA to coopmat2. Instead, the models can understand the existing CUDA and CPU kernels, and adapt them accordingly to non-nvidia devices.
* However, the standard API is also lacking. The models struggled to make prompt processing compute-bound on strix halo when the graph is complex. Upfront standard API might just be an evolution dead end.
- my123 19d agocoopmat2 can be implemented by anybody non-nvidia. It's not EEE when the regular coopmat extension is not good enough to get good performance
- throwdbaaway 19d agoFrom what I can tell, coopmat2 can get to about 75~90% of cuda performance on a single device, and there is no good way to do direct communication across devices. It is fair to say that nobody would replace cuda with coopmat2? That looks like a EEE project that can assigned to a couple of nvidia engineers, to fragment the ecosystem.
- my123 18d agoFor reference: https://vulkan.org/user/pages/09.events/vulkanised-2025/T47-Jeff-Bolz-NVIDIA.pdf https://vulkan.org/user/pages/09.events/vulkanised-2025/T47-... coopmat2 features will eventually be rolled elsewhere. coopmat also started as an NVIDIA extension. The client use cases that coopmat was intended for are customer machines, not multi-GPU, which is broadly seen as a datacenter feature instead. That said coopmat orthogonal to this.
- throwdbaaway 18d agoSo when I said "a couple of nvidia engineers", I indeed meant Jeff. VK_KHR_cooperative_matrix - embrace? VK_NV_cooperative_matrix2 - extend? I am pretty sure VkImportSemaphoreFdInfoKHR, mentioned in https://github.com/ggml-org/llama.cpp/issues/22648 https://github.com/ggml-org/llama.cpp/issues/22648, works across multiple AMD devices, but somehow doesn't work across multiple nvidia devices.
- my123 18d ago> I am pretty sure VkImportSemaphoreFdInfoKHR, mentioned in https://github.com/ggml-org/llama.cpp/issues/22648 https://github.com/ggml-org/llama.cpp/issues/22648, works across multiple AMD devices, but somehow doesn't work across multiple nvidia devices. p2p is disabled on nvidia customer cards, vulkan device groups are shipped for the RTX 6000s > Added support for creating Vulkan logical devices from multiple physical devices on select cards via VK_KHR_device_group_creation. This feature can be enabled by setting the environment variable __VK_ENABLE_DEVICE_GROUPS=1.
- throwdbaaway 18d agoBack to the topic about cuda moat, in the slide with title "Problems with Coopmat1", the current frontier open models have absolutely no issue with: * manual pipelining * shared memory staging * tiling * bounds checking
- my123 18d agoA big problem there is ensuring performance portability between different GPUs
- throwdbaaway 19d agoOn the other hand, despite my complain about the standard API, the models were able to come up with cooptmat1 kernels that run dsv4 flash faster than whatever the guys at antirez/ds4 can come up with using rocm, on a strix halo, with the added benefit that I can also pair the strix halo with an egpu to drastically speed things up.