3 ms·

I think ROCm is just a total second class citizen in the space TBH. It's a shame coz it's not even really what we want, we would obviously all be better served

I think ROCm is just a total second class citizen in the space TBH. It's a shame coz it's not even really what we want, we would obviously all be better served if we could use Vulkan or something. But I guess it's inevitable that a generic framework lags behind here. If I was AMD I'd hire a whole ecosystem team to sit next to the ROCm people and just support big users like llama.cpp to work better on their HW, e.g. giving OSS maintainers access to their board farms. Maybe they have already done that, in which case I guess I should say I'd double the size of that team.

roenxi 2mo ago
You probably can just use Vulkan. Why don't you think Vulkan is an alternative? Can it not handle BLAS and memory access? What is a graphics card supposed to be doing that isn't efficient in Vulkan?
- binary132 2mo ago
  Sometimes it’s just that a particular backend is poorly optimized or has a regression on a particular platform as compared to the “mainstream” backends. For example, whisper.cpp’s Vulkan backend performs 2-3x worse than CPU on my Snapdragon X2 laptop when using the ggerganov v3 turbo model. It’s probably a simple fix, but it does need to be fixed.