4 ms·
better than Vulkan?
by metalliqaz 7mo ago
better than Vulkan?
- cpburns2009 7mo agoIn my experience using llama.cpp (which ollama uses internally) on a Strix Halo, whether ROCm or Vulkan performs better really depends on the model and it's usually within 10%. I have access to an RX 7900 XT I should compare to though.
- metalliqaz 7mo agoPerhaps I should just google it, but I'm under the impression that ollama uses llama.cpp internally, not the other way around. Thanks for that data point I should experiment with ROCm
- cpburns2009 7mo agoI meant ollama uses llama.cpp internally. Sorry for the confusion.
- naasking 7mo agoFrom what I understand, ROCm is a lot buggier and has some performance regressions on a lot of GPUs in the 7.x series. Vulkan performance for LLMs is apparently not far behind ROCm and is far more stable and predictable at this time.
- hrmtst93837 7mo ago[flagged]
- metalliqaz 7mo agoI was talking about ROCm vs Vulkan. On AMD GPUs, Vulkan has been commonly recognized as the faster API for some time. Both have been slower than CUDA due to most of the hosting projects focusing entirely on Nvidia. Parent post seemed to indicate that newer ROCm releases are better.
- naasking 7mo agoYes, Vulkan is currently faster due to some ROCm regressions: https://github.com/ROCm/ROCm/issues/5805#issuecomment-4141615579 https://github.com/ROCm/ROCm/issues/5805#issuecomment-414161... ROCm should be faster in the end, if they ever fix those issues.
- 0x457 7mo agoFor me Vulkan performs better on integrated cards, but ROCm (MIGraphX) on 7900 XTX.
- nijave 7mo agoAs I understand it, it depends on your GPU and ROCm version but they're similar-ish