4 ms·
It also seems like the tools aren't there to fully utilize them. Unless I misunderstood he was running off CPU only for all the test so there's still the iGPU a
by rtkwe 1y ago
It also seems like the tools aren't there to fully utilize them. Unless I misunderstood he was running off CPU only for all the test so there's still the iGPU and NPU performance that's not been utilized in these tests.
- geerlingguy 1y agoNo, only a couple initial tests with Ollama used CPU. I ran most tests on Vulkan / iGPU, and some on ROCm (read further down the thread). I found it difficult to install ROCm on Fedora 42 but after upgrading to Rawhide it was easy, so I re-tested everything with ROCm vs Vulkan. Ollama, for some silly reason, doesn't support Vulkan even though I've used a fork many times to get full GPU acceleration with it on Pi, Ampere, and even this AMD system... (moral of the story just stick with llama.cpp).
- edwinjones 1y agoSadly, the reason they give is subjectively terrible: https://x.com/ollama/status/1952783981000446029 https://x.com/ollama/status/1952783981000446029 No experimental flag option, no "you can use the fork that works fine but we don't have capacity to support this" just a hard "no, we think it's unreliable". I guess they just want you to drop them and use llama.cpp.
- geerlingguy 1y agoYeah, my conspiracy theory is Nvidia is somehow influencing the decision. If you can do Vulkan with Ollama, it opens up people to using Intel/AMD/other iGPUs and you might not be incentivized to buy an Nvidia GPU. ROCm support is not wonderful. It's certainly worse for an end user to deal with than Vulkan, which usually 'just works'.
- edwinjones 1y agoI agree. AMD should just go all in on vulkan I think, The ROCm compatibility list is terrible compared to...every modern device and probably some ancient gpus that can be made to work with vulkan as well. Considering they created mantle, you would think it would be the obvious move too.
- MindSpunk 1y agoVulkan is Mantle. Vulkan was developed out of the original Mantle API that AMD brought to Khronos. What do you mean "AMD should just go all in on Vulkan"? They've been "all in" on Vulkan from the beginning because they were one of the lead authors of the API.
- dagmx 1y agoVulkan is a derivative of mantle sure, but it is quite different than what Mantle was. There was a period in between where AMD basically EOL’d mantle and Vulkan wasn’t even in the works yet.
- edwinjones 1y agoI would say vulkan derives from mantle, mantle development stopped some time ago.
- zozbot234 1y agoiGPUs (and NPUs) are not very useful for LLM inference, they only help somewhat in the prompt pre-processing phase. The CPU has worse bulk compute but far better access to system memory bandwidth, so it wins in token generation where that's the main factor. My conspiracy theory is that it would help if contributors kept the Vulkan Compute proposed support up to date with new Ollama versions; no maintainer wants to deal with out-of-date pull req's.
- jcastro 1y agoHi Jeff, I'm a linux ambassador for Framework and I have one of these units. It'd be interesting if you would install ramalama in fedora and test that. I've been using that out of the box as a drop in replacement for ollama and everything was GPU accelerated out of the box. It pulls rocm from a container and just figures it out, etc. Would love to see actual numbers though. Great work on this!
- nicolaslem 1y agoI gave it a try on my Strix Halo laptop on Arch. ROCm and everything else worked out of the box with two commands: uv tool install ramalama ramalama serve <model>