4 ms·

One thing AMD can do is working with ggml to make llama.cpp running on AMD GPUs. Compile modern ML framework is a quite complex due to number of OPs. However, r

by smy20011 3y ago

One thing AMD can do is working with ggml to make llama.cpp running on AMD GPUs. Compile modern ML framework is a quite complex due to number of OPs. However, running LLMs does not need a lot of OPs. Just einsum & relu & softmax. Having LLM works with llama.cpp could be done by a team within a week or so.

lhl 3y ago
I agree that AMD should be dedicating an engineer to making sure all the top popular ML/AI projects can run on their hardware, but maybe they also need to spin up a wiki listing compatibility... AMD GPU acceleration via CLBlast was merged back in mid-May in llama.cpp:master - it works and gives a boost (although not all AMD GPUs have been tuned for CLBLast - this is something that AMD should be doing tbt: https://github.com/RadeonOpenCompute/ROCm/issues/2161 https://github.com/RadeonOpenCompute/ROCm/issues/2161) There is also a hipBLAS fork, which is slightly faster (~10% on my old Radeon VII) which maybe someone at AMD should be supporting to make its way into master: https://github.com/ggerganov/llama.cpp/pull/1087 https://github.com/ggerganov/llama.cpp/pull/1087 I'll also note that exllama merged ROCm support and it runs pretty impressively - it runs 2X faster than the hipBLAS llama.cpp, and in fact, on exllama, my old Radeon VII manages to run inference >50% faster than my old (roughly equal class/FP32 perf) GTX 1080 Ti (GCN5 has 2X fp16, and 2X memory bandwidth, so there's probably more headroom even) for a relatively easy port. That's really impressive: https://github.com/turboderp/exllama/pull/7 https://github.com/turboderp/exllama/pull/7 (It's worth noting that for the latter, all you need to do is install the ROCm version of PyTorch and "it just works," which is refreshing: https://pytorch.org/get-started/locally/ https://pytorch.org/get-started/locally/)
- qwertox 3y ago
  > An engineer More like a dozen of engineers over a couple of weeks. Make sure popular LLMs can run on their hardware, as well as Stable Diffusion and other popular projects, and then they will see consumers flock to their hardware. With consumers I mean something like what "gamers" used to be during all these past decades (and still are), those who won't be using it for business cases (what Quadro used to be) but for their hobby; this, but oriented towards AI, which currently is limited to LLM, image generation and ASR. If they focus on this, the community will start helping, maybe even cleaning up their mess of repositories they have on GitHub. There are two benefits over Nvidia: They have more VRAM and they have an open source software stack. They just need to get the basics working for all those hobbyists, those who want to run little projects on their home hardware.

lhl 3y ago

I agree that AMD should be dedicating an engineer to making sure all the top popular ML/AI projects can run on their hardware, but maybe they also need to spin up a wiki listing compatibility... AMD GPU acceleration via CLBlast was merged back in mid-May in llama.cpp:master - it works and gives a boost (although not all AMD GPUs have been tuned for CLBLast - this is something that AMD should be doing tbt: https://github.com/RadeonOpenCompute/ROCm/issues/2161 https://github.com/RadeonOpenCompute/ROCm/issues/2161) There is also a hipBLAS fork, which is slightly faster (~10% on my old Radeon VII) which maybe someone at AMD should be supporting to make its way into master: https://github.com/ggerganov/llama.cpp/pull/1087 https://github.com/ggerganov/llama.cpp/pull/1087 I'll also note that exllama merged ROCm support and it runs pretty impressively - it runs 2X faster than the hipBLAS llama.cpp, and in fact, on exllama, my old Radeon VII manages to run inference >50% faster than my old (roughly equal class/FP32 perf) GTX 1080 Ti (GCN5 has 2X fp16, and 2X memory bandwidth, so there's probably more headroom even) for a relatively easy port. That's really impressive: https://github.com/turboderp/exllama/pull/7 https://github.com/turboderp/exllama/pull/7 (It's worth noting that for the latter, all you need to do is install the ROCm version of PyTorch and "it just works," which is refreshing: https://pytorch.org/get-started/locally/ https://pytorch.org/get-started/locally/)

qwertox 3y ago
> An engineer More like a dozen of engineers over a couple of weeks. Make sure popular LLMs can run on their hardware, as well as Stable Diffusion and other popular projects, and then they will see consumers flock to their hardware. With consumers I mean something like what "gamers" used to be during all these past decades (and still are), those who won't be using it for business cases (what Quadro used to be) but for their hobby; this, but oriented towards AI, which currently is limited to LLM, image generation and ASR. If they focus on this, the community will start helping, maybe even cleaning up their mess of repositories they have on GitHub. There are two benefits over Nvidia: They have more VRAM and they have an open source software stack. They just need to get the basics working for all those hobbyists, those who want to run little projects on their home hardware.