7 ms·
For inference, if you have a supported card (or probably architecture if you are on Linux and can use HSA_OVERRIDE_GFX_VERSION), then you can probably run anyt
by lhl 2y ago
For inference, if you have a supported card (or probably architecture if you are on Linux and can use HSA_OVERRIDE_GFX_VERSION), then you can probably run anything with (upstream) PyTorch and transformers. Also, compiling llama.cpp is has been pretty trouble-free for me for at least a year.
(If you are on Windows, there is usually a win-hip binary of llama.cpp in the project's releases or if things totally refuse to work, you can use the Vulkan build as a (less performant) fallback).
Having more options can't hurt, but ROCm 5.4.2 is almost 2 years old, and things have come a long way since then, so I'm curious about this being published freshly today, in October 2024.
BTW, I recently went through and updated my compatibility doc (focused on RDNA3) w/ ROCm 6.2 for those interested. A lot has changed just in the past few months (upstream bitsandbytes, upstream xformers, and Triton-based Flash Attention): https://llm-tracker.info/howto/AMD-GPUs https://llm-tracker.info/howto/AMD-GPUs
- woodrowbarlow 2y agoi also have been playing with inference on the amd 7900xtx, and i agree. there are no hoops to jump through these days. just make sure to install the rocm version of torch (if using a1111 or similar, don't trust requirements.txt), as shown clearly on the pytorch homepage. obsidian is a similar story. hip is straightforward, at least on arch and ubuntu (fedora still requires some twiddling, though). i didn't realize xformers is also functional! that's good news.
- conshama 2y agorelated: https://www.nonbios.ai/post/deploying-large-405b-models-in-full-precision-on-runpod https://www.nonbios.ai/post/deploying-large-405b-models-in-f... tldr: uses the latest rocm 6.2 to run full precision inference for llama 405b on a single node 8 x MI300x AMD GPU How mature do you think Rocm 6.2-AMD stack is compared to Nvidia ?
- qamononep 2y agoIt would be great if you included a section on running with Docker on Linux. The only one that worked out of the box was Ollama, and it had an example. https://github.com/ollama/ollama/blob/main/docs/docker.md https://github.com/ollama/ollama/blob/main/docs/docker.md has a docker image but no examples to run it https://github.com/ggerganov/llama.cpp/blob/master/docs/docker.md https://github.com/ggerganov/llama.cpp/blob/master/docs/dock... has a docker image but no examples to run it https://github.com/LostRuins/koboldcpp?tab=readme-ov-file#docker https://github.com/LostRuins/koboldcpp?tab=readme-ov-file#do... docker image was broken for me on 7800xt running rhel9 https://github.com/Atinoda/text-generation-webui-docker https://github.com/Atinoda/text-generation-webui-docker
- fazkan 2y agogood feedback thanks, would you be able to open an issue
- qamononep 2y agothis repo? https://github.com/AUGMXNT/llm-tracker.info-vault/issues https://github.com/AUGMXNT/llm-tracker.info-vault/issues
- lhl 2y agoI think fazkan was confused about which repo you were talking about. For the llm-tracker doc, that's something I maintain. It's based on stuff I test but if you want to submit a PR or issue w/ info in a way that I can verify then I'm happy to add a Docker section.
- fazkan 2y agohaha, I was a bit confused, but I was referring to this one https://github.com/slashml/amd_inference https://github.com/slashml/amd_inference. But the comment applies to other repos as well, do open issues in them, helps the maintainers prioritize features.