10 ms·
AMD GPU Inference
- a2128 2y agoIt seems to use an old, 2 year old version of ROCm (5.4.2) which I'm doubtful would support my RX 7900 XTX. I personally found it easiest to just use the latest `rocm/pytorch` image and run what I need from there
- slavik81 2y agoThe RX 7900 XTX (gfx1100) was first enabled in the math libraries (e.g. rocBLAS) for ROCm 5.4, but I don't think the AI libraries (e.g. MIOpen) had it enabled until ROCm 5.5. I believe the performance improved significantly in later releases, as well.
- ashirviskas 2y agoI'm all for having more open source projects, but I do not see how it can be useful in this ecosystem, especially for people with newer AMD GPUs (not supported in this project) which are already supported in most popular projects?
- fazkan 2y agoJust something that, we found helpful, support for new architectures is just a package update. This is more of a cookie cutter
- lhl 2y agoFor inference, if you have a supported card (or probably architecture if you are on Linux and can use HSA_OVERRIDE_GFX_VERSION), then you can probably run anything with (upstream) PyTorch and transformers. Also, compiling llama.cpp is has been pretty trouble-free for me for at least a year. (If you are on Windows, there is usually a win-hip binary of llama.cpp in the project's releases or if things totally refuse to work, you can use the Vulkan build as a (less performant) fallback). Having more options can't hurt, but ROCm 5.4.2 is almost 2 years old, and things have come a long way since then, so I'm curious about this being published freshly today, in October 2024. BTW, I recently went through and updated my compatibility doc (focused on RDNA3) w/ ROCm 6.2 for those interested. A lot has changed just in the past few months (upstream bitsandbytes, upstream xformers, and Triton-based Flash Attention): https://llm-tracker.info/howto/AMD-GPUs https://llm-tracker.info/howto/AMD-GPUs
- woodrowbarlow 2y agoi also have been playing with inference on the amd 7900xtx, and i agree. there are no hoops to jump through these days. just make sure to install the rocm version of torch (if using a1111 or similar, don't trust requirements.txt), as shown clearly on the pytorch homepage. obsidian is a similar story. hip is straightforward, at least on arch and ubuntu (fedora still requires some twiddling, though). i didn't realize xformers is also functional! that's good news.
- conshama 2y agorelated: https://www.nonbios.ai/post/deploying-large-405b-models-in-full-precision-on-runpod https://www.nonbios.ai/post/deploying-large-405b-models-in-f... tldr: uses the latest rocm 6.2 to run full precision inference for llama 405b on a single node 8 x MI300x AMD GPU How mature do you think Rocm 6.2-AMD stack is compared to Nvidia ?
- qamononep 2y agoIt would be great if you included a section on running with Docker on Linux. The only one that worked out of the box was Ollama, and it had an example. https://github.com/ollama/ollama/blob/main/docs/docker.md https://github.com/ollama/ollama/blob/main/docs/docker.md has a docker image but no examples to run it https://github.com/ggerganov/llama.cpp/blob/master/docs/docker.md https://github.com/ggerganov/llama.cpp/blob/master/docs/dock... has a docker image but no examples to run it https://github.com/LostRuins/koboldcpp?tab=readme-ov-file#docker https://github.com/LostRuins/koboldcpp?tab=readme-ov-file#do... docker image was broken for me on 7800xt running rhel9 https://github.com/Atinoda/text-generation-webui-docker https://github.com/Atinoda/text-generation-webui-docker
- leonheld 2y agoPeople use "Docker-based" all the time but what they mean is that they ship $SOFTWARE in a Docker image. "Docker-based" reads, to me, as if you were doing Inference on AMD cards with Docker somehow, which doesn't make sense.
- a_vanderbilt 2y agoYou can do inference from a Docker container, just as you'd do it with NVidia. OpenAI runs a K8s cluster doing this. I have personally only worked with NVidia, but the docs are present for AMD too. Like anything AI and AMD, you need the right card(s) and rocm version along with sheer dumb luck to get it working. AMD has Docker images with rocm support, so you could merge your app in with that as the base layer. Just pass through the GPU to the container and you should get it working. It might just be the software in a Docker image, but it removes a variable I would otherwise have to worry about during deployment. It literally is inference on AMD with Docker, if that's what you meant.
- mikepurvis 2y agoDocker became part of the standard toolkit for ML because deploying Python that links to underlying system libraries is a gong show unless you ship that layer too.
- tannhaeuser 2y agoEven Docker doesn't guarantee reproducible results due to sensitivity towards host GPU drivers, and ML frontends/integrations bringing their own "helpful" newby-friendly all-in-one dependency checks and updater services.
- jeffhuys 2y agoWhy doesn’t it make sense? You can talk to devices from a Docker container - you just have to attach it.
- fazkan 2y agoyou can mount a specific device to docker. If you read the script, we are mounting GPUs https://github.com/slashml/amd_inference/blob/main/run-docker-amd.sh https://github.com/slashml/amd_inference/blob/main/run-docke...
- khurdula 2y agoAre we supposed to use AMD GPUs for this to work? Or Does it work on any GPU?
- karamanolev 2y ago> This project provides a Docker-based inference engine for running Large Language Models (LLMs) on AMD GPUs. First sentence of the README in the repo. Was it somehow unclear?
- tomxor 2y agoI almost tried to install AMD rocm a while ago after discovering the simplicity of llamafile. sudo apt install rocm Summary: Upgrading: 0, Installing: 203, Removing: 0, Not Upgrading: 0 Download size: 2,369 MB / 2,371 MB Space needed: 35.7 GB / 822 GB available I don't understand how 36 GB can be justified for what amounts to a GPU driver.
- greenavocado 2y agoIt's not just you; AMD manages to completely shit-up the Linux kernel with their drivers: https://www.phoronix.com/news/AMD-5-Million-Lines https://www.phoronix.com/news/AMD-5-Million-Lines
- anthk 2y agoOpenBSD, too.
- striking 2y ago> Of course, much of that is auto-generated header files... A large portion of it with AMD continuing to introduce new auto-generated header files with each new generation/version of a given block. These verbose header files has been AMD's alternative to creating exhaustive public documentation on their GPUs that they were once known for.
- NekkoDroid 2y agoThere have been talks about moving those headers to a separate repo and only including the needed headers upstream[1] [1]: https://gitlab.freedesktop.org/drm/amd/-/issues/3636 https://gitlab.freedesktop.org/drm/amd/-/issues/3636
- atq2119 2y agoSo no doubt modern software is ridiculously bloated, but ROCm isn't just a GPU driver. It includes all sorts of tools and libraries as well. By comparison, if you go and download the CUDA toolkit as a single file, you get a download file that's over 4GB, so quite a bit larger than the download size you quoted. I haven't checked how much that expands to (it seems the ROCm install has a lot of redundancy given how well it compresses), but the point is, you get something that seems insanely large either way.
- sandGorgon 2y agois anyone using the new HX370 based laptops for any LLM work ? i mean the ipex-llm libraries of Intel's new Lunar Lake is already supporting Llama 3.2 (https://www.intel.com/content/www/us/en/developer/articles/technical/intel-ai-solutions-support-the-new-llama-3-2-model.html https://www.intel.com/content/www/us/en/developer/articles/t...), but AMD's new Zen5 chips dont seem to be much active here.
- dhruvdh 2y agoWhy would you use this over vLLM?
- fazkan 2y agowe have vllm in certin production instances, it is a pain for most non-nvidia related architectures. A bit of digging around and we realized that most of it is just a wrapper on top of pytorch function calls. If we can do away with batch processing with vllm supports, we can be good, this is what we did here.
- dhruvdh 2y agoBatching is how you get ~350 tokens/sec on Qwen 14b on vLLM (7900XTX). By running 15 requests at once. Also, there is a Dockerfile.rocm at the root of vLLM's repo. How is it a pain?
- fazkan 2y agodriver mismatch issues, we mostly use publicly available instances, so the drivers change as the instances change, according to their base image. Not saying it won't work, but it was more painful to figure out vllm, than to write a simple inference script and do it ourselves.
- white_waluigi 2y agoIsn't this just a wrapper for huggingface-transformers?
- fazkan 2y agoyes, but handles all the dependencies for AMD architecture. So technically its just a requirements file :). Author of the repo above.
- freeqaz 2y agoWhat's the best bang-for-your-buck AMD GPU these days? I just bought 2 used 3090s for $750ish refurb'd on eBay. Curious what others are using for running LLMs locally.
- 3abiton 2y agoPersonal experience: It's not even worth it. AMD (i)GPU breaks with every pytorch, ROCm, xformers, or ollama updates. You'll sleep more compfortably at night.
- fazkan 2y agoThats our observation, which is why we wrote the scripts ourselves that way we can control the dependencies at least.
- sangnoir 2y agoWhen dealing with ROCM, it's critical that once you have a working configuration, you freeze everything in place (except your application). Docker is one way to achieve this if your host machine is subject to kernel or package updates
- horsebridge 2y agoI don't really have any problem with ROCm these days, although I only use system packages. It used to be quite wonky though, and I've totally given up on custom ROCm installs.
- FeepingCreature 2y agoSame here on my 7900 XTX. Used to be terrible, now it's (seemingly) fine (for now).
- wkat4242 2y agoIt's not the experience I have. I've been using ollama for 6 months on mine and never had any issues with ROCm breaking.
- stefan_ 2y agoThis seems to be some AI generated wrapper around a wrapper of a wrapper. > # Other AMD-specific optimizations can be added here > # For example, you might want to set specific flags or use AMD-optimized libraries What are we doing here, then?
- fazkan 2y agoits just a big requirements file, and a dockerfile :) the rest are mostly helper scripts.
- deleted 2y ago[deleted]
- phkahler 2y agoDoes it work with an APU? I just put 64GB in my system and gonna drop in a 5700G. Will that be enough? SFF inference if so.
- Const-me 2y agoThe integrated GPU of the 5700G uses old architecture from 2017, this one: https://en.wikipedia.org/wiki/Radeon_RX_Vega_series https://en.wikipedia.org/wiki/Radeon_RX_Vega_series Pretty sure it does not support ROCm. BTW if you just want to play with a local LLM, you can try my old port of Mistral: https://github.com/Const-me/Cgml/tree/master/Mistral/MistralChat https://github.com/Const-me/Cgml/tree/master/Mistral/Mistral... Unlike CUDA or ROCm my port is based on Direct3D 11 GPU API, runs on all GPUs regardless of the brand.
- fazkan 2y ago@Const-me according to this it should work, https://github.com/ROCm/ROCm/issues/2216 https://github.com/ROCm/ROCm/issues/2216
- fazkan 2y agohaven't tested it, but it should according to https://github.com/ROCm/ROCm/issues/2216 https://github.com/ROCm/ROCm/issues/2216 You just need to update the version check here https://github.com/slashml/amd_inference/blob/4b9ec069c4b2accc764eaa677eb43e2d1d5ff1f3/src/amd_setup.py#L11 https://github.com/slashml/amd_inference/blob/4b9ec069c4b2ac... feel free to open an issue, with the requirements and we will test it.
- Chordless 2y agoI'm able to run Ollama and llama.cpp on my Ryzen 4600G APU following this guide: https://agieverywhere.com/apuguide/AMDAPU/APU_Linux https://agieverywhere.com/apuguide/AMDAPU/APU_Linux Your APU should be similar, just faster. There are some magic environment variables you want to set to get ROCM to work with this technically unsupported APU: HSA_OVERRIDE_GFX_VERSION=9.0.0 HSA_ENABLE_SDMA=0 Performance is not great, but slightly better than running inference on the CPU, with the bonus that your CPU is essentially free for other tasks even while running LLMs.
- BaculumMeumEst 2y agohow about they follow up 7900 XTX with a card that actually has some VRAM
- deleted 2y ago[deleted]
- tcdent 2y agoThe rise of generated slop ml libraries is staggering. This library is 50% print statements. And where it does branch, it doesn't even need to. Defines two environment variables and sets two flags on torch.
- mdaniel 2y agoI also had to go to therapy to cure myself of the misunderstanding that data scientists and machine learning folks are software engineers, and expecting the same work product from those disparate audiences only raises your blood pressure Expectation management is a huge part of any team/organization, I think
- tpoacher 2y agoThey can be the same or different, given how you define them. People throw these words around with little thought, especially ones superficial to or outside the field. I wouldn't disparage an entire field for lack of a clear definition in the buzzwords people use to refer to it.
- tcdent 2y agoAs suggested by my original comment, I read the source of a lot of the open source projects around ML. mdaniel is absolutely correct. They are not software engineers. I'm not vocal about the naïve stuff, poor design, sloppy formatting, bad english. I am vocal about projects that have no place in the ecosystem.
- driverdan 2y agoI thought you were being overly harsh until I looked at the repo. You're not kidding, there's very little to it.
- TechDebtDevin 2y agoWhile I see where you are coming from, these are the types of comments that keep people from sharing their code, contributing to OSS or continuing to program in general.
- slavik81 2y agoOn Ubuntu 24.04 (and Debian Unstable¹), the OS-provided packages should be able to get llama.cpp running on ROCm on just about any discrete AMD GPU from Vega onwards²³⁴. No docker or HSA_OVERRIDE_GFX_VERSION required. The performance might not be ideal in every case⁵, but I've tested a wide variety of cards: # install dependencies sudo apt -y update sudo apt -y upgrade sudo apt -y install git wget hipcc libhipblas-dev librocblas-dev cmake build-essential # ensure you have permissions by adding yourself to the video and render groups sudo usermod -aG video,render $USER # log out and then log back in to apply the group changes # you can run `rocminfo` and look for your GPU in the output to check everything is working thus far # download a model, build llama.cpp, and run it wget https://huggingface.co/TheBloke/dolphin-2.2.1-mistral-7B-GGUF/resolve/main/dolphin-2.2.1-mistral-7b.Q5_K_M.gguf?download=true -O dolphin-2.2.1-mistral-7b.Q5_K_M.gguf git clone https://github.com/ggerganov/llama.cpp.git cd llama.cpp git checkout b3267 HIPCXX=clang-17 cmake -H. -Bbuild -DGGML_HIPBLAS=ON -DCMAKE_HIP_ARCHITECTURES="gfx803;gfx900;gfx906;gfx908;gfx90a;gfx1010;gfx1030;gfx1100;gfx1101;gfx1102" -DCMAKE_BUILD_TYPE=Release make -j16 -C build build/bin/llama-cli -ngl 32 --color -c 2048 --temp 0.7 --repeat_penalty 1.1 -n -1 -m ../dolphin-2.2.1-mistral-7b.Q5_K_M.gguf --prompt "Once upon a time" I'd suggest RDNA 3, MI200 and MI300 users should probably use the AMD-provided ROCm packages for improved performance. Users that need PyTorch should also use the AMD-provided ROCm packages, as PyTorch has some dependencies that are not available from the system packages. Still, you can't beat the ease of installation or the compatibility with older hardware provided by the OS packages. ¹ https://lists.debian.org/debian-ai/2024/07/msg00002.html https://lists.debian.org/debian-ai/2024/07/msg00002.html ² Not including MI300 because that released too close to the Ubuntu 24.04 launch. ³ Pre-Vega architectures might work, but have known bugs for some applications. ⁴ Vega and RDNA 2 APUs might work with Linux 6.10+ installed. I'm in the process of testing that. ⁵ The version of rocBLAS that comes with Ubuntu 24.04 is a bit old and therefore lacks some optimizations for RDNA 3. It's also missing some MI200 optimizations.
- mindcrime 2y agoI was able to install (AMD provided) ROCm and Ollama on Ubuntu 22.04.5 with an RX 7900 XTX with no real problems to speak of, and I can execute LLMs using Ollama on ROCm just fine. Take that FWIW.
- kn100 2y agoSad that RDNA2 cards aren't supported. Not even that old!
- rglullis 2y agoSo, this is all I needed to add to NixOS workstation: hardware.graphics.enable = true; services.ollama = { enable = true; acceleration = "rocm"; environmentVariables = { ROC_ENABLE_PRE_VEGA = "1"; HSA_OVERRIDE_GFX_VERSION = "11.0.0"; }; };
- ekianjo 2y agoDoes it work with GGUF files?
- kristianp 2y agoIf you're interested in how much the AMD graphics cards cost compared to the NVidia ones, I have https://gpuquicklist.com/ https://gpuquicklist.com/ which gives you a quick table view of lowest prices available on Amazon that I can find. </ selfpromotion>
- danielEM 2y agoIt has been like 8 months since I got Ryzen 8700G with NPU just for the purpose of inferencing NN, and so far only acceleration I'm getting is through vulkan on iGPU, not NPU (I'm using Linux only). On the bright side, with 64GB of RAM had no isues with trying models over 32GB. Kudos to llama.cpp for supporting vulkan backend!
- lhl 2y agoYou should have ROCm/HIP support on the iGPU as well, be sure to compile llama.cpp w/ the LLAMA_HIP_UMA=1 flag. If you take a look at https://github.com/amd/RyzenAI-SW https://github.com/amd/RyzenAI-SW you can see there's a fair amount of software to play with on the NPU now, but Phoenix is only 16 TOPS, so I've never bothered testing it.
- terminalcommand 2y agoWill this work on older cards such as RX 570? Does anyone know?
- talles 2y agoAnyone else with an Intel Arc card idle waiting for some support?
- lenova 2y agoWhy ROCm 5.4, and not the latest (6.2)? https://github.com/slashml/amd_inference/blob/main/Dockerfile#L2 https://github.com/slashml/amd_inference/blob/main/Dockerfil... Also looks like the Docker image provided by this project doesn't successfully build: https://github.com/slashml/amd_inference/issues/2 https://github.com/slashml/amd_inference/issues/2
- fazkan 2y agotested with the old Rocm, created a PR with the latest.