3 ms·
Ask HN: Replacing Nvidia GPU with AMD
Dear community
I am currently exploring the feasibility of replacing Nvidia GPUs with AMD GPUs for LLM inference. Can anyone share their experience or point me to any relevant research on the performance differences between these GPUs for this task? Are there any particular hardware or software limitations that may affect the feasibility of such a switch? Thank you for your insights!
- InitEnabler 3y agoI'd take a look ROCm docs: https://www.amd.com/en/developer/resources/rocm-hub.html https://www.amd.com/en/developer/resources/rocm-hub.html
- textcortex 3y agoI heard lot of bad things about ROCm, hope things improved since then.
- gvd 3y agoTGI has ROCm support.
- Havoc 3y agoMy understanding is inference on latest amds (79*) is mostly fine but training is still shaky
- LorenDB 3y agoOllama just had AMD support merged. I haven't got it working with my 6700 XT eGPU yet, but I anticipate getting there soon.
- textcortex 3y agoAwesome! Looking into it. Thanks
- cjbprime 3y agoSince the amount of VRAM is low (max 24GB), one question to look into before investing might be whether there's support for chaining multiple cards together.
- textcortex 3y agoI would be interested in running inference on Instinct GPUs(MI250 with 128gb)BUT I can’t find any cloud provider to spin up a machine. It seems they are not yet available or cloud providers are not interested in supporting AMD hardware..
- htrp 3y agoYou'll be fine for inference but probably struggle to run any large models requiring multi gpu ops.
- brudgers 3y agoIs there a clear business rationale? Can that rationale be mitigated by increasing revenue as an alternative to adding the technical risks implicit in your question? I mean it is one thing if this is a hobby project and using AMD will provide interesting challenges to keep yourself occupied. How you spend your own time and money is entirely up to you. But spending other people’s money is another thing entirely (unless they are mom and dad). And even more so spending other people’s time particularly when it comes to paychecks. Finally, I am not saying there aren’t business cases where AMD doesn’t make sense. Just that Nvidia is the default for good reasons and is the simplest thing that might work. Good luck.