4 ms·
What's the best bang-for-your-buck AMD GPU these days? I just bought 2 used 3090s for $750ish refurb'd on eBay. Curious what others are using for running LLMs l
by freeqaz 2y ago
What's the best bang-for-your-buck AMD GPU these days? I just bought 2 used 3090s for $750ish refurb'd on eBay. Curious what others are using for running LLMs locally.
- 3abiton 2y agoPersonal experience: It's not even worth it. AMD (i)GPU breaks with every pytorch, ROCm, xformers, or ollama updates. You'll sleep more compfortably at night.
- fazkan 2y agoThats our observation, which is why we wrote the scripts ourselves that way we can control the dependencies at least.
- sangnoir 2y agoWhen dealing with ROCM, it's critical that once you have a working configuration, you freeze everything in place (except your application). Docker is one way to achieve this if your host machine is subject to kernel or package updates
- horsebridge 2y agoI don't really have any problem with ROCm these days, although I only use system packages. It used to be quite wonky though, and I've totally given up on custom ROCm installs.
- FeepingCreature 2y agoSame here on my 7900 XTX. Used to be terrible, now it's (seemingly) fine (for now).
- wkat4242 2y agoIt's not the experience I have. I've been using ollama for 6 months on mine and never had any issues with ROCm breaking.
- elorant 2y agoProbably the 7900xtx. $1k for 24GB of RAM.
- freeqaz 2y agoThat's about the same price as a 3090 and it's also 24GB. Are they faster at inference?
- elorant 2y agoI doubt it, but the 3090 is a four year old card which means it might have a lot of mileage from the previous owner. A lot of them are from mining rigs.
- sipjca 2y agoit is not, at least in llama.cpp/llamafile https://benchmarks.andromeda.computer/compare https://benchmarks.andromeda.computer/compare
- JudasGoat 2y agoAccording to that benchmark a 7900xtx is equal to a 2080ti in performance.
- wkat4242 2y agoI got my radeon pro vii for €300 new. Was not a bad deal IMO especially since it comes with HBM2 and has the same memory bandwidth as the 4090 (1TB/s). It's got only 16GB though.
- snep 2y agoI bought an MI100 recently for $650. 32GB of HBM2 and it performs around around 0-5% faster than a 3090 on the default flash attention 2 benchmarks. Performance on actual applications can be mixed though, as many are not well optimised for CDNA's matrix cores - even where work has been done for RDNA, which is not that often, it doesn't necessarily carry over. It's also frustrating when efforts to improve performance get turned back by maintainers: llama.cpp closing PR for flash attention on AMD because the requisite (header-only) lib is supposedly adding an unneeded dependency (https://github.com/ggerganov/llama.cpp/pull/7011 https://github.com/ggerganov/llama.cpp/pull/7011). There's also a few tricks/updates I'd like to try which may improve performance, e.g. hipblaslt support being added next rocm release - of course these are "maybes". To give you a rough idea of practical performance, default SDXL with xformers is around 4.5-5it/s (between 3090 and 4090 from my understanding), and exllamav2 with qwen 72B at 3bpw is around 7t/s (slower than a 3090, though a 3090 has to use a lower precision to fit). As others have pointed out, I can't really see what this project offers for AMD users over existing options like llama.cpp, exllamav2, mlc-ai, etc. Most projects work relatively easily these days.