11 ms·
Making AMD GPUs competitive for LLM inference
- tails4e 3y agoBeing 90% of a 4090 makes the 7900 xtx very attractive in a cost per compute perspective, as its about 65%, and power is significantly lower too
- nkingsy 3y ago[flagged]
- GaggiX 3y agoIt has been a while since I saw anyone send a Userbenchmark link. Userbenchmark is not a site with a good reputation, you can find many reasons online why this is, but I guess the site moderators just ignore it by pretending it's "Advanced Marketing", meanwhile even the CPU competitor of AMD has banned the site from their subreddit, one of my favorite explanation https://youtu.be/RQSBj2LKkWg https://youtu.be/RQSBj2LKkWg but there are of course more recent ones.
- CuriousCosmic 3y agoReally? you are referring to userbenchmark's assessment of AMD? Look at literally any of their assessments for AMD products and they'll be dragging them through the mud while pointing to worse performing intel products to flex how much better they are because they aren't AMD.
- zdw 3y agoThere's a reason this site is banned from most hardware forums.
- tbruckner 3y agoIsn't it closer to 80%? Where is it 90% of a 4090?
- MezzoDelCammin 3y agoit's not the 4090 that the 7900 XTX should be compared to. It's the 3090Ti. They could be had for just about the same price when You consider 2nd hand cards.
- superkuh 3y ago> AMD GPUs using ROCm Oh great. The AMD RX 580 was released in April 2018. AMD had already dropped ROCm support for it by 2021. They only supported the card for 3 years. 3 years. It's so lame it's bordering on fraudulent, even if not legally fraud. Keep this in mind when reading this news. The support won't last long, especially if you don't buy at launch. Then you'll be stuck in the dependency hell that is trying to use old drivers/stack.
- crowwork 3y agoThere is also vulkan support which should be more universal(also included in the post), for example, the post also shows running LLM on a steamdeck APU.
- junrushao1994 3y agotbh im not sure what amds plan is on ROCm support on consumer devices, but i dont really think amd is being fraudulent or something. Both rocm and vulkan are supported in MLC LLM as mentioned in our blog post. we are aware that rocm is not sufficient to cover consumer hardwares, and in this case vulkan is a nice backup!
- zorgmonkey 3y agoHow does the performance with Vulkan compare to the ROCm performance on the same hardware?
- junrushao1994 3y agoWe haven't done any comparison them yet, but generally we believe Vulkan as a more generic cross-vendor API should be slower than ROCm. Same for CUDA vs Vulkan.
- ttctciyf 3y agoIf you click the "Radeon" tab here[1], dated 27 Jul, AMD claim ROCm support on a wide range of consumer cards, with HIP SDK support on RX 6800 and up, under Windows. The Linux situation seems less clear. 1: https://rocm.docs.amd.com/en/latest/release/windows_support.html#windows-supported-gpus https://rocm.docs.amd.com/en/latest/release/windows_support....
- junrushao1994 3y agoOne of the authors here. Glad it’s on HackerNews! There are two points I personally wanted to make through this project: 1) With a sufficiently optimized software stack, AMD GPUs can be sufficiently cost-efficient to use in LLM serving; 2) ML compilation (MLC) techniques, through its underlying TVM Unity software stack, are the best fit in terms of cross-hardware generalizable performance optimizations, quickly delivering time-to-market values, etc. So far, to the best of our knowledge, MLC LLM delivers the best performance across NVIDIA and AMD GPUs in single-batch inference on quantized models, and batched/distributed inference is on the horizon too.
- tails4e 3y agoWhen you say best performance on nvidia, do you mean against any other method of running this model an nvidia card?
- junrushao1994 3y agoyeah we tried out popular solutions like exllama and llama.cpp among others that support inference of 4bit quantized models
- brucethemoose2 3y agoI can confirm this, mlc is shockingly fast on my RTX 2060. The catch is: - MLC's quantization is somewhat different (though I havent run any perplexity tests yet) - There is no CPU offloading (or splitting onto an IGP) like Llama.cpp yet (unless its new and I missed it).
- junrushao1994 3y agoTrue and there are some other issues to be addressed. Those two particular issue is on our roadmap. Regarding quantization, we wanted to develop a code path that absorbs any quantization formats, for example, those from GGML or GPTQ, so that they could be all used. ML compilation (MLC) is agnostic to any quantization formats, but we just haven't exposed such abstractions yet. On CPU offloading, imagine if you are writing PyTorch, it should be as simple as a one-liner `some_tensor.cpu()` to bring something down to host memory, and `some_tensor.cuda()` to get it back to CUDA - seems a low-hanging fruit but it's not implemented yet in MLC LLM :( Lots of stuff to do and we should make this happen soon.
- ineedasername 3y agoSomewhat as a general question, not just for AMD/Nvidia: At what point does RAM stop being the bottleneck? By which I mean, given current chips, how much RAM could you theoretically bolt on to these cards before the limiting factor in performance is the GPU instead of RAM? And does that change when the task is training vs. deployment/prompting?
- coherentpony 3y agoWhat do you mean? Are you talking about capacity? Or bandwidth from RAM? I'm in the HPC space, and pretty much everything I do on the GPU is bound by how quickly I can get data in and out of DRAM. The point at which data motion to/from DRAM is not the bottleneck is when you do enough work per byte of data moved. How much work is that? On today's server GPUs it's in the region of 50--100 double precision floating point operations per byte. You can work out an exact number by taking the theoretical maximum floating point operations per unit time you can execute and divide by DRAM throughput (data moved per unit time). O(50--100) double precision flops per byte is a _lot_ of work. We're talking BLAS-3 type operations. Anything level 2 or lower, or sparse operations, are typically bandwidth bound.
- BobbyJo 3y agoThe problem with a lot of machine learning algorithms is that you do hundreds or even thousands of operations per-value, but you do it in specialized orderings on incredibly large numbers of values, forcing you to swap partitions in and out of RAM. So, while you may do thousands of ops per-value, you may only be able to do tens of ops per-value per-write to RAM. The more RAM you have on device, the fewer swaps you need to do (none at all if it's big enough), and the less those operations get amortized, bringing you closer to theoretical max throughput. Matrix multiples are such that going from fitting 75% of your values to fitting 100% of your values can mean an order of magnitude speedup.
- coherentpony 3y agoDisclaimer: I have no idea how machine learning algorithms work. I work with problems so huge they do not fit on a single device. Multiple devices each own a small piece of the global problem. They solve a local problem and they must communicate over a network in order to solve the global problem. This is almost universally true. You would know more than I in this field, and I expect it really is better to swap a partition than it is to use a network. There are certain methods in HPC applications that are almost universally avoided because of how terribly they scale to a large distributed memory system. Matrix multiplies are one of them. Outside of a handful of ab initio computational chemistry algorithms (which are incredibly important), basically the only reason someone does a large dense matrix-multiply on a supercomputer is usually because they're running a benchmark and they're not solving a real science problem. Folks more knowledgeable than me here feel free to jump in.
- 71a54xd 3y agoI really wonder what kind of Koolaid they serve at AMD? This feels very similar to AMD shilling "multi core" CPU's back in the mid 2000's when anyone seriously using multi-core GPUs didn't care about cost and was just buying intel while AMD continued to fumble mediocre products.
- unethical_ban 3y agoTalking s** about AMD CPUs in the mid-2000s is a gutsy move.
- piskov 3y agoIs Geohot (George Hotz) part of this project or is it a parallel undertaking?
- zorex 3y agono. different project https://tinygrad.org/ https://tinygrad.org/
- cosmojg 3y agoIt's parallel. Notably, Geohot's focus has been on improving ROCm directly while this project focuses more on improving the downstream tooling.
- xiphias2 3y agoHe is doing full stack, and is selling tinybox (a PC with 6 AMD GPUs in it). As he went through the hard work of debugging the driver and getting AMD to care about it, I was expecting to see him in the acknowledgements.
- pokeypokes 3y agoDidn't he just find/report a bug in the multi-gpu case? Then have a melt-down after an AMD engineer sent him a fix a couple of days later? I think you are overstating his contributions, in fact it's not clear to me he made any.
- kanwisher 3y agoHe literally had a phone call with CEO of AMD, and got them to organize people to fix the problems. He has been whipping AMD into fixing their drivers
- pokeypokes 3y agoDo you work at AMD and therefore have some kind of insight that we don't? Based on my own experience it's much more likely the call was just a marketing exercise, completely orthogonal to actual development. Gfx drivers are complicated, long term projects. They're not something that get fixed with a phone call. He found a bug, and made a bunch of noise about it. No need to make it more than it was.
- vladgur 3y agoWhat about using 3 8GB RX 6600 in place of 1 24GB RX 7900?
- MezzoDelCammin 3y agojust out of curiosity - how would You go about that link of the 3 cards? Does AMD have anything like NVLink? (honest question, I really don't know and suspect I might have missed a bit somewhere in the idea)
- aPoCoMiLogin 3y agonvidia dropped nvlink from their consumer cards, so for the consumer nvidia and amd cards it would work the same - trough PCIE
- slinger 3y agoThrough pcie using GPU P2P with a supported AMD chipset (epyc processor) so they can talk to each other or going through your CPU and RAM. I think the latter is what people using nvidia RTX 4090s do because P2P is disabled and NVlink is unsupported
- yieldcrv 3y agoHm, this is interesting, is this enough to go long AMD?
- polyterative 3y agonah
- treprinum 3y agoWhy is 4090 so slow in your tests? It regularly demolishes my 3090 in my own tests (2x speedup in most tasks).
- junrushao1994 3y agoLLM decoding is dominated by memory bandwidth, and 3090Ti and 4090 happen to have the identical theoretical memory bandwidth
- gok 3y agoMemory bandwidth for unbatched decoder-only autoregressive inference should be the only bottleneck. The listed numbers imply you're only saturating about 500 GB/sec, ignoring KV cache traffic. Implies there's still quite a bit of inference performance left to squeeze out of the 960 GB/s spec.
- MezzoDelCammin 3y agoNice writeup. A minor nitpick though - why is the price of the 3090Ti stated as almost 2k USD? I guess the card might be hard to get new and this might indeed be the price somewhere, but since it's a previous generation, in my opinion it would make sense to use the price of a 2nd hand (probably under 1k USD, depending on place / luck). BTW - that's exactly the dillema I'm pondering now when thinking about a new PC build to play with some machine learning / LLMs. In my personal situation, 7900XTX and 3090Ti are about the same price / TDP / profile. Almost the only difference is then CUDA vs ROCm and the fact that one is new and might therefore give me some basic warranty.
- voytec 3y agoDo I understand correctly that it only applies to dedicated cards, not these GPUs bundled with Ryzens?
- yc-kraln 3y agoIt wasn't clear to me from the blog post: If I have an AMD GPU and an AMD CPU and everything with unified memory, can I use system memory for inferencing? that's a difference of "just" 16gb, or a combined 72gb available for me, and would br a big deal.
- andrewstuart 3y agoCan the GPUs that are built into CPUs be used for AI? They’re not fast but they do share memory with the CPU. The most interesting at the moment being the AMD Ryzen™ 9 7940HS Processor, 8 Cores/16 Threads.
- codedokode 3y agoThey have lots of ALUs so in theory yes.