6 ms·
Whilst this is an excellent post from vLLM, one of the truly baffling things from either their team or AMDs team, is how much the workstation grade AMD r9700 ha
by intothemild 25d ago
Whilst this is an excellent post from vLLM, one of the truly baffling things from either their team or AMDs team, is how much the workstation grade AMD r9700 has been ignored.
Stock vLLM runs so slowly on these cards compared with vLLM forks like Radiance. Going from say 20-30t/s gen, to 150-200t/s
Most of AMD/vLLM work seems to be around their data centre cards, or the AMD AI Halo/Ryzen and ignores the R9700 AI Pro.
Really wish this would change.
- minraws 25d agoI don't see a reason why it should AMD doesn't care about lower end prosumers atm. They might in the future but future is in the future ofc Edit: to be clear I think it's ridiculous they don't but from a company's stand point it doesn't make much sense
- websap 25d agoYeah, as a business AMD should first care about getting their DC grade hardware optimized for inference workloads. It's unfortunate that most of HN discussion has devolved to me-ish.
- _factor 25d agoThen they should stop selling hardware they don’t plan to support. Me-ish when you spend $1,500 on a piece of hardware is completely acceptable.
- brookst 25d agoNever buy hardware based on expectations of future features, especially if there’s no promise from the vendor.
- mrhenio 25d ago[flagged]
- dist-epoch 25d agoGeorge Hotz in June 2023: > I have had direct contact with members of the AMD RTG team and I was disgusted to find that AMD doesn't even provide them with hardware to work on. The developer I was working with had to buy the GPU he was writing drivers for.
- da-x 25d agoI think this has changed since then, their policies toward open source improved (e.g ROCm).
- dist-epoch 25d agoThe market says the problem is still there. An NVIDIA consumer GPU sells for 50+% or more than an equivalent AMD GPU. Because people are buying NVIDIA GPUs to run local models instead of AMD ones. I did the same thing, I paid 50% more to get an 5070 Ti instead of the equivalent AMD. This is probably good for gamers, AMD GPUs are not price inflating to the same degree as NVIDIA, because they are bad at LLMs. > That was the reason for comparing them in the first place: based on performance, they are direct competitors, or at least they are meant to be. However, as things stand today, there is a massive price divide between the two, with the RTX Ti GPU now commanding a premium of more than 50%. https://www.techspot.com/review/3168-geforce-rtx-5070-vs-radeon-rx-9070-xt/ https://www.techspot.com/review/3168-geforce-rtx-5070-vs-rad...
- androiddrew 25d agoThankfully, there are still people willing to jump on the R9700 bandwagon and get a vLLM fork working. If you have an RDNA4 card check out https://hub.docker.com/r/stilldeadcode/vllm-radiance https://hub.docker.com/r/stilldeadcode/vllm-radiance
- intothemild 25d agoDeadcode is currently working on INT4 right now on his R4D kernel. The MXFP4 fork is excellent too. Its my daily driver right now. https://codeberg.org/ggz14/radiance-vllm-mxfp4 https://codeberg.org/ggz14/radiance-vllm-mxfp4 Also has PARO quant support there too (early stage) Also speedups in both repos for 4x R9700s
- karmakaze 25d agoThanks! Didn't expect to see this here. Exactly what I needed to run Qwen3.8-27B-Quark-AWQ-MXFP4-native.gguf as well as other experiments on one or 2x R9700's (I hope).
- nicce 25d agoThat MXFP4 is an excellent project. But I have difficulties on reading that README. Is it intentionally generated like that with LLMs?
- intothemild 25d agoFeed the setup and run scripts to your LLM.
- karmakaze 25d agoThe way it does tensor splitting without all-reduce cost over PCIe bus wasn't something I thought was possible. What kind of performance are you getting with 4x R9700s--what do you do with all the VRAM (batching, concurrent requests, etc)?
- intothemild 25d ago
- roenxi 25d ago> ...one of the truly baffling things from either their team or AMDs team, is how much the workstation grade AMD r9700 has been ignored. It makes a huge amount of sense after considering AMD's approach to graphics cards from around 2010 to 2025. They just didn't see graphics cards as viable compute platform and many who made the mistake of believing that good specs would translate into in-practice performance got badly burned. I'd have been involved in the AI boom but for an expensive AMD graphics card, I'm not going to forget that for a while. George Hotz was interesting as a public example, but I think his story probably repeated a few times outside the public eye. People tried to make AMD work and ended up the worse for it. People who had an interest in using AMD cards to get things done are probably by and large waiting for a new generation of hopefuls to prove this time is different. The mutterings out of AMD are promising, but that isn't persuasive enough given the scale of the failures.
- hgoel 25d agoGCN was such a promising compute architecture, AMD even pioneered stuff like async compute and compute shader heavy rendering pipelines, only to never seriously go beyond that on consumer gear. I agree with your assessment that the story of supporting the competition, only to get burned, has repeated many times with AMD outside the public eye. It's why I don't put much stock in claims that things work great as long as specific flags are used.
- lrvick 25d agoI have 4x r9700s as my coding daily drivers running qwen 3.8 27b at 80tps each. Zero complaints. Especially at $1200 each.
- SomeHacker44 25d agoPlease share what operating system and model runtime you use? I have two and don't get close to that with AMD's own Lemonade. Thanks!
- lrvick 24d agoLemonade is one of the worst performing options. Run any modern Linux distro and ask your current LLM to setup llama.cpp with dflash2 for you as an unprivileged container running from a systemd user unit. Obviously only on a system you do not trust at all.
- Roark66 25d agoI'd rather buy two used rtx3090 than a single r9700 AI pro. More VRAM (some wasted due to it being non continuous), more RAM bandwidth, more aggregate compute. Only if AMD made a card like this with 48G+ I'd consider it. Also these 20-30t/s jumping to 150-200... Watch out for the massaged numbers coming from vendors. I believe Intel has claimed something like 1400tok/s (generation! Not prefill) of Qwen3.6-moe on Arc b70. I was actually very interested in this so I checked the details. Turns out it was 200 simultaneous users running the same 1024 token prompt :D so all the experts got maximum parallelism. How often are you going to run 200 parallel sessions with a tiny context and same prompt running at 7tok/s. Based on how much my rtx3090 is getting on a single user (150tok/s) I'm estimating b70 to probably get less than that. Sadly nvidia is king now. Also, most of us already have nvidia cards and no inference software supports mixing let's say nvidia, Intel and amd cards in inference of one model.
- nicce 25d ago2x RTX 3090 is not enough for proper use. ideal is 64GB+ so that you get proper cache and 256k context with good enough models. (e.g. Qwen 3.8 27B with MXFP4). And that is tight already. You would need 3x RTX 3090 - but then the PCIe bandwidth comes an issue if you really want tensor parallelism with three cards. 3x PCIe5 x16 is not cheap with direct CPU access. So surprisingly, 2x r9700 starts be a nice deal. > Also these 20-30t/s jumping to 150-200... Watch out for the massaged numbers coming from vendors. Well, luckily these are not vendor numbers. Prefill also scales almost linearly with the amount of GPUs.
- formerly_proven 25d agoYou don't need PCIe 5.0 x16 since RTX 30 are not PCIe 5.0 to begin with.
- nicce 25d agoWell, that makes them just even slower then
- 25d ago
- 3abiton 24d agoHonestly a big part of this is AMD's lackluster strategy to GPU software. To say it's lacking is understatement. At least for non-data center gpus.