8 ms·
$95 AMD CPU Becomes 16GB GPU to Run AI Software
- brucethemoose2 3y ago> one minute and 50 seconds to generate a 512 x 512-pixel image with the default setting of 50 steps 2s/iter? That is... Not great. It is slower than CPU diffusion: https://github.com/rupeshs/fastsdcpu https://github.com/rupeshs/fastsdcpu Stable Diffusion in particular doesn't need much VRAM anyway. I get that many people are stuck on lower end computers, but ~4GB is not a totally unreasonable requirement.
- samsartor 3y agoLooks more like 4 s/iter which is actually quite good for CPU inference in Automatic111. It is an apples-to-oranges comparison to the LCM models used by fastsdcpu (since those can get away with fewer sampling steps) or to the quantized models favored by apple (since they use totally different underlying compute capabilities). Currently automatic111 and other popular GUIs are pretty focused on users with beefy GPUs that can sample images in seconds, so they haven't yet implemented some of these really significant CPU-friendly speedups. But we'll get there!
- mosselman 3y agoAre those cpu speedups out there? Where can I find them?
- brucethemoose2 3y agoNot for A1111's SAI backend, as far as I know. VoltaML has some CPU optimized options for the torch.compile optimization.
- KeplerBoy 3y agoThat's probably exactly what you would expect from a GPU running off DDR4 instead off GDDR6X.
- bongobingo1 3y ago> Stable Diffusion in particular doesn't need much VRAM anyway. I get that many people are stuck on lower end computers, but ~4GB is not a totally unreasonable requirement Wow, did it really get that low? I played with SD during the initial days, and I know there was some strides then to get it running on 8gb cards and maybe squeaking through on 6gb with a smaller image size (like, 256x256px). Can you generate reasonably large images on 4gb now, with just a higher wait time? I was never sure why it wasn't simple to just shuffle bits in and out of main RAM but I guess the model themselves are mostly big binary blobs that are hard to split up?
- deleted 3y ago[deleted]
- jandrese 3y agoYeah, I’m not sure where that figure came from either. My experience on an 8GB card is that it is always nudging up close to the limit and you don’t want to try to turn the knobs up much past the default. SDXL is a total nonstarter as well on 8GB.
- ColonelPhantom 3y agoUsing vladmandic sd.next with v1-5 works fine* on my RX 580 4GB at around 1s/it on ROCm and 1.5s/it on DirectML. fine = requires --medvram or --lowvram in order to not run out of memory. The speeds above are with medvram, which is just about on the edge of what's possible. These options compromise speed but are still better than running on the CPU generally.
- brucethemoose2 3y agoVoltaML is also quite good on low vram cards.
- ColonelPhantom 3y agoDoes VoltaML have any support for AMD/Intel GPUs or CPU inference? e.g. I think Intel's openVINO supports ONNX, but the Github page only mentions Nvidia and CUDA.
- mrtksn 3y agoIt's also significantly slower than Apple M1, which is 2.2 iteration per second in iPad. Though, maybe it's not a fair comparison since Apple optimises heavily their library. [0] https://github.com/apple/ml-stable-diffusion https://github.com/apple/ml-stable-diffusion
- Zpalmtree 3y agoSDXL needs a lot more VRAM however, especially when training LoRAs
- lithiumii 3y agoIf you switch to the test-fp8 branch of AUTOMATIC1111 web UI, you can do 512 * 1024 on 4G GPU with SDXL.
- pella 3y agorelated: (74 days ago ) "AMD Ryzen APU turned into a 16GB VRAM GPU and it can run Stable Diffusion" https://news.ycombinator.com/item?id=37162762 https://news.ycombinator.com/item?id=37162762
- iAkashPaul 3y agoAt this point the M3 Max in a MacBook Pro with 96/128GB V/RAM needs to be benched against various Nvidia cards like 4090 & the H100 for $/performance numbers
- NavinF 3y agoWhy the H100? Comparing against similarly priced consumer gear (3x 4090) makes a lot more sense, but I think you already know that one of these systems has an order of magnitude better perf/$ than the other
- iAkashPaul 3y agoYeah even that as well
- NavinF 3y agoNot sure if you missed my point. 3x 4090 will be 10x faster while costing the same. There's no point making the comparison. (Assuming both machines are running the same model. I see a lotta crazy numbers thrown around based on comparing very different quantizations)
- iAkashPaul 3y agoINT4 or FP4 ones are of interest to me
- ianpurton 3y agoRunning LLama 2 70b a Mac Studio gets 13 tokens per second. 2 x 4090 is more like 38 tokens per second. But it's harder than that. As the 2 x 4090 is using https://llm.mlc.ai/ https://llm.mlc.ai/ which is highly optimized so perhaps the Mac Studio inference could be better when tuned. Then a lot of production inference engines use continous batching, which means they perform really well when processing multiple prompts at a time.
- 3y ago
- anaisbetts 3y agoThis is similar to the Apple GPU trick where unified memory means it's easier to load large models into GPU space, but as others have said, the AMD APU built into these chips is _really_ underpowered, even compared to other RDNA APUs like the one in the Steam Deck
- MindSpunk 3y agoMinor nitpick but the 4600G and 5600G mentioned in the article are Vega, not RDNA. The iGPUs in those two processors are/were quite powerful by iGPU standards at the time, especially considering they were competing with Intel HD graphics or very early Iris Xe iGPUs. Not a scratch on a good dedicated GPU though. Steam Deck is RDNA2 though, and is significantly faster. I don't think RDNA1 ever made it into an APU.
- ianpurton 3y agoI've just finished writing an article on hardware required for LLM's in production https://bionic-gpt.com/blog/llm-hardware/ https://bionic-gpt.com/blog/llm-hardware/ Would be interesting to see how this performed running Llama 2 7b.
- lhl 3y agoLast month I did testing on the latest gen AMD APU (7940HS @ 65W - 8C Zen4 CPU, Radeon 780M RDNA3 GPU) on how llama2-7b (4-bit quants) performed w/ a variety of inferencers (llama.cpp, exllama v2, mlc w/ CPU, CLBlast, ROCm, and Vulkan backends) and only llama.cpp's ROCm implementation convincingly beating CPU inferencing. Even then, interactive (batch=1) inferencing was still barely faster. I suspect memory bandwidth (I ran w/ 2 x DDR5-5600) is the main bottleneck. Just as a point of reference, the inferencing performance is roughly in the ballpark of (but slightly lower than) my 16GB M2 MBA. Details: https://docs.google.com/spreadsheets/d/1kT4or6b0Fedd-W_jMwYpb63e1ZR3aePczz3zlbJW-Y4/edit#gid=1041125589 https://docs.google.com/spreadsheets/d/1kT4or6b0Fedd-W_jMwYp...
- atq2119 3y agoYes. Without batching, every weight is only used once per token. On discrete GPUs, you generally need around ~20 flops/byte of memory bandwidth to be compute bound. I don't know where APUs are in that regard, but probably not entirely different. Memory bandwidth divided by model size is more or less the token rate you should expect to get from a good LLM inference implementation.
- lhl 3y ago2 channels of DDR5-5600 = 89.6GB/s theoretical max bandwidth and a Q4_0 quant takes about 6.7GB of VRAM used, which is 13.4X, so that maps out pretty well for inferencing speed (I think the divided by parameters maybe only works for 4bit in this case). Interestingly, the ROCm implementation does go a bit faster (>15 t/s so it must have some tricks up its sleeve). The Radeon 780M does 8.9 TFLOPS of FP32, 17.8 TFLOPS of FP16, so if 20 flops/byte applies, it has roughly 10X more compute than memory bandwidth for inferencing.
- dale_glass 3y agoSo I'm a bit out my field here but I've been wondering... Is there something that prevents AI software from augmenting VRAM with system RAM? My impression was that for instance OpenGL doesn't even have a standard way to get VRAM usage because you're not supposed to care, stuff is just shuffled around in the background for you if VRAM runs out.
- Astronaut3315 3y agoMy SDXL workflow was often spilling over my RTX 3080’s 10GB of VRAM into system RAM. I’ve since tweaked my workflow to prevent that, as the slowdowns are dramatic. Tiled VAE encode/decode was a big help.
- zirgs 3y agoThe latest nvidia drivers do this automatically, but it's very slow.
- osmarks 3y agoIt's not very useful because the point of VRAM is that it's very very high-bandwidth. CPU RAM is slow and the PCIe link to the GPU is even slower.
- dale_glass 3y agoWhat's the usage pattern like? Is all of the VRAM used extensively in AI workloads, or one could hope to augment things a bit with system RAM with little performance impact?
- Zetobal 3y agoThe performance impact is not little it's huge.
- filterfiber 3y agoThe moment you can't fit the model in VRAM and have to use PCIE the performance impact is MASSIVE. Depending on the type of model/usecase/how much you can fit in VRAM it very well may still be usable, however compared to fitting everything in vram it'll be abysmal for most any model type. From my understanding the entire model must be read per iteration (or very close to all of it. This is why offloading any part of it impacts the performance so much. Diffusion models depending on the sampler and some other stuff can potentially get away with only a few steps (usually 2-20 best case) and are relatively small (3B or less). However LLM's are generally much bigger (they mostly start at 7B, there's a few exceptions depending on the use case), and they need an entire iteration per token which is only a few characters (think 2 to 3). So llm's especially are far more useful if you can keep them in vram.
- spacecadet 3y agolol "non-serious tasks" what are those?
- qayxc 3y agoHm. Every task that isn't directly or indirectly going into a production system? Any task where reliability, performance (including performance per $) and quality don't matter? This includes hobby projects, tinkering out of curiosity, tasks performed for learning purposes (not necessarily educational) and more.
- spacecadet 3y agoLook, my hobby is edge ML projects using SOC hardware. They are hacky AF but still more serious than some hack CPU approach like this. If anything, the next malware is turning unknowing teenager gamers rigs into piles of slag thanks to this useful discovery. I guess this article just rubs my net negative lifestyle and feels like its content for contents sake.
- CyberDildonics 3y agoI read this twice and have no idea what you're trying to say.
- spacecadet 3y agoJust saying you can do better for less and with less effort.
- Havoc 3y agoDid similar with a 4000 series APU. It’s a bit weak for LLMs under near any circumstances but thinking it could probably handle two whisper cpp decodes over opencl and maybe some embeddings api stuff.
- KETpXDDzR 3y agoMisleading title and article. The CPU doesn't "become" a GPU, it already shippes with one. What the engineer did is solely use the Ryzen APU's GPU for ML.
- Koshkin 3y agoThe author seems to "define" GPU as "what can run A.I. workload". Well: despite the fact that modern GPUs have indeed become more CPU-like, in that they are now suitable for GP (general-purpose) computing, it does not make sense to call a CPU a GPU, even out of good intentions - because it's confusing; a GPU still only plays the role of a co-processors, and its single-thread performance is low compared to the CPU's, among other limitations.