4 ms·
It used to be that for locally running GenAI, VRAM per dollar was king, so used NVidia RTX 3090 cards were the undisputed darlings of DYI LLM with 24GB for 600€
by Tepix 2y ago
It used to be that for locally running GenAI, VRAM per dollar was king, so used NVidia RTX 3090 cards were the undisputed darlings of DYI LLM with 24GB for 600€-800€ or so. Sticking two of these in one PC isn't too difficult despite them using 350W each.
Then Apple introduced Macs with 128 GB and more unified memory at 800GB/s and the ability to load models as large as 70GB (70b FP8) or even larger ones. The M1 Ultra was unable to take full advantage of the excellent RAM speed, but with the M2 and the M3, performance is improving. Just be prepared to spend 5000€ or more for a M3 Ultra. Another alternative would be a EPYC 9005 system with 12x DDR5-6000 RAM for 576GB/s of memory bandwidth with the LLM (preferably MoE) running on the CPU instead of a GPU.
However today, with the latest, surprisingly good reasoning models like QwQ-32B using up thousands or tens of thousands of tokens in their replies, performance is getting more important than previously and these systems (Macs and even RTX 3090s) might fall out of favor, because waiting for a finished reply will take several minutes or even tens of minutes. Nvidia Ampere and Apple silicon (AFAIK) are also missing FP4 support in hardware, which doesn't help.
For the same reason AMD Halo Strix with a mere 273GB/s of RAM bandwidth and perhaps also NVidia Project Digits (also speculated to offer similar RAM bandwidth) might just be too slow for reasoning models with more than 50GB or so of active parameters.
On the other hand, if the prices for the RTX 5090 remain at 3500€, they will likely remain insignificant for the DIY crowd for that reason alone.
Perhaps AMD will take the crown with a variant of their RDNA4 RX 9070 card with 32GB of VRAM priced at around 1000€? Probably wishful thinking…
- BoredPositron 2y agoThe 3090 was always a fallacy without native fp8.
- Tepix 2y agoPerformance is good enough for non-reasoning models even if they're FP8 or FP4. Check the phoronix article, the difference between the 3090 and 4090 is rather small. There's weight-only FP8 in vLLM on NVidia Ampere: https://docs.vllm.ai/en/latest/features/quantization/fp8.html https://docs.vllm.ai/en/latest/features/quantization/fp8.htm...
- imtringued 2y agoHow? It's not like Nvidia is some also-ran company for which people did not build custom kernels that combine dequantization and GEMV/GEMM in a single kernel.
- threeducks 2y agoPerformance for text generation is memory-limited, so lack of native fp8 support does not matter. You have more than enough compute left over to do the math in whichever floating point format you fancy.
- qeternity 2y agoWhat? There are highly optimized Marlin kernels for W8A16 that function very well on a 3090 in both FP8 and INT8 formats.
- porphyra 2y agoThere are Chinese modded 4090s with 48 and 96 GB of ram that seem like a sweet spot for fast inference of these moderately sized models.
- BoorishBears 2y agoThe 96GB 4090 is still at the "blurry shots of nvidia-smi on Chinese TikTok" stage. Granted the 48 GB 4090s started there too before materializing and even becoming common enough to be on eBay, but this time there are more technical barriers and it's less likely they'll ever show up in meaningful numbers.
- noahbp 2y agoWe've progressed to the "more detailed photos and screen grabs on Twitter" stage: https://x.com/bdsqlsz/status/1898307273967145350 https://x.com/bdsqlsz/status/1898307273967145350
- johnisgood 2y agoFor how much? From where would one obtain them? Are they legit? I have found a lot of Chinese knock-offs for various GPUs for super cheap, but they were faulty (in my experience).
- qingcharles 2y agoIf you look on the local llama sub on Reddit there are threads about them; I bet there are links to the Chinese sellers.
- Tepix 2y agoThey are promising but they are priced very far beyond the 30€/GB of the used RTX 3090
- sgt 2y agoI've heard however that the EPYC rigs are in practice getting very low tokens/sec and the Macs like the Ultras and high memory are getting much beter - like by an order of magnitude. So in that sense, the only sensible option now (i.e. "local energy efficient LLM on a budget") is to get the Mac.
- magicalhippo 2y agoThe previous time this article was submitted, I did some calculations based on the charts and found[1] that for the NVIDIA 40 and 50-series GPUs, the results are almost entirely explained by memory bandwidth: Each of the cards except the 5090 gets almost exactly 0.1 token/s per GB/s memory bandwidth. My understanding is that the Macs have soldered memory which allows for much higher memory bandwidth. The M4 has ~400-550 GB/s max depending on configuration[2], while EPYCs seem to have more like 250GB/s max[3]. [1]: https://news.ycombinator.com/item?id=42847284 https://news.ycombinator.com/item?id=42847284 [2]: https://support.apple.com/en-us/121553 https://support.apple.com/en-us/121553 [3]: https://www.servethehome.com/here-is-why-you-should-fully-populate-memory-channels-on-cpus-featuring-amd-epyc-genoa/ https://www.servethehome.com/here-is-why-you-should-fully-po...
- ac29 2y ago> EPYCs seem to have more like 250GB/s max Your link goes to info on the 2022 EPYC CPUs, the current generation can do 576GB/s: https://chipsandcheese.com/p/amds-turin-5th-gen-epyc-launched https://chipsandcheese.com/p/amds-turin-5th-gen-epyc-launche... Intel's current 12ch Xeons should be even faster with MRDIMMs, though I couldnt find a memory specific benchmark.
- magicalhippo 2y agoAh shoot, that's what one gets for being in a hurry and on the phone. Saw the date of the article and mention of the EPYC 9004, but forgot that it's the 9005 that's the new series and missed the details. Thanks for the correction. edit: found a llama.cpp issue discussing performance bottlenecks on modern dual-socket EPYC here[1]. Also includes single-socket benchmarks, and includes some optimizations. Just thought it was interesting. [1]: https://github.com/ggml-org/llama.cpp/discussions/11733 https://github.com/ggml-org/llama.cpp/discussions/11733
- xiphias2 2y agoDoes FP4 support really matter if local inferencing is RAM bandwidth limited anyways? I think we should put more effort into compressing the models well to be able to use local GPUs better.
- qeternity 2y agoWell, FP4 (mostly INT4 these days) means you are moving fewer bits per weight around, so the memory bandwidth that you have performs better.
- kccqzy 2y agoThat doesn't make sense to me. Memory bandwidth refers to the throughput of moving data from memory. Even if FP4 calculations aren't natively supported, you still move 4 bits of data per number from memory and then cast it to a FP8 or FP16 or other higher-precision number, right?
- xiphias2 2y agoLlama.cpp already moves 1.5bit to the cores, they are converted in SRAM which has much higher bandwidth. FP4 only helps batched inferencing.
- ai-christianson 2y ago> However today, with the latest, surprisingly good reasoning models like QwQ-32B using up thousands or tens of thousands of tokens in their replies, performance is getting more important than previously and these systems (Macs and even RTX 3090s) might fall out of favor, because waiting for a finished reply will take several minutes or even tens of minutes. 3090 is the last gen to have proper nvlink support, which is supported for LLM inference in some frameworks. Would 1x 5090 be faster than 2x 3090?
- whalesalad 2y ago1x5090 not enough vram
- reginald78 2y agoI was browsing r/localllm recently and was surprised that some people were purchasing two 3060 12GB and using them in tandem somehow. I actually didn't think this would work at all without nvlink.
- magicalhippo 2y agoYou just copy data over PCIe, however it is slower than nvlink. There are several ways to utilize multiple GPUs, the main contenders as I understand it is pipelined and so-called tensor parallelism. Think of it as slicing a loaf of bread in regular slices vs along its longer axis. The former can have higher peak throughput while the latter can have lower latency, though it depends on the details[1]. [1]: https://blog.squeezebits.com/vllm-vs-tensorrtllm-9-parallelism-strategies-36310 https://blog.squeezebits.com/vllm-vs-tensorrtllm-9-paralleli...
- ai-christianson 2y ago> You just copy data over PCIe AFAIK this only happens directly over PCIe if using hacked drivers with p2p enabled (I think tinygrad/tinybox provided these drivers initially.) Otherwise, data goes through system bus/CPU first. `nvidia-smi topo -m` will show how the GPUs are connected.
- nyrikki 2y agoQwQ-32B on 3*1080ti does work with a 4096 window @ ~14 eval tokens/s But the lower bounds on intermediate tokens on CoT to get close to PTIME expressability + QwQ-32B's verbosity/repeating on those tokens does eat up that pretty quickly. In theory the DAG walking QwQ-32B appears to be doing should require O(|E| log|V|) scratch space. IMHO they need to train to target that more. It seems fairly trivial to extend the context loss problem to the public UI, The Ω(n) scratch space may be hard to hit in reality but they seem almost exponential in scratch space right now, even on tasks that a traditional LLM could answer with just approximate retrieval. I may just be suffering confirmation bias, but the problem with memory in this case seems to be related to making the intermediate tokens palatable to users IMHO.