4 ms·
I've been kicking this around in my head for a while. If I want to run LLMs locally, a decent GPU is really the only important thing. At that point, the questio
by yjftsjthsd-h 11mo ago
I've been kicking this around in my head for a while. If I want to run LLMs locally, a decent GPU is really the only important thing. At that point, the question becomes, roughly, what is the cheapest computer to tack on the side of the GPU? Of course, that assumes that everything does in fact work; unlike OP I am barely in a position to understand eg. BAR problems, let alone try to fix them, so what I actually did was build a cheap-ish x86 box with a half-decent GPU and called it a day:) But it still is stuck in my brain: there must be a more efficient way to do this, especially if all you need is just enough computer to shuffle data to and from the GPU and serve that over a network connection.
- zeusk 11mo agoGet the DGX Spark computers? They’re exactly what you’re trying to build.
- Gracana 11mo agoThey’re very slow.
- geerlingguy 11mo agoThey're okay, generally, but slow for the price. You're more paying for the ConnectX-7 networking than inference performance.
- Gracana 11mo agoYeah, I wouldn’t complain if one dropped in my lap, but they’re not at the top of my list for inference hardware. Although... Is it possible to pair a fast GPU with one? Right now my inference setup for large MoE LLMs has shared experts in system memory, with KV cache and dense parts on a GPU, and a Spark would do a better job of handling the experts than my PC, if only it could talk to a fast GPU. [edit] Oof, I forgot these have only 128GB of RAM. I take it all back, I still don’t find them compelling.
- tcdent 11mo agoWe're not yet to the point where a single PCIe device will get you anything meaningful; IMO 128 GB of ram available to the GPU is essential. So while you don't need a ton of compute on the CPU you do need the ability address multiple PCIe lanes. A relatively low-spec AMD EPYC processor is fine if the motherboard exposes enough lanes.
- skhameneh 11mo agoThere is plenty that can run within 32/64/96gb VRAM. IMO models like Phi-4 are underrated for many simple tasks. Some quantized Gemma 3 are quite good as well. There are larger/better models as well, but those tend to really push the limits of 96gb. FWIW when you start pushing into 128gb+, the ~500gb models really start to become attractive because at that point you’re probably wanting just a bit more out of everything.
- tcdent 11mo agoIDK all of my personal and professional projects involve pushing the SOTA to the absolute limit. Using anything other than the latest OpenAI or Anthropic model is out of the question. Smaller open source models are a bit like 3d printing in the early days; fun to experiment with but really not that valuable for anything other than making toys. Text summarization, maybe? But even then I want a model that understands the complete context and does a good job. Even things like "generate one sentence about the action we're performing" I usually find I can just incorporate it into the output schema of a larger request instead of making a separate request to a smaller model.
- xyzzy123 11mo agoIt seems to me like the use case for local GPUs is almost entirely privacy. If you buy a 15k AUD rtx 6000 96GB, that card will _never_ pay for itself on a gpt-oss:120b workload vs just using openrouter - no matter how many tokens you push through it - because the cost of residential power in Australia means you cannot generate tokens cheaper than the cloud even if the card were free.
- dist-epoch 11mo agoThis problem was already solved 10 years ago - crypto mining motherboards, which have a large number of PCIe slots, a CPU socket, one memory slot, and not much else. > Asus made a crypto-mining motherboard that supports up to 20 GPUs https://www.theverge.com/2018/5/30/17408610/asus-crypto-mining-motherboard-gpus https://www.theverge.com/2018/5/30/17408610/asus-crypto-mini... For LLMs you'll probably want a different setup, with some memory too, some m.2 storage.
- jsheard 11mo agoThose only gave each GPU a single PCIe lane though, since crypto mining barely needed to move any data around. If your application doesn't fit that mould then you'll need a much, much more expensive platform.
- dist-epoch 11mo agoAfter you load the weights into the GPU and keep the KV cache there too, you don't need any other significant traffic.
- numpad0 11mo agoEven in tensor parallel modes? I thought it could only work if you're fine stalling all but n GPU for n users at any given moments.
- skhameneh 11mo agoIn theory, it’s only sufficient for pipeline parallel due to limited lanes and interconnect bandwidth. Generally, scalability on consumer GPUs falls off between 4-8 GPUs for most. Those running more GPUs are typically using a higher quantity of smaller GPUs for cost effectiveness.
- zozbot234 11mo agoM.2 is mostly just a different form factor for PCIe anyway.
- seanmcdirmid 11mo agoAnd you don’t want to go the M4 Max/M3 Ultra route? It works well enough for most mid sized LLMs.
- binsquare 11mo agoI run a crowd sourced website to collect data on the best and cheapest hardware setup for local LLM here: https://inferbench.com/ https://inferbench.com/ Source code: https://github.com/BinSquare/inferbench https://github.com/BinSquare/inferbench
- nodja 11mo agoCool site, I noticed the 3090 is on there twice. https://inferbench.com/gpu/NVIDIA%20GeForce%20RTX%203090 https://inferbench.com/gpu/NVIDIA%20GeForce%20RTX%203090 https://inferbench.com/gpu/NVIDIA%20RTX%203090 https://inferbench.com/gpu/NVIDIA%20RTX%203090
- binsquare 11mo agoOh nice catch, I'll fix that --- Edit: Fixed
- kilpikaarna 11mo agoNice! Though for older hardware it would be nice if the price reflected the current second hand market (harder to get data for, I know). Eg. Nvidia RTX 3070 ranks as second best GPU in tok/s/$ even at the MSRP of $499. But you can get one for half that now.
- binsquare 11mo agoGreat idea - I've added it by manually browsing ebay for that initial data. So it's just a static value in this hardware list: https://github.com/BinSquare/inferbench/blob/main/src/lib/hardware-data.ts#L55 https://github.com/BinSquare/inferbench/blob/main/src/lib/ha... Let me know if you know of a better way, or contribute :D
- _ea1k 11mo agoIt seems like verification might need to be improved a bit? I looked at Mistral-Large-123B. Someone is claiming 12 tokens/sec on a single RTX 3090 at FP16. Perhaps some filter could cut out submissions that don't really make sense?
- deleted 11mo ago[deleted]
- Eisenstein 11mo agoThere is a whole section in here on how to spec out a cheap rig and what to look for: * https://jabberjabberjabber.github.io/Local-AI-Guide/ https://jabberjabberjabber.github.io/Local-AI-Guide/