9 ms·
I shelled out for a 4090 when they came out thinking it would be the key factor for running local llms. It turns out that anything worth running takes way more
by reilly3000 2y ago
I shelled out for a 4090 when they came out thinking it would be the key factor for running local llms. It turns out that anything worth running takes way more than 24GB VRAM. I would have been better off with 2+ 3090s and a custom power supply. It’s a pity because I thought it would be a great solution for coding and a home assistant, but performance and quality isn’t there yet for small models (afaik). Perhaps DIGITS will scratch the itch for local LLM developers, but performant models really want big metal for now, not something I can afford to own or rent at my scale.
- UncleOxidant 2y agoThere was a post on r/localLlama the other day about a presentation by the company building Digits hardware for Nvidia. The gist was that Digits is going to be aimed at academic AI research folks and as such don't expect them to be available in large numbers (at least not for this first version). It was disappointing. Now I'm awaiting the AMD Strix Halo based systems.
- wmf 2y agoLaptops with the same chip however...
- unethical_ban 2y agoI haven't tested programming tasks with a local LLM vs. say, Claude 3.5. But it is nice to be able to run 14-32B LLMs locally and get an instant response. I have a single 3090.
- prettyblocks 2y agoSame here. I just built a pc with a 3090 for local llm and stable diffusion and have zero regrets.
- 01HNNWZ0MV43FF 2y agoGosh. Good thing I haven't bought a GPU in almost a decade. With a little luck I'll catch this wave on the back end. I haven't had to learn web or mobile development thoroughly either
- zozbot234 2y agoThe best bang for the buck on VRAM is a maxed out Mac Studio.
- fabiensanglard 2y agoAbsolutely! I have been playing with Ollama on a Macbook Pro 192 GiB RAM and it is able to run most models whereas my 3090 runs our of RAM.
- sepositus 2y agoDo you mean 128GB? Not aware of any variant of the Macbook Pro with that much RAM.
- simonw 2y ago192GB is available for the M2 Mac Studio.
- mdaniel 2y agoI was curious "how bad is it?" and it seems $5500-ish https://www.ebay.com/sch/i.html?_nkw=192gb+studio&_sop=15 https://www.ebay.com/sch/i.html?_nkw=192gb+studio&_sop=15
- fiberhood 2y ago$6500 depending on VAT. But 10-12 times M4 Mac mini's with 100 Gbps networking gives you triple the cores and 160 GB with 2.5 times the memeory bandwith if the sharding of the NN layers is done right.
- fennecfoxy 2y ago$6500!! You may as well buy 5x3090s for $1000 each for 120GB ram, spend the extra $1500 on the sundries. Like, I'm sure Nvidia is aware of Apple's "unified memory" as an alternative to their cards and yet...they aren't offering >24GB consumer cards yet, so clearly they don't feel threatened. Don't get me wrong, I've always disliked Apple as a company, but the M series chips are brilliant, I'm writing this on one right now. But people seem to think that Apple will be able to get the same perf increases yoy when they're really stretching process limits by dumping everything onto the same die like that - where do they go from here? That said Nvidia is using HBM so it does make me wonder why they aren't also doing memory on package with HBM, I think SK Hynix et al were looking at making this possible. I'm glad we're headed in the direction of 3d silicon though, always seemed like we may as well scale in z, I imagine they can stack silicon/cooling/silicon/cooling etc. I'm sure they can use lithography to create cooling dies to sandwich between everything else. Then just pass connections/coolant through those.
- helpfulclippy 2y agoMy thoughts, expenses and regrets exactly.
- jwr 2y agoI started buying Macs with more memory, no regrets. An M4 Max with 64GB (in a laptop, no less!) runs most small models comfortably (but get 96GB or more if you really intend to use 70B models regularly). And when I'm not running LLMs, the memory is useful for other stuff.
- ignoramous 2y ago> And when I'm not running LLMs, the memory is useful for other stuff Let's be honest, the other stuff is just Chrome: Tell me 96gb is enough?
- simonw 2y agoFirefox and VS Code for me: I have 64GB and I can't run Llama 70B locally without closing a ton of windows and tabs first!
- LtdJorge 2y agoHave you tried auto tab discard?
- user_7832 2y agoNot the person you replied to but auto discard only helps so much. Even with it on I often use 80-90% memory (albeit on Windows, not sure if macOS is any different in this regard).
- bornfreddy 2y agoFirefox here too, no such problems. I do however run Google services in a separate browser (Brave or Chromium) because most of them hog the Firefox browser. For example running earth.google.com on FF is... a very special experience. :-/
- deleted 2y ago[deleted]
- SunlitCat 2y agoNothing stops you from getting a second 4090 as well. ^^
- raymond_goo 2y agono https://www.reddit.com/r/24gb/ https://www.reddit.com/r/24gb/