6 ms·
I was in the market for a new GPU and I debated between a 4090 and a 4070. 4070 is fine for the type of work and gaming I do, but I was thinking about the addit
by lukevp 3y ago
I was in the market for a new GPU and I debated between a 4090 and a 4070. 4070 is fine for the type of work and gaming I do, but I was thinking about the additional RAM. Then I realized that the 4070 has more vram than most consumers do, so many models will be optimized for that, and at some point someone will come out with a high-ram neural network only card at some point, and that would end up being better than the 4090 anyway because 24 gigs of vram is still quite small. I don’t know a ton about this stuff so who knows if that was the right decision… but I’m thinking along the same lines as you.
- brucethemoose2 3y agoMy compromise: A 3090. Ampere will be well supported until its very obsolete because of the A100.
- squeaky-clean 3y ago> someone will come out with a high-ram neural network only card at some point, and that would end up being better than the 4090 anyway because 24 gigs of vram is still quite small These already exist, they're just disproportionately expensive compared to consumer cards. You can get a Nvidia A100 with 80GB VRAM on eBay for about $14,000. 3.33x the memory for 9x the price.
- YetAnotherNick 3y agoOr you could get A6000 for $4k with 48 GB VRAM.
- moonchrome 3y agoIs 48GB enough to run 70B models ?
- sacred_numbers 3y agoYes, with 4 bit quantization.
- 7speter 3y agoYou can also buy used enterprise cards on ebay. There may not be many 32gb+ cards up yet, but they'll be on there eventually. The big issue is how to cool these cards though, both amd and nvidia make the cards with fan-less heat sinks because the servers they get packed into provide cooling via their loud, ultra fast, potentially digit severing fans, but people who are salvaging these cards have to find their own solution to cool them, and fit them in a given case.
- yrro 3y agoHow purchasers get drivers, don't they require a subscription, or are they pirated?
- 7speter 3y agoAs far as AMD drivers are concerned, you can at least reflash some cards. Nvidia cards... well I've only seen videos where people say they have a given driver so there might be underhanded means involved.
- muttled 3y agoI'm surprised nobody's yet making custom coolers for them (complete coolers with heatsink/fans, not gluing a 3d-printed fan shroud to the front) and re-selling them. Seems like a niche gap in the market.
- brucethemoose2 3y agoAlso, 24GB is good because it's just barely big enough for the (currently unreleased) Llama V2 34B. In Llama V1, 7B was clever but kinda dumb, 13B was still short sighted, but 33B was a big jump to "disturbingly good" territory. 7B/13B finetunes required very specific prompting, but 33B would still give good responses even outside the finetuning format. 16GB is probably perfect for a ~13B model with a very long context and better (5 bit?) quantization, or just partially offloading a ~33B model.
- zacmps 3y agoI don't know, maybe I'm desensitized by GPT4 but I'm running Llama2 70B and I certainly wouldn't call it 'disturbingly good'.
- brucethemoose2 3y agoBase Llama is not good at all, but the finetunes are quite good at their niches. Llama in general is not great for code completion and fact retrieval, from what I have tried
- thewataccount 3y agoCheck out the hugging face leader board and try one of the finetunes. I can't run llama2 70B - but I remember llama1-Guanaco30B was a major leap over the base llama30B model for me.
- zacmps 3y agoThanks! That's useful to know. Unfortunately it looks like a lot of them are non-commercial which rules them out for me but there's a couple that look like they might work.
- brucethemoose2 3y agoIts rather outdated. Search for "70b" on huggingface, sort by date, and you will find more cutting edge models. And you can run this locally with enough combined RAM + VRAM.
- RcouF1uZ4gsC 3y ago> Then I realized that the 4070 has more vram than most consumers do The M1 and M2 Apple processors might change that equation. The 32gb and 64gb MacBooks can run inference on a lot of these big models. They might be quite a bit slower in TFLOPS than a 4070, but if they can run the model while the 4070 can't run it due to limited RAM, then they come out the winner
- pram 3y agoYes I could run llama 65b on my M1 Max with 64GB of ram.
- washadjeffmad 3y agoThat's true, and you do make a disclaimer about performance, but often when this is brought up, people either haven't actually tried MPS or are unaware that a GPU isn't required for inference. The long and short of it is, right now, CUDA beats TFLOPS. Now for training, $10K for an M2 with 192GB memory could break even somewhere around a dozen models with current costs, iff the compute is not saturated. The limit of memory bandwidth is only a function of the compute; it doesn't scale infinitely.
- lukevp 3y agoI do also have a MacBook Pro with 32 gigs of Ram and an M2 Studio with 64 gigs, so if a great model comes out that supports unified memory, I should be good there!
- h11h 3y agoIf anything, 24GB is probably the sweet spot for what's optimised for as many local LLM enthusiasts are running 3090s and 4090s (and all want more VRAM).