12 ms·
Ship something reasonably priced with 32gb+ and I'd be interested!
by mrbuttons454 3y ago
Ship something reasonably priced with 32gb+ and I'd be interested!
- lukevp 3y agoI was in the market for a new GPU and I debated between a 4090 and a 4070. 4070 is fine for the type of work and gaming I do, but I was thinking about the additional RAM. Then I realized that the 4070 has more vram than most consumers do, so many models will be optimized for that, and at some point someone will come out with a high-ram neural network only card at some point, and that would end up being better than the 4090 anyway because 24 gigs of vram is still quite small. I don’t know a ton about this stuff so who knows if that was the right decision… but I’m thinking along the same lines as you.
- brucethemoose2 3y agoMy compromise: A 3090. Ampere will be well supported until its very obsolete because of the A100.
- squeaky-clean 3y ago> someone will come out with a high-ram neural network only card at some point, and that would end up being better than the 4090 anyway because 24 gigs of vram is still quite small These already exist, they're just disproportionately expensive compared to consumer cards. You can get a Nvidia A100 with 80GB VRAM on eBay for about $14,000. 3.33x the memory for 9x the price.
- YetAnotherNick 3y agoOr you could get A6000 for $4k with 48 GB VRAM.
- moonchrome 3y agoIs 48GB enough to run 70B models ?
- sacred_numbers 3y agoYes, with 4 bit quantization.
- 7speter 3y agoYou can also buy used enterprise cards on ebay. There may not be many 32gb+ cards up yet, but they'll be on there eventually. The big issue is how to cool these cards though, both amd and nvidia make the cards with fan-less heat sinks because the servers they get packed into provide cooling via their loud, ultra fast, potentially digit severing fans, but people who are salvaging these cards have to find their own solution to cool them, and fit them in a given case.
- yrro 3y agoHow purchasers get drivers, don't they require a subscription, or are they pirated?
- 7speter 3y agoAs far as AMD drivers are concerned, you can at least reflash some cards. Nvidia cards... well I've only seen videos where people say they have a given driver so there might be underhanded means involved.
- muttled 3y agoI'm surprised nobody's yet making custom coolers for them (complete coolers with heatsink/fans, not gluing a 3d-printed fan shroud to the front) and re-selling them. Seems like a niche gap in the market.
- brucethemoose2 3y agoAlso, 24GB is good because it's just barely big enough for the (currently unreleased) Llama V2 34B. In Llama V1, 7B was clever but kinda dumb, 13B was still short sighted, but 33B was a big jump to "disturbingly good" territory. 7B/13B finetunes required very specific prompting, but 33B would still give good responses even outside the finetuning format. 16GB is probably perfect for a ~13B model with a very long context and better (5 bit?) quantization, or just partially offloading a ~33B model.
- zacmps 3y agoI don't know, maybe I'm desensitized by GPT4 but I'm running Llama2 70B and I certainly wouldn't call it 'disturbingly good'.
- brucethemoose2 3y agoBase Llama is not good at all, but the finetunes are quite good at their niches. Llama in general is not great for code completion and fact retrieval, from what I have tried
- thewataccount 3y agoCheck out the hugging face leader board and try one of the finetunes. I can't run llama2 70B - but I remember llama1-Guanaco30B was a major leap over the base llama30B model for me.
- zacmps 3y agoThanks! That's useful to know. Unfortunately it looks like a lot of them are non-commercial which rules them out for me but there's a couple that look like they might work.
- brucethemoose2 3y agoIts rather outdated. Search for "70b" on huggingface, sort by date, and you will find more cutting edge models. And you can run this locally with enough combined RAM + VRAM.
- RcouF1uZ4gsC 3y ago> Then I realized that the 4070 has more vram than most consumers do The M1 and M2 Apple processors might change that equation. The 32gb and 64gb MacBooks can run inference on a lot of these big models. They might be quite a bit slower in TFLOPS than a 4070, but if they can run the model while the 4070 can't run it due to limited RAM, then they come out the winner
- pram 3y agoYes I could run llama 65b on my M1 Max with 64GB of ram.
- washadjeffmad 3y agoThat's true, and you do make a disclaimer about performance, but often when this is brought up, people either haven't actually tried MPS or are unaware that a GPU isn't required for inference. The long and short of it is, right now, CUDA beats TFLOPS. Now for training, $10K for an M2 with 192GB memory could break even somewhere around a dozen models with current costs, iff the compute is not saturated. The limit of memory bandwidth is only a function of the compute; it doesn't scale infinitely.
- lukevp 3y agoI do also have a MacBook Pro with 32 gigs of Ram and an M2 Studio with 64 gigs, so if a great model comes out that supports unified memory, I should be good there!
- h11h 3y agoIf anything, 24GB is probably the sweet spot for what's optimised for as many local LLM enthusiasts are running 3090s and 4090s (and all want more VRAM).
- brucethemoose2 3y agoI just ordered a 3090, but I would have bought a 32GB A770 instead without even blinking. The card is 256 bit... Intel could have done this, just like the 16GB RTX 4060 with a 128 bit bus.
- lumost 3y agoVery hopeful for unified/configurable memory. Why are gpu vendors deciding how much ram the gpu gets? This feels so 1980s. GenAI workloads have radically variable memory footprints, some folks want 256GB per card so they have a spitting shot at trying something out - and other folks want 8GB per card to maximize throughput. To add complication, I really don’t want to deal with copying 256GB models between cpu/card. We should have unified memory.
- brucethemoose2 3y agoCXL is that, but a compromise. Practically, that means an APU. Intel/AMD are reportedly coming out with wide M1-Pro like chips.
- bick_nyers 3y agoAt the end of the day it's limited by the PCIE lanes it is attached to, no different from a 2nd GPU. Just saw one yesterday, 128GB PCIE 5.0 x8 which is a maximum of 32 GB/s in and out, where the 4090 has 1 TB/s memory bandwidth. IMO for inference, dual socket EPYC (24 channel DDR5) is the way to go and CXL can theoretically allow you to bump up the bandwidth in that situation (assuming that you can optimize the software properly). Already in llama.cpp there seems to be some issues using 2P servers regarding numa.
- smoldesu 3y ago> Why are gpu vendors deciding how much ram the gpu gets? ...because they make them? You can go buy 256gb systems if you want, at extreme cost. You can also use an 8gb card for whatever you want too. What are they doing wrong? > To add complication, I really don’t want to deal with copying 256GB models between cpu/card. For deployment, it doesn't really matter. You just memory-map it to the CPU or keep it hot-loaded in GPU memory. With a unified memory model you're still limited by your disk speed, so the performance difference probably wouldn't come out great either way.
- lumost 3y agoIntel makes microchips, but they do not control how much memory I put in them. There are no 256 GB GPUs available today outside of multi-GPU setups. Would love to see giant NUMA GPU boards.