3 ms·
We want cheap(ish) desktop GPUs with 80G or more VRAM, so not only large corporations could meaningfully participate in AI research. The price divide between “
by atemerev 2y ago
We want cheap(ish) desktop GPUs with 80G or more VRAM, so not only large corporations could meaningfully participate in AI research.
The price divide between “desktop” and “datacenter” GPUs is artificial and no doubt there is a collusion of some kind.
- BoredPositron 2y agoYou know that their top tier datacenter card has "only" 80GB? Their workstation cards with 48GB are also reasonably priced... with 3-4k pricing.
- atemerev 2y agoNvidia B100 has 192G VRAM.
- BoredPositron 2y agoThey are not shipping in any reasonable quantity. They are also 2x96GB.
- ginko 2y ago>Their workstation cards with 48GB are also reasonably priced... with 3-4k pricing. That's twice the price of a 4090. You can't tell me the extra RAM costs that much.
- BoredPositron 2y agoIt's not only VRAM. more cores, more bandwith, better power delivery, workstation features...
- atemerev 2y agoWhich card do you mean that has more cores/computing performance than 4090 and costs 4-5k?
- colonelspace 2y agoNVIDIA RTX 6000 Ada Generation Although they retail for more than $4k. The previous generation RTX A6000 is probably what they're referring to above.
- atemerev 2y agoIt is $7.5k, and slower than 4090 compute-wise. Two 4090s (which are cheaper than A6000) are strictly superior to it.
- colonelspace 2y agoIndeed two 4090s provide more CUDA/RT/Tensor cores, but the A6000 gives you 48GB of RAM which can end up making the card faster at certain tasks that would require swapping out data to system RAM on a 24GB card.
- atemerev 2y agoTwo 4090s have 48G too. For most (if not all) LLM-related tasks, there is no difference how many cards are used for the same amount of VRAM.
- brookst 2y agoYou can get Macs with 192GB of UMA, so maybe 180GBusable VRAM. But of course the GPU horsepower is much less tban an A100.
- atemerev 2y agoSort of works for inference, but not for anything else. For now, I bought a 4x4090 workstation… but I am crying how power-inefficient it is.
- ra120271 2y agoFor my education, would you be able to expand on what it doesn't work for and why? Thank you!
- Gracana 2y agoInference (token generation) is memory-bound, KV cache prefill (prompt processing) is compute-bound. The ARM Macintoshes have lots of memory bandwidth but not a lot of compute power, so they're great for outputting text but terrible for tasks like analyzing documents. I've never done fine-tuning but my understanding is that that is a highly-parallelizable compute hog as well. You might like this article, which looks at the arithmetic intensity of LLM processing: https://www.baseten.co/blog/llm-transformer-inference-guide/ https://www.baseten.co/blog/llm-transformer-inference-guide/