3 ms·
4060Ti w/ 16GB VRAM or 3090 w/ 24 GB VRAM Of course with those you'll also have to spend some money on motherboard, ram, SSD, PSU, CPU, ect. I think the best
by oaththrowaway 3y ago
4060Ti w/ 16GB VRAM or 3090 w/ 24 GB VRAM
Of course with those you'll also have to spend some money on motherboard, ram, SSD, PSU, CPU, ect.
I think the best bang for the buck is probably a Mac studio with as much ram as you can afford.
I bought an RTX A2000 (12GB VRAM), and it's fine for 7B models and some 13B models with 4 bit quantization, but I kind of regret not getting something with more VRAM.
- MPSimmons 3y agoI hate how the market is right now. I understand that NVidia doesn't want to provide a consumer level graphics card with truly impressive RAM specs, even though they could, because they feel it would eat into their datacenter market (and truth be told, it probably would), but it's super frustrating that you need to pay so much for decent performance, even as a single machine for personal use.
- theogravity 3y agoJust FYI you can get a hacked 2080 Ti with 22GB for half the cost of a used 3090. https://news.ycombinator.com/item?id=38573884 https://news.ycombinator.com/item?id=38573884