4 ms·
I thought home use is whatever fits in 24GB (a single 3090 GPU, which is pretty affordable), not 8 or 16. 30B models fit.
by drdaeman 2y ago
I thought home use is whatever fits in 24GB (a single 3090 GPU, which is pretty affordable), not 8 or 16. 30B models fit.
- michaelt 2y agoWhile some home users do indeed have 24GB of vram, the fact is a 4090 costs $1700 Such models will never top the number of downloads charts, or the community hype, as there’s just loads more people who can use the smaller models. And if you can afford one 4090 you can probably afford two.
- drdaeman 2y agoWhy 4090, though? I read (and agree) that 3090 is generally considered to be the best bang for the buck: 24GB, priced at $800-1000 range, and giving decent TPS for LLMs.