3 ms·
I suspect 30B models are in a weird spot, too big for widespread home use, too small for cutting edge performance. For home users 7B models (which can fit on a
by michaelt 2y ago
I suspect 30B models are in a weird spot, too big for widespread home use, too small for cutting edge performance.
For home users 7B models (which can fit on an 8GB GPU) and 13B models (which can fit on a 16GB GPU) are in far more demand. If you're a researcher, you want a 70B model to get the best performance, and so your benchmarks are comparable to everyone else.
- drdaeman 2y agoI thought home use is whatever fits in 24GB (a single 3090 GPU, which is pretty affordable), not 8 or 16. 30B models fit.
- michaelt 2y agoWhile some home users do indeed have 24GB of vram, the fact is a 4090 costs $1700 Such models will never top the number of downloads charts, or the community hype, as there’s just loads more people who can use the smaller models. And if you can afford one 4090 you can probably afford two.
- drdaeman 2y agoWhy 4090, though? I read (and agree) that 3090 is generally considered to be the best bang for the buck: 24GB, priced at $800-1000 range, and giving decent TPS for LLMs.