3 ms·
I kind of wonder why a lot of these places don't give "amateur" sized models anymore at around the 18B & 30B parameter sizes that you can run on a single 3090 o
by novok 2y ago
I kind of wonder why a lot of these places don't give "amateur" sized models anymore at around the 18B & 30B parameter sizes that you can run on a single 3090 or M2 Max at reasonable speeds and RAM requirements? It's all 7B, 70B, 400B sizing nowadays.
- TobTobXX 2y agoJust a few days ago, Mistral released a 12B model: https://mistral.ai/news/mistral-nemo/ https://mistral.ai/news/mistral-nemo/
- logicchains 2y agoBecause you can just quantise the 70B model to 3-4 bits and it'll perform better than a 30B model but be a similar size.
- novok 2y agoA 70B 4bit model does not fit in a 24GB VRAM card, 30B models are the sweet spot for that size of card at 20GB, with 4GB left for the system to still function.