4 ms·
Gemma 27b? Command R (32b) would be ideal middle grounds but won’t fit in a 16gb card. There are a handful of 12gb like the new mistral though. I doubt that 12b
by vineyardmike 2y ago
Gemma 27b? Command R (32b) would be ideal middle grounds but won’t fit in a 16gb card. There are a handful of 12gb like the new mistral though. I doubt that 12b overs much improvement over a 7b to compare to 70b. Seems like an entirely different class.
You probably want to limit yourself if you do have a 16gb card because you still need to fit the context window in memory too.
- wkat4242 2y agoI tried gemma2 but it has to be quantified to hell to fit in a 16gb card and then it performs a lot worse than llama3 8b unforutnately. I hope the new mistral comes out soon for ollama. True about the context window but with llama3 this was not a problem as it has such a small context window anyway.