3 ms·
Noob here. If you try to 128K into a 12GB card, would it fail or just be super slow?
by mycall 3y ago
Noob here. If you try to 128K into a 12GB card, would it fail or just be super slow?
- deleted 3y ago[deleted]
- brucethemoose2 3y agoexllamav2 is super efficient about this, with its 8 bit cache. I'm not sure about that specifically (I am away from my desktop at the moment), but my 24GB card can handle Yi 34B 200K out to 50K+, depending on how much its quantized. I would bet you can do 50K+ on 12GB with Mistral.