exllamav2 is super efficient about this, with its 8 bit cache. I'm not sure about that specifically (I am away from my desktop at the moment), but my 24GB card

exllamav2 is super efficient about this, with its 8 bit cache. I'm not sure about that specifically (I am away from my desktop at the moment), but my 24GB card can handle Yi 34B 200K out to 50K+, depending on how much its quantized. I would bet you can do 50K+ on 12GB with Mistral.