4 ms·
This is super cool. Mistral is great, the full 128K context will fit on cheap (24GB, maybe less) GPUs thanks to exLlamav2, and they just pulled support into tra
by brucethemoose2 3y ago
This is super cool. Mistral is great, the full 128K context will fit on cheap (24GB, maybe less) GPUs thanks to exLlamav2, and they just pulled support into transformers instead of dropping some weird proprietary code.
One critique of the graph though... I measured perplexity of MistralLite myself, and its good out to 32K, but in the graph it drops off where the base model drops: https://huggingface.co/SciPhi/SciPhi-Mistral-7B-32k/discussions/3#6541bd13ae02affd743d595e https://huggingface.co/SciPhi/SciPhi-Mistral-7B-32k/discussi...
- mycall 3y agoNoob here. If you try to 128K into a 12GB card, would it fail or just be super slow?
- deleted 3y ago[deleted]
- brucethemoose2 3y agoexllamav2 is super efficient about this, with its 8 bit cache. I'm not sure about that specifically (I am away from my desktop at the moment), but my 24GB card can handle Yi 34B 200K out to 50K+, depending on how much its quantized. I would bet you can do 50K+ on 12GB with Mistral.