4 ms·
Yarn-Mistral-7B-128k
- brucethemoose2 3y agoThis is super cool. Mistral is great, the full 128K context will fit on cheap (24GB, maybe less) GPUs thanks to exLlamav2, and they just pulled support into transformers instead of dropping some weird proprietary code. One critique of the graph though... I measured perplexity of MistralLite myself, and its good out to 32K, but in the graph it drops off where the base model drops: https://huggingface.co/SciPhi/SciPhi-Mistral-7B-32k/discussions/3#6541bd13ae02affd743d595e https://huggingface.co/SciPhi/SciPhi-Mistral-7B-32k/discussi...
- mycall 3y agoNoob here. If you try to 128K into a 12GB card, would it fail or just be super slow?
- deleted 3y ago[deleted]
- brucethemoose2 3y agoexllamav2 is super efficient about this, with its 8 bit cache. I'm not sure about that specifically (I am away from my desktop at the moment), but my 24GB card can handle Yi 34B 200K out to 50K+, depending on how much its quantized. I would bet you can do 50K+ on 12GB with Mistral.
- siquick 3y agoIs there any rough rules on what model sizes will work on various RAM sizes on consumer machines?
- washadjeffmad 3y agoFor "full" context, typically: 7B: 8GB (4GB min for model only) 13B: 12GB (8GB min) 34B: 28GB (18GB min) 70B: 60GB (38GB min) Lower bpw quants and 8-bit cache size can reduce memory requirements / expand context, but if you're sticking to mainstream L1/L2 models, just go by that.
- coder543 3y agoFor 128k context (even with a 7B model), I don't think 8GB is nearly enough. I've heard there might be tricks to get it under 24GB... but I haven't personally seen it under about 28GB, IIRC. Definitely not heard of it being possible with 8GB.
- kiviuq 3y agothere's a calculator / estimator somewhere on Hugging Face that can give you some approximate numbers based on the model you select.
- brucethemoose2 3y agoWith exllama, the number is "flexible" because it supports completely arbitrary quantization levels. 7B fits comfortably in 6GB. 70B will squeeze, into 24GB, albeit with significant (but not catastrophic) quality loss.
- seydor 3y agowhy does this require trust_remote_code
- minimaxir 3y agoBecause it requires shenanigans to get to the 128k context window. It's not malicious, you can view the code yourself in the .py files: https://huggingface.co/NousResearch/Yarn-Mistral-7b-128k/tree/main https://huggingface.co/NousResearch/Yarn-Mistral-7b-128k/tre...
- luke-stanley 3y agoYou could also ask, why Pickle and not Safer Tensors or Msgpack?
- sp332 3y agoI think any model type that is not built in to the huggingface library needs that.
- pram 3y agoLooks like you can already use it with ollama, nice https://ollama.ai/library/yarn-mistral https://ollama.ai/library/yarn-mistral
- wenc 3y agoNice! I can finally run a model locally on my 16GB M2 Pro Mac without a lot of futzing around with CUDA. Ollama is very nice.
- deleted 3y ago[deleted]
- geckbench 3y ago[dead]
- faizshah 3y agoProbably a dumb question, is this instruction tuned?
- skissane 3y agoIt says it is based on https://huggingface.co/mistralai/Mistral-7B-v0.1 https://huggingface.co/mistralai/Mistral-7B-v0.1 not https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.1 https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.1 so I am guessing "No"
- brucethemoose2 3y agoNo, but there is already a 64K Airoboros instruction tune based on this.
- kristianp 3y agoThere's gguf versions (for llama.cpp) at https://huggingface.co/TheBloke/Yarn-Mistral-7B-128k-GGUF https://huggingface.co/TheBloke/Yarn-Mistral-7B-128k-GGUF Llama.cpp is good for running on macs and for partial gpu offloading on non-macs. There's also a table of memory usage depending on which file you use.