4 ms·
TheBloke's GGUF model card tells you how much RAM you'll want to run the various versions of the model: https://huggingface.co/TheBloke/Mixtral-8x7B-v0.1-GGUF#p
by loudmax 3y ago
TheBloke's GGUF model card tells you how much RAM you'll want to run the various versions of the model:
https://huggingface.co/TheBloke/Mixtral-8x7B-v0.1-GGUF#provided-files https://huggingface.co/TheBloke/Mixtral-8x7B-v0.1-GGUF#provi...
In the article, Simon mentions the Q6_K.gguf model, which is about 40GB. A Mac Studio can handle this, but any of these models are going to be a tight fit or impossible on a Mac laptop without swapping to disk. Maybe NVME is fast enough that swapping isn't too terrible.
In my experience, the Mixtral models work pretty well on llama.cpp on my Linux workstation with a 10GB GPU, and offloading the rest to CPU.
It is impressive how fast the smaller models are improving. Still, a safe rule of thumb is the more RAM the better.
Also, really question how much you need to run these models locally. If you just want to play around with these models, it's probably far more cost effective to rent something in the cloud.
- williamstein 3y agoI tried Mixtral via ollama on my Apple M1 Max with 32GB of RAM, and was a total nonstarter. I ended up having to powercycle my machine. I then just used two L4 GPU's on Google Cloud (so 48GB of GPU RAM, see [1]) and it was very smooth and fast there. [1] https://github.com/sagemathinc/cocalc-howto/blob/main/ollama.md https://github.com/sagemathinc/cocalc-howto/blob/main/ollama...
- jmorgan 3y agoWow, as an author of the project I'm so sorry about you having to restart your computer. The memory management in Ollama needs a lot of improvement – will be working on this a bunch going forward. I also have a M1 32GB Mac and it's unfortunately just below the amount of memory Mixtral needs to run well (for now!)