3 ms·

You can also run Mixtral, at a decent token rate, on a post-2020 Apple Macbook Pro M1/M2/M3 with 32GB+ of RAM. 16GB RAM also works, sort of ok, which I suspect

You can also run Mixtral, at a decent token rate, on a post-2020 Apple Macbook Pro M1/M2/M3 with 32GB+ of RAM. 16GB RAM also works, sort of ok, which I suspect is the same quantization a 3090 is using, but I do notice a difference in the quantization. On my M2 Pro, the token rate and intelligence feels like GPT-3.5turbo. This is the first model I've started actually using (vs playing around with for the love of the tech) instead of GPT-3.5. An Apple M2 Pro with 32GB of RAM is in the same price range as a gaming PC with a 3090, but its another example of normal people with moderately high performance systems "accidentally" being able to run a GPT-3.5 comparable model. If you have an Apple meeting these specs and want to play around, LLM Studio is open source and has made it really easy to get started: https://lmstudio.ai/ https://lmstudio.ai/ I hope to see a LOT more hobby hacking as a result of Mixtral and successors.

cjbprime 3y ago
I don't think it's true that LM Studio is open source. Maybe I'm missing something?
- eyegor 3y ago
  Lmstudio (that they linked) is definitely not open source, and doesn't even offer a pricing model for business use. Llmstudio is, but I suspect that was a typo in their comment. https://github.com/TensorOpsAI/LLMStudio https://github.com/TensorOpsAI/LLMStudio
nraford 3y ago
How did you get Mixtral to run an a 32gb M1? I tried using Ollama on my machine (same specs as above) and it told me I needed 49gb RAM minimum.
- eurekin 3y ago
  I'm using: ollama run dolphin-mixtral:8x7b-v2.5-q3_K_S
  - mark_l_watson 3y ago
    That runs on 32G? The original mixtral q3 wouldn’t run for me. Maybe the dolphin tuned version is smaller? EDIT: I just checked, it runs great, thanks.
barnabee 3y ago
I have so far run it on my M1 MacBook using llamafile [1] and found it to be great. Is there any speed/performance/quality/context size/etc. advantage to using LLM Studio or any of the other *llama tools that require more setup than downloading and running a single llamafile executable? [1] https://github.com/Mozilla-Ocho/llamafile/ https://github.com/Mozilla-Ocho/llamafile/
Me1000 3y ago
LM Studio, sadly, is not open source.

cjbprime 3y ago

I don't think it's true that LM Studio is open source. Maybe I'm missing something?

eyegor 3y ago
Lmstudio (that they linked) is definitely not open source, and doesn't even offer a pricing model for business use. Llmstudio is, but I suspect that was a typo in their comment. https://github.com/TensorOpsAI/LLMStudio https://github.com/TensorOpsAI/LLMStudio

nraford 3y ago

How did you get Mixtral to run an a 32gb M1? I tried using Ollama on my machine (same specs as above) and it told me I needed 49gb RAM minimum.

eurekin 3y ago
I'm using: ollama run dolphin-mixtral:8x7b-v2.5-q3_K_S
- mark_l_watson 3y ago
  That runs on 32G? The original mixtral q3 wouldn’t run for me. Maybe the dolphin tuned version is smaller? EDIT: I just checked, it runs great, thanks.

barnabee 3y ago

I have so far run it on my M1 MacBook using llamafile [1] and found it to be great. Is there any speed/performance/quality/context size/etc. advantage to using LLM Studio or any of the other *llama tools that require more setup than downloading and running a single llamafile executable? [1] https://github.com/Mozilla-Ocho/llamafile/ https://github.com/Mozilla-Ocho/llamafile/

Me1000 3y ago

LM Studio, sadly, is not open source.