4 ms·

Whats the best model that can work on a Mac Studio?

akmittal 3y ago
Llama or mistral both are good. Running using ollama
LeoPanthera 3y ago
RAM is the limiting factor, but Mixtral 8x7 is probably the state of the art for self-hosted LLMs right now. If you are low on RAM you can run a quantized version, at the expense of reduced quality.
resource_waste 3y ago
IMO Berkley Sterling which is a fine-tuned llama. I'm not really sure what you are expecting to do with CPU. You might be able to get some <400 token responses and have fun, but you aren't going to be doing 2000 token encyclopedia style responses unless you are going to wait ~20 minutes for a response. Alternatively, you can get like a $800 gaming laptop that has 8gb vram or use something like vastai where you can get 12gb vram for $0.10 an hour.
- dartos 3y ago
  Newer Apple computers have unified gpu and cpu memory, so they are BLAZINGLY fast with local models.
  - resource_waste 3y ago
    This is just Apple marketing, integrated GPUs into CPUs are still CPUs. There is a reason no one ever shows >2000 token reponses.
    - acchow 3y ago
      It isn't that bad. One user with an M2 studio: > For a 34b q8 sending in 6000 context (out of a total of 16384) I get about 4 tokens per second. https://www.reddit.com/r/LocalLLaMA/comments/18b1qgy/comment/kc27t0o/?utm_source=reddit&utm_medium=web2x&context=3 https://www.reddit.com/r/LocalLLaMA/comments/18b1qgy/comment... You can't even fit this on 2x RTX 4090.
      - resource_waste 3y ago
        But why? Like, use the 7B berkley sterling on a freaking 3060 laptop and you are still getting better results on both output and tokens. I don't understand what they are getting out of these large models that perform worse.
- acchow 3y ago
  What is the use case of buying a $800 gaming laptop to run a smaller model instead of using GPT 3.5?
  - thoughtpeddler 3y ago
    One word: privacy
woadwarrior01 3y ago
You can run 4-bit quantized Mixtral 8x7B with unquantized MoE gates, if you've got at least 32GB of RAM. The model itself takes up about 24GB of RAM.
- yawnxyz 3y ago
  I have a 32GB M3 and... Mixtral 8x7B likes to completely crash the machine haha
  - woadwarrior01 3y ago
    What software are you using for inference? I hate plugging my own app[1] here, but I know many people on my app's discord that are running 4-bit OmniQuant quantized Mixtral 8x7B on it on >= 32GB M1, M2 and M3 Macs. I run it all the time 64GB M2 Mac Studio and it takes up just under 24GB of RAM. Also runs Yi-34B-Chat, which takes up ~18.15GB of RAM. [1]: https://privatellm.app https://privatellm.app
mark_l_watson 3y ago
I have a M2 Pro 32B memory Mac mini, and I can run mixtral-7-8-Q3 - that is with 3 bit quantization.
- woadwarrior01 3y ago
  IIRC, you're mentioned once before that you've used Private LLM. :) Please try the 4-bit OmniQuant quantized Mixtral 8x7B Instruct model in it. It runs circles around RTN Q3 models at speed and RTN Q8 models at text generation quality.

Llama or mistral both are good. Running using ollama

RAM is the limiting factor, but Mixtral 8x7 is probably the state of the art for self-hosted LLMs right now. If you are low on RAM you can run a quantized version, at the expense of reduced quality.

resource_waste 3y ago

IMO Berkley Sterling which is a fine-tuned llama. I'm not really sure what you are expecting to do with CPU. You might be able to get some <400 token responses and have fun, but you aren't going to be doing 2000 token encyclopedia style responses unless you are going to wait ~20 minutes for a response. Alternatively, you can get like a $800 gaming laptop that has 8gb vram or use something like vastai where you can get 12gb vram for $0.10 an hour.

dartos 3y ago
Newer Apple computers have unified gpu and cpu memory, so they are BLAZINGLY fast with local models.
- resource_waste 3y ago
  This is just Apple marketing, integrated GPUs into CPUs are still CPUs. There is a reason no one ever shows >2000 token reponses.
  - acchow 3y ago
    It isn't that bad. One user with an M2 studio: > For a 34b q8 sending in 6000 context (out of a total of 16384) I get about 4 tokens per second. https://www.reddit.com/r/LocalLLaMA/comments/18b1qgy/comment/kc27t0o/?utm_source=reddit&utm_medium=web2x&context=3 https://www.reddit.com/r/LocalLLaMA/comments/18b1qgy/comment... You can't even fit this on 2x RTX 4090.
    - resource_waste 3y ago
      But why? Like, use the 7B berkley sterling on a freaking 3060 laptop and you are still getting better results on both output and tokens. I don't understand what they are getting out of these large models that perform worse.
acchow 3y ago
What is the use case of buying a $800 gaming laptop to run a smaller model instead of using GPT 3.5?
- thoughtpeddler 3y ago
  One word: privacy

woadwarrior01 3y ago

You can run 4-bit quantized Mixtral 8x7B with unquantized MoE gates, if you've got at least 32GB of RAM. The model itself takes up about 24GB of RAM.

yawnxyz 3y ago
I have a 32GB M3 and... Mixtral 8x7B likes to completely crash the machine haha
- woadwarrior01 3y ago
  What software are you using for inference? I hate plugging my own app[1] here, but I know many people on my app's discord that are running 4-bit OmniQuant quantized Mixtral 8x7B on it on >= 32GB M1, M2 and M3 Macs. I run it all the time 64GB M2 Mac Studio and it takes up just under 24GB of RAM. Also runs Yi-34B-Chat, which takes up ~18.15GB of RAM. [1]: https://privatellm.app https://privatellm.app

mark_l_watson 3y ago

I have a M2 Pro 32B memory Mac mini, and I can run mixtral-7-8-Q3 - that is with 3 bit quantization.

woadwarrior01 3y ago
IIRC, you're mentioned once before that you've used Private LLM. :) Please try the 4-bit OmniQuant quantized Mixtral 8x7B Instruct model in it. It runs circles around RTN Q3 models at speed and RTN Q8 models at text generation quality.