3 ms·

> the newer Apple Silicon Macs with unified memory Mixtral has been MLXd already? Write ups, if any?

LoganDark 3y ago
Not to my knowledge. But because the unified memory doubles as VRAM for the onboard GPU, normal GPU acceleration can access the entire model even if it's 50+ GB. That's why ASi Macs are currently the holy grail for at-home inferencing, and also why projects like llama.cpp focus so much on ASi above all else, and why so many UIs release for macOS first before other operating systems. Certain Mac models offer up to 192GB of unified memory.
- mkesper 3y ago
  But that's not a Macbook. And a Macbook M3Max with 128GB of RAM is almost 8000€.
  - zarzavat 3y ago
    Considering how inaccessible and expensive 128GB of pro-level cards is, that is believe it or not, a good price.
    - LoganDark 3y ago
      Not to mention that's 128GB for a single GPU. No need to shard or split.
vessenes 3y ago
Yes it has, actually: https://github.com/ml-explore/mlx-examples https://github.com/ml-explore/mlx-examples. It's right in the main repo. NB, I haven't tried this, I'm using llama.cpp with a non-K-quant quantization on my MBP.
- summarity 3y ago
  I have and don't consider MLX to be production ready. I've tested it on M1Max and M1Ultra (128) machines. It's completely non-deterministic in its resource consumption, sometimes using the GPU fully, sometimes getting seemingly stuck while processing, sometimes the GPU throttles. However, there's one curious thing: llama.cpp _always_ leads to GPU throttling on Apple Silicon (e.g. the M1Max GPU will go from 1200MHz to around 700MHz), and then fully saturates it. In the rare cases I could get MLX to stay on the GPU, it was able to keep it at the maximum clock rate. However the unpredictable pauses and seemingly unoptimized prompt processing makes it hard to pick a winner in end-to-end tokens/s
Terretta 3y ago
Many options for running Mistral models in your terminal using LLM: https://simonwillison.net/2023/Dec/18/mistral/ https://simonwillison.net/2023/Dec/18/mistral/ I liked "Using Llamafile’s OpenAI API endpoint" described there, using Justine Tunney's llamafiles for Mixtral, but the article link is out of date, as the models have been replaced with newer: https://huggingface.co/jartine https://huggingface.co/jartine
reexpressionist 3y ago
[dead]
- teilo 3y ago
  I'm running it on an M2 Max with 96GB, and have plenty of room to spare. And it's fast. Faster than I can get responses from ChatGPT.
  - coder543 3y ago
    How many tokens/s? Which quantization? If you could test Q4KM and Q3KM, it would be interesting to hear how the M2 Max does!
    - teilo 3y ago
      No quantization (8_0). The full 48GB model. As for token count, I haven't tested it on more than 200 or so.
      - pilotneko 3y ago
        Isn’t 8_0 8-bit quantization?
        teilo 3y ago
        Sorry. That was a major brain fart. Yes. 8-bit quantization, and using 49G of RAM.

Not to my knowledge. But because the unified memory doubles as VRAM for the onboard GPU, normal GPU acceleration can access the entire model even if it's 50+ GB. That's why ASi Macs are currently the holy grail for at-home inferencing, and also why projects like llama.cpp focus so much on ASi above all else, and why so many UIs release for macOS first before other operating systems. Certain Mac models offer up to 192GB of unified memory.

mkesper 3y ago
But that's not a Macbook. And a Macbook M3Max with 128GB of RAM is almost 8000€.
- zarzavat 3y ago
  Considering how inaccessible and expensive 128GB of pro-level cards is, that is believe it or not, a good price.
  - LoganDark 3y ago
    Not to mention that's 128GB for a single GPU. No need to shard or split.

vessenes 3y ago

Yes it has, actually: https://github.com/ml-explore/mlx-examples https://github.com/ml-explore/mlx-examples. It's right in the main repo. NB, I haven't tried this, I'm using llama.cpp with a non-K-quant quantization on my MBP.

summarity 3y ago
I have and don't consider MLX to be production ready. I've tested it on M1Max and M1Ultra (128) machines. It's completely non-deterministic in its resource consumption, sometimes using the GPU fully, sometimes getting seemingly stuck while processing, sometimes the GPU throttles. However, there's one curious thing: llama.cpp _always_ leads to GPU throttling on Apple Silicon (e.g. the M1Max GPU will go from 1200MHz to around 700MHz), and then fully saturates it. In the rare cases I could get MLX to stay on the GPU, it was able to keep it at the maximum clock rate. However the unpredictable pauses and seemingly unoptimized prompt processing makes it hard to pick a winner in end-to-end tokens/s

Terretta 3y ago

Many options for running Mistral models in your terminal using LLM: https://simonwillison.net/2023/Dec/18/mistral/ https://simonwillison.net/2023/Dec/18/mistral/ I liked "Using Llamafile’s OpenAI API endpoint" described there, using Justine Tunney's llamafiles for Mixtral, but the article link is out of date, as the models have been replaced with newer: https://huggingface.co/jartine https://huggingface.co/jartine

reexpressionist 3y ago

[dead]

teilo 3y ago
I'm running it on an M2 Max with 96GB, and have plenty of room to spare. And it's fast. Faster than I can get responses from ChatGPT.
- coder543 3y ago
  How many tokens/s? Which quantization? If you could test Q4KM and Q3KM, it would be interesting to hear how the M2 Max does!
  - teilo 3y ago
    No quantization (8_0). The full 48GB model. As for token count, I haven't tested it on more than 200 or so.
    - pilotneko 3y ago
      Isn’t 8_0 8-bit quantization?
      - teilo 3y ago
        Sorry. That was a major brain fart. Yes. 8-bit quantization, and using 49G of RAM.