4 ms·
There are plenty of other ways to run Mistral models on a Mac. I'm a big fan of Mistral Small 3.1. I've run that using both Ollama (easiest) and MLX. Here are
by simonw 2y ago
There are plenty of other ways to run Mistral models on a Mac. I'm a big fan of Mistral Small 3.1.
I've run that using both Ollama (easiest) and MLX. Here are the Ollama models: https://ollama.com/library/mistral-small3.1/tags https://ollama.com/library/mistral-small3.1/tags - the 15GB one works fine.
For MLX https://huggingface.co/mlx-community/Mistral-Small-3.1-24B-Instruct-2503-8bit https://huggingface.co/mlx-community/Mistral-Small-3.1-24B-I... and https://huggingface.co/mlx-community/Mistral-Small-3.1-24B-Instruct-2503-4bit https://huggingface.co/mlx-community/Mistral-Small-3.1-24B-I... should work, I use the 8bit one like this:
llm install llm-mlx
llm mlx download-model mlx-community/Mistral-Small-3.1-Text-24B-Instruct-2503-8bit -a mistral-small-3.1
llm chat -m mistral-small-3.1
The Ollama one supports image inputs too:
llm install llm-ollama
ollama pull mistral-small3.1
llm -m mistral-small3.1 'describe this image' \
-a https://static.simonwillison.net/static/2025/Mpaboundrycdfw-1.png
Output here: https://gist.github.com/simonw/89005e8aa2daef82c53c2c2c62207f6a#response https://gist.github.com/simonw/89005e8aa2daef82c53c2c2c62207...
- indigodaddy 2y agoSimon, can you recommend some small models that would be usable for coding on a standard M4 Mac Mini (only 16G ram) ?
- simonw 2y agoThat's pretty tough - the problem is that you need to have RAM left over to run actual applications! Qwen 3 8B on MLX runs in just 5GB of RAM and can write basic code but I don't know if it would be good enough for anything interesting: https://simonwillison.net/2025/May/2/qwen3-8b/ https://simonwillison.net/2025/May/2/qwen3-8b/ Honestly though with that little memory I'd stick to running against hosted LLMs - Claude 3.7 Sonnet, Gemini 2.5 Pro, o4-mini are all cheap enough that it's hard to spend much money with them for most coding workflows.
- codetrotter 2y agoHow about on an MacBook Pro M2 Max with 64GB RAM? Any recommendations for local models for coding on that? I tried to run some of the differently sized DeepSeek R1 locally when those had recently come out, but couldn’t manage at the time to run any of them. And I had to download a lot of data to try those. So if you know a specific size of DeepSeek R1 that will work on 64GB RAM on MacBook Pro M2 Max, or another great local LLM for coding on that, that would be super appreciated
- freeqaz 2y agoI imagine that this in quantized form would fit pretty well and be decent. (Qwen R1 32b[1] or Qwen 3 32b[2]) Specifically the `Q6_K` quant looks solid at ~27gb. That leaves enough headroom on your 64gb Macbook that you can actually load a decent amount of context. (It takes extra VRAM for every token of context you need) Rough math, based on this[0] calculator is that it's around ~10gb per 32k tokens of context. And that doesn't seem to change based on using a different quant size -- you just have to have enough headroom. So with 64gb: - ~25gb for Q6 quant - 10-20gb for context of 32-64k That leaves you around 20gb for application memory and _probably_ enough context to actually be useful for larger coding tasks! (It just might be slow, but you can use a smaller quant to get more speed.) I hope that helps! 0: https://huggingface.co/spaces/NyxKrage/LLM-Model-VRAM-Calculator https://huggingface.co/spaces/NyxKrage/LLM-Model-VRAM-Calcul... 1: https://huggingface.co/bartowski/DeepSeek-R1-Distill-Qwen-32B-GGUF https://huggingface.co/bartowski/DeepSeek-R1-Distill-Qwen-32... 2: https://huggingface.co/Qwen/Qwen3-32B-GGUF https://huggingface.co/Qwen/Qwen3-32B-GGUF
- simonw 2y agoI really like Mistral Small 3.1 (I have a 64GB M2 as well). Qwen 3 is worth trying in different sizes too. I don't know if they'll be good enough for general coding tasks though - I've been spoiled by API access to Claude 3.7 Sonnet and o4-mini and Gemini 2.5 Pro.
- aukejw 2y agoHow do you determine peak memory usage? Just look at activity monitor? I've yet to find a good overview of how much memory each model needs for different context lengths (other than back of the envelope #weights * bits). LM Studio warns you if a model will likely not fit, but it's not very exact.
- reichardt 2y agoWith around 4.6 GiB model size the new Qwen3-8B quantized to 4-bit should fit comfortably in 16 GiB of memory: https://huggingface.co/mlx-community/Qwen3-8B-4bit https://huggingface.co/mlx-community/Qwen3-8B-4bit
- jychang 2y ago16GB on a mac with unified memory is too small for good coding models. Anything on that machine is severely compromised. Maybe in ~1 year we will see better models that fit in ~8gb vram, but not yet. Right now, for a coding LLM on a Mac, the standard is Qwen 3 32b, which runs great on any M1 mac with 32gb memory or better. Qwen 3 235b is better, but fewer people have 128gb memory. Anything smaller than 32b, you start seeing a big drop off in quality. Qwen 3 14b Q4_K_M is probably your best option at 16gb memory, but it's significantly worse in quality than 32b.
- chedabob 2y agoWhat do you use to interface with Qwen? I have LMStudio installed, and use Continue in VSCode, but it doesn't feel nearly as feature rich compared to using something like Cursor's IDE, or the GitHub Copilot plugin.
- Lalabadie 2y agoContinue can be your autocomplete provider – and use a smaller and faster model. Something like Cline (or Roo or Kilocode or another fork) would be the more Cursor-like assistant there.
- martin_a 2y agoStrange idea, but if I'd like to set up a solid LLM for use in my home network, how much processing power would I need for a multi-purpose model? A Raspberry Pi? And old ThinkPad? A fully speced-out latest gen Macbook? edit: One of those old Mac Pros?
- wsintra2022 2y agoThat’s what I tried initially, an old black tin can Mac Pro, but it couldn’t do it. Next splashed on an m2 ultra 64gb mpro, runs ollama with qwen3 32b - reverse shell into the localhost with open web-ui and automatic111 and voila AI on my home network
- martin_a 2y agoHm, that seems like a lot of power use. I thought I could get away with somewhat less.
- the_other_mac 2y agoRun Mistral 7b in under 4gb ram: https://github.com/garagesteve1155/Overload https://github.com/garagesteve1155/Overload (As announced this morning in the FB group "Dull Men's Club!)