4 ms·

Simon, can you recommend some small models that would be usable for coding on a standard M4 Mac Mini (only 16G ram) ?

simonw 1y ago
That's pretty tough - the problem is that you need to have RAM left over to run actual applications! Qwen 3 8B on MLX runs in just 5GB of RAM and can write basic code but I don't know if it would be good enough for anything interesting: https://simonwillison.net/2025/May/2/qwen3-8b/ https://simonwillison.net/2025/May/2/qwen3-8b/ Honestly though with that little memory I'd stick to running against hosted LLMs - Claude 3.7 Sonnet, Gemini 2.5 Pro, o4-mini are all cheap enough that it's hard to spend much money with them for most coding workflows.
- codetrotter 1y ago
  How about on an MacBook Pro M2 Max with 64GB RAM? Any recommendations for local models for coding on that? I tried to run some of the differently sized DeepSeek R1 locally when those had recently come out, but couldn’t manage at the time to run any of them. And I had to download a lot of data to try those. So if you know a specific size of DeepSeek R1 that will work on 64GB RAM on MacBook Pro M2 Max, or another great local LLM for coding on that, that would be super appreciated
  - freeqaz 1y ago
    I imagine that this in quantized form would fit pretty well and be decent. (Qwen R1 32b[1] or Qwen 3 32b[2]) Specifically the `Q6_K` quant looks solid at ~27gb. That leaves enough headroom on your 64gb Macbook that you can actually load a decent amount of context. (It takes extra VRAM for every token of context you need) Rough math, based on this[0] calculator is that it's around ~10gb per 32k tokens of context. And that doesn't seem to change based on using a different quant size -- you just have to have enough headroom. So with 64gb: - ~25gb for Q6 quant - 10-20gb for context of 32-64k That leaves you around 20gb for application memory and _probably_ enough context to actually be useful for larger coding tasks! (It just might be slow, but you can use a smaller quant to get more speed.) I hope that helps! 0: https://huggingface.co/spaces/NyxKrage/LLM-Model-VRAM-Calculator https://huggingface.co/spaces/NyxKrage/LLM-Model-VRAM-Calcul... 1: https://huggingface.co/bartowski/DeepSeek-R1-Distill-Qwen-32B-GGUF https://huggingface.co/bartowski/DeepSeek-R1-Distill-Qwen-32... 2: https://huggingface.co/Qwen/Qwen3-32B-GGUF https://huggingface.co/Qwen/Qwen3-32B-GGUF
  - simonw 1y ago
    I really like Mistral Small 3.1 (I have a 64GB M2 as well). Qwen 3 is worth trying in different sizes too. I don't know if they'll be good enough for general coding tasks though - I've been spoiled by API access to Claude 3.7 Sonnet and o4-mini and Gemini 2.5 Pro.
    - aukejw 1y ago
      How do you determine peak memory usage? Just look at activity monitor? I've yet to find a good overview of how much memory each model needs for different context lengths (other than back of the envelope #weights * bits). LM Studio warns you if a model will likely not fit, but it's not very exact.
      - simonw 1y ago
        MLX reports peak memory usage at the end of the response. Otherwise I'll use Activity Monitor.
        aukejw 1y ago
        I'm also trusting `get_peak_memory` + some small buffer for now. Still, it reports accurate peak memory usage for tensors living on GPU, but seems to miss some of the non-Metal overhead, however small (https://github.com/aukejw/mlx_transformers_benchmark/issues/1 https://github.com/aukejw/mlx_transformers_benchmark/issues/...).
  - aukejw 1y ago
    There are plenty of smaller (quantized) models that fit well on your machine! On a M4 with 24GB it’s already possible to comfortably run 8B quantized models. Im benchmarking runtime and memory usage for a few of them: https://aukejw.github.io/mlx_transformers_benchmark/ https://aukejw.github.io/mlx_transformers_benchmark/
reichardt 1y ago
With around 4.6 GiB model size the new Qwen3-8B quantized to 4-bit should fit comfortably in 16 GiB of memory: https://huggingface.co/mlx-community/Qwen3-8B-4bit https://huggingface.co/mlx-community/Qwen3-8B-4bit
jychang 1y ago
16GB on a mac with unified memory is too small for good coding models. Anything on that machine is severely compromised. Maybe in ~1 year we will see better models that fit in ~8gb vram, but not yet. Right now, for a coding LLM on a Mac, the standard is Qwen 3 32b, which runs great on any M1 mac with 32gb memory or better. Qwen 3 235b is better, but fewer people have 128gb memory. Anything smaller than 32b, you start seeing a big drop off in quality. Qwen 3 14b Q4_K_M is probably your best option at 16gb memory, but it's significantly worse in quality than 32b.
- chedabob 1y ago
  What do you use to interface with Qwen? I have LMStudio installed, and use Continue in VSCode, but it doesn't feel nearly as feature rich compared to using something like Cursor's IDE, or the GitHub Copilot plugin.
  - Lalabadie 1y ago
    Continue can be your autocomplete provider – and use a smaller and faster model. Something like Cline (or Roo or Kilocode or another fork) would be the more Cursor-like assistant there.
martin_a 1y ago
Strange idea, but if I'd like to set up a solid LLM for use in my home network, how much processing power would I need for a multi-purpose model? A Raspberry Pi? And old ThinkPad? A fully speced-out latest gen Macbook? edit: One of those old Mac Pros?
- wsintra2022 1y ago
  That’s what I tried initially, an old black tin can Mac Pro, but it couldn’t do it. Next splashed on an m2 ultra 64gb mpro, runs ollama with qwen3 32b - reverse shell into the localhost with open web-ui and automatic111 and voila AI on my home network
  - martin_a 1y ago
    Hm, that seems like a lot of power use. I thought I could get away with somewhat less.
the_other_mac 1y ago
Run Mistral 7b in under 4gb ram: https://github.com/garagesteve1155/Overload https://github.com/garagesteve1155/Overload (As announced this morning in the FB group "Dull Men's Club!)

That's pretty tough - the problem is that you need to have RAM left over to run actual applications! Qwen 3 8B on MLX runs in just 5GB of RAM and can write basic code but I don't know if it would be good enough for anything interesting: https://simonwillison.net/2025/May/2/qwen3-8b/ https://simonwillison.net/2025/May/2/qwen3-8b/ Honestly though with that little memory I'd stick to running against hosted LLMs - Claude 3.7 Sonnet, Gemini 2.5 Pro, o4-mini are all cheap enough that it's hard to spend much money with them for most coding workflows.

codetrotter 1y ago
How about on an MacBook Pro M2 Max with 64GB RAM? Any recommendations for local models for coding on that? I tried to run some of the differently sized DeepSeek R1 locally when those had recently come out, but couldn’t manage at the time to run any of them. And I had to download a lot of data to try those. So if you know a specific size of DeepSeek R1 that will work on 64GB RAM on MacBook Pro M2 Max, or another great local LLM for coding on that, that would be super appreciated
- freeqaz 1y ago
  I imagine that this in quantized form would fit pretty well and be decent. (Qwen R1 32b[1] or Qwen 3 32b[2]) Specifically the `Q6_K` quant looks solid at ~27gb. That leaves enough headroom on your 64gb Macbook that you can actually load a decent amount of context. (It takes extra VRAM for every token of context you need) Rough math, based on this[0] calculator is that it's around ~10gb per 32k tokens of context. And that doesn't seem to change based on using a different quant size -- you just have to have enough headroom. So with 64gb: - ~25gb for Q6 quant - 10-20gb for context of 32-64k That leaves you around 20gb for application memory and _probably_ enough context to actually be useful for larger coding tasks! (It just might be slow, but you can use a smaller quant to get more speed.) I hope that helps! 0: https://huggingface.co/spaces/NyxKrage/LLM-Model-VRAM-Calculator https://huggingface.co/spaces/NyxKrage/LLM-Model-VRAM-Calcul... 1: https://huggingface.co/bartowski/DeepSeek-R1-Distill-Qwen-32B-GGUF https://huggingface.co/bartowski/DeepSeek-R1-Distill-Qwen-32... 2: https://huggingface.co/Qwen/Qwen3-32B-GGUF https://huggingface.co/Qwen/Qwen3-32B-GGUF
- simonw 1y ago
  I really like Mistral Small 3.1 (I have a 64GB M2 as well). Qwen 3 is worth trying in different sizes too. I don't know if they'll be good enough for general coding tasks though - I've been spoiled by API access to Claude 3.7 Sonnet and o4-mini and Gemini 2.5 Pro.
  - aukejw 1y ago
    How do you determine peak memory usage? Just look at activity monitor? I've yet to find a good overview of how much memory each model needs for different context lengths (other than back of the envelope #weights * bits). LM Studio warns you if a model will likely not fit, but it's not very exact.
    - simonw 1y ago
      MLX reports peak memory usage at the end of the response. Otherwise I'll use Activity Monitor.
      - aukejw 1y ago
        I'm also trusting `get_peak_memory` + some small buffer for now. Still, it reports accurate peak memory usage for tensors living on GPU, but seems to miss some of the non-Metal overhead, however small (https://github.com/aukejw/mlx_transformers_benchmark/issues/1 https://github.com/aukejw/mlx_transformers_benchmark/issues/...).
- aukejw 1y ago
  There are plenty of smaller (quantized) models that fit well on your machine! On a M4 with 24GB it’s already possible to comfortably run 8B quantized models. Im benchmarking runtime and memory usage for a few of them: https://aukejw.github.io/mlx_transformers_benchmark/ https://aukejw.github.io/mlx_transformers_benchmark/

reichardt 1y ago

With around 4.6 GiB model size the new Qwen3-8B quantized to 4-bit should fit comfortably in 16 GiB of memory: https://huggingface.co/mlx-community/Qwen3-8B-4bit https://huggingface.co/mlx-community/Qwen3-8B-4bit

jychang 1y ago

16GB on a mac with unified memory is too small for good coding models. Anything on that machine is severely compromised. Maybe in ~1 year we will see better models that fit in ~8gb vram, but not yet. Right now, for a coding LLM on a Mac, the standard is Qwen 3 32b, which runs great on any M1 mac with 32gb memory or better. Qwen 3 235b is better, but fewer people have 128gb memory. Anything smaller than 32b, you start seeing a big drop off in quality. Qwen 3 14b Q4_K_M is probably your best option at 16gb memory, but it's significantly worse in quality than 32b.

chedabob 1y ago
What do you use to interface with Qwen? I have LMStudio installed, and use Continue in VSCode, but it doesn't feel nearly as feature rich compared to using something like Cursor's IDE, or the GitHub Copilot plugin.
- Lalabadie 1y ago
  Continue can be your autocomplete provider – and use a smaller and faster model. Something like Cline (or Roo or Kilocode or another fork) would be the more Cursor-like assistant there.

martin_a 1y ago

Strange idea, but if I'd like to set up a solid LLM for use in my home network, how much processing power would I need for a multi-purpose model? A Raspberry Pi? And old ThinkPad? A fully speced-out latest gen Macbook? edit: One of those old Mac Pros?

wsintra2022 1y ago
That’s what I tried initially, an old black tin can Mac Pro, but it couldn’t do it. Next splashed on an m2 ultra 64gb mpro, runs ollama with qwen3 32b - reverse shell into the localhost with open web-ui and automatic111 and voila AI on my home network
- martin_a 1y ago
  Hm, that seems like a lot of power use. I thought I could get away with somewhat less.

the_other_mac 1y ago

Run Mistral 7b in under 4gb ram: https://github.com/garagesteve1155/Overload https://github.com/garagesteve1155/Overload (As announced this morning in the FB group "Dull Men's Club!)