3 ms·
Which of these can I run locally on a 64GB Mac Mini Pro? And how much does quantization affect the quality?
by JSR_FDED 1y ago
Which of these can I run locally on a 64GB Mac Mini Pro? And how much does quantization affect the quality?
- simonw 1y agoI use a 64GB M2 MacBook Pro. I tend to find any model that's smaller than 32B works well (I can just about run a 70B but it's not worth it as I have to quit all other apps first). My current favorite to run on my machine is OpenAI's gpt-oss-20b because it only uses 11GB of RAM and it's designed to run at that quantization size. I also really like playing with the Qwen 3 family at various sizes and I'm fond of Mistral Small 3.2 as a vision LLM that works well.
- JSR_FDED 1y agoThanks. Do you get any value from those for coding?
- simonw 1y agoOnly when I'm offline (on planes for example) - I've had both Mistral Small and gpt-oss-20b be useful for Python and JavaScript stuff. If I have an internet connection I'll use GPT-5 or Claude 4 or Gemini 2.5 instead - they're better and they don't need me to dedicate a quarter of my RAM or run down my battery.
- mettamage 1y agoUseful info! I have an M1 Mac with 64 GB and haven't experimented with offline models recently. I'll come back to this when I need my AI Maccie snackie, haha. Offline Apple Intelligence isn't at the level I want it, yet.