7 ms·
I think gemma-3-27b-it-qat-4bit is my new favorite local model - or at least it's right up there with Mistral Small 3.1 24B. I've been trying it on an M2 64GB
by simonw 2y ago
I think gemma-3-27b-it-qat-4bit is my new favorite local model - or at least it's right up there with Mistral Small 3.1 24B.
I've been trying it on an M2 64GB via both Ollama and MLX. It's very, very good, and it only uses ~22Gb (via Ollama) or ~15GB (MLX) leaving plenty of memory for running other apps.
Some notes here: https://simonwillison.net/2025/Apr/19/gemma-3-qat-models/ https://simonwillison.net/2025/Apr/19/gemma-3-qat-models/
Last night I had it write me a complete plugin for my LLM tool like this:
llm install llm-mlx
llm mlx download-model mlx-community/gemma-3-27b-it-qat-4bit
llm -m mlx-community/gemma-3-27b-it-qat-4bit \
-f https://raw.githubusercontent.com/simonw/llm-hacker-news/refs/heads/main/llm_hacker_news.py \
-f https://raw.githubusercontent.com/simonw/tools/refs/heads/main/github-issue-to-markdown.html \
-s 'Write a new fragments plugin in Python that registers
issue:org/repo/123 which fetches that issue
number from the specified github repo and uses the same
markdown logic as the HTML page to turn that into a
fragment'
It gave a solid response! https://gist.github.com/simonw/feccff6ce3254556b848c27333f52543#response https://gist.github.com/simonw/feccff6ce3254556b848c27333f52... - more notes here: https://simonwillison.net/2025/Apr/20/llm-fragments-github/ https://simonwillison.net/2025/Apr/20/llm-fragments-github/
- littlestymaar 2y ago> and it only uses ~22Gb (via Ollama) or ~15GB (MLX) Why is the memory use different? Are you using different context size in both set-ups?
- simonw 2y agoNo idea. MLX is its own thing, optimized for Apple Silicon. Ollama uses GGUFs. https://ollama.com/library/gemma3:27b-it-qat https://ollama.com/library/gemma3:27b-it-qat says it's Q4_0. https://huggingface.co/mlx-community/gemma-3-27b-it-qat-4bit https://huggingface.co/mlx-community/gemma-3-27b-it-qat-4bit says it's 4bit. I think those are the same quantization?
- jychang 2y agoThose are the same quant, but this is a good example of why you shouldn't use ollama. Either directly use llama.cpp, or use something like LM Studio if you want something with a GUI/easier user experience. The Gemma 3 17b QAT GGUF should be taking up ~15gb, not 22gb.
- Patrick_Devine 2y agoThe vision tower is 7GB, so I was wondering if you were loading it without vision?
- aukejw 2y ago[dead]
- rs186 2y agoCan you quote tps? More and more I start to realize that cost saving is a small problem for local LLMs. If it is too slow, it becomes unusable, so much that you might as well use public LLM endpoints. Unless you really care about getting things done locally without sending information to another server. With OpenAI API/ChatGPT, I get response much faster than I can read, and for simple question, it means I just need a glimpse of the response, copy & paste and get things done. Whereas on local LLM, I watch it painstakingly prints preambles that I don't care about, and get what I actually need after 20 seconds (on a fast GPU). And I am not yet talking about context window etc. I have been researching about how people integrate local LLMs in their workflows. My finding is that most people play with it for a short time and that's about it, and most people are much better off spending money on OpenAI credits (which can last a very long time with typical usage) than getting a beefed up Mac Studio or building a machine with 4090.
- simonw 2y agoMy tooling doesn't measure TPS yet. It feels snappy to me on MLX. I agree that hosted models are usually a better option for most people - much faster, higher quality, handle longer inputs, really cheap. I enjoy local models for research and for the occasional offline scenario. I'm also interested in their applications for journalism, specifically for dealing with extremely sensitive data like leaked information from confidential sources.
- freeamz 2y ago>I'm also interested in their applications for journalism, specifically for dealing with extremely sensitive data like leaked information from confidential sources. Think it is NOT just you. Most company with decent management also would not want their data going to anything outside the physical server they have in control of. But yeah for most people just use an app and hosted server. But this is HN,there are ppl here hosting their own email servers, so shouldn't be too hard to run llm locally.
- simonw 2y ago"Most company with decent management also would not want their data going to anything outside the physical server they have in control of." I don't think that's been true for over a decade: AWS wouldn't be trillion dollar business if most companies still wanted to stay on-premise.
- nico 2y agoBeen super impressed with local models on mac. Love that the gemma models have 128k token context input size. However, outputs are usually pretty short Any tips on generating long output? Like multiple pages of a document, a story, a play or even a book?
- simonw 2y agoThe tool you are using may set a default max output size without you realizing. Ollama has a num_ctx that defaults to 2048 for example: https://github.com/ollama/ollama/blob/main/docs/faq.md#how-can-i-specify-the-context-window-size https://github.com/ollama/ollama/blob/main/docs/faq.md#how-c...
- nico 2y agoBeen playing with that, but doesn’t seem to have much effect. It works very well to limit output to smaller bits, like setting it to 100-200. But above 2-4k the output seems to never get longer than about 1 page Might try using the models with mlx instead of ollama to see if that makes a difference Any tips on prompting to get longer outputs? Also, does the model context size determine max output size? Are the two related or are they independent characteristics of the model?
- simonw 2y agoInterestingly the Gemma 3 docs say: https://ai.google.dev/gemma/docs/core/model_card_3#:~:text=Total%20output%20context%20up%20to,subtracting%20the%20request%20input%20tokens https://ai.google.dev/gemma/docs/core/model_card_3#:~:text=T... > Total output context up to 128K tokens for the 4B, 12B, and 27B sizes, and 32K tokens for the 1B size per request, subtracting the request input tokens I don't know how to get it to output anything that length though.
- nico 2y agoThank you for the insights and useful links Will keep experimenting, will also try mistral3.1 edit: just tried mistral3.1 and the quality of the output is very good, at least compared to the other models I tried (llama2:7b-chat, llama2:latest, gemma3:12b, qwq and deepseek-r1:14b) Doing some research, because of their training sets, it seems like most models are not trained on producing long outputs so even if they technically could, they won’t. Might require developing my own training dataset and then doing some fine tuning. Apparently the models and ollama have some safeguards against rambling and repetition
- tomrod 2y agoSimon, what is your local GPU setup? (No doubt you've covered this, but I'm not sure where to dig up).
- simonw 2y agoMacBook Pro M2 with 64GB of RAM. That's why I tend to be limited to Ollama and MLX - stuff that requires NVIDIA doesn't work for me locally.
- Elucalidavah 2y ago> MacBook Pro M2 with 64GB of RAM Are there non-mac options with similar capabilities?
- simonw 2y agoYes, but I don't really know anything about those. https://www.reddit.com/r/LocalLLaMA/ https://www.reddit.com/r/LocalLLaMA/ is full of people running models on PCs with NVIDIA cards. The unique benefit of an Apple Silicon Mac at the moment is that the 64GB of RAM is available to both the GPU and the CPU at once. With other hardware you usually need dedicated separate VRAM for the GPU.
- _neil 2y agoIt’s not out yet, but the upcoming Framework desktop [0] is supposed to have a similar unified memory setup. [0] https://frame.work/desktop https://frame.work/desktop
- dwood_dev 2y agoAnything with the Radeon 8060S/Ryzen AI Max+ 395. One of the popular MiniPC Chinese brands has them for preorder[0] with shipping starting May 7th. Framework also has them, but shipping Q3. 0: https://www.gmktec.com/products/prepaid-deposit-amd-ryzen™-ai-max-395-evo-x2-ai-mini-pc https://www.gmktec.com/products/prepaid-deposit-amd-ryzen™-a...
- 2y ago
- paprots 2y agoThe original gemma3:27b also took only 22GB using Ollama on my 64GB MacBook. I'm quite confused that the QAT took the same. Do you know why? Which model is better? `gemma3:27b`, or `gemma3:27b-qat`?
- nolist_policy 2y agoI suspect your "original gemma3:27b" was a quantized model since the non-quantized (16bit) version needs around 54gb.
- kgwgk 2y agoLook up 27b in https://ollama.com/library/gemma3/tags https://ollama.com/library/gemma3/tags You'll find the id a418f5838eaf which also corresponds to 27b-it-q4_K_M
- carbocation 2y agoJust following this comment up as a note-to-self: just as `kgwgk noted, the default gemma3:27B model has ID a418f5838eaf, which corresponds to 27b-it-q4_K_M. But the new gemma3:27B quantization-aware training (QAT) model being discussed is gemma3:27b-it-qat with ID 29eb0b9aeda3.
- superkuh 2y agoQuantization aware training just means having the model deal with quantized values a bit during training so it handles the quantization better when it is quantized after training/etc. It doesn't change the model size itself.
- zorgmonkey 2y agoBoth versions are quantized and should use the same amount of RAM, the difference with QAT is the quantization happens during training time and it should result in slightly better (closer to the bf16 weights) output
- prvc 2y ago> ~15GB (MLX) leaving plenty of memory for running other apps. Is that small enough to run well (without thrashing) on a system with only 16GiB RAM?
- simonw 2y agoI expect not. On my Mac at least I've found I need a bunch of GB free to have anything else running at all.
- mnoronha 2y agoAny idea why MLX and ollama use such different amounts of ram?
- jychang 2y agoI don't think ollama is quantizing the embeddings table, which is still full FP16. If you're using MLX, that means you're on a mac, in which case ollama actually isn't your best option. Either directly use llama.cpp if you're a power user, or use LM Studio if you want something a bit better than ollama but more user friendly than llama.cpp. (LM Studio has a GUI and is also more user friendly than ollama, but has the downsides of not being as scriptable. You win some, you lose some.) Don't use MLX, it's not as fast/small as the best GGUFs currently (and also tends to be more buggy, it currently has some known bugs with japanese). Download the LM Studio version of the Gemma 3 QAT GGUF quants, which are made by Bartowski. Google actually directly mentions Bartowski in blog post linked above (ctrl-f his name), and his models are currently the best ones to use. https://huggingface.co/bartowski/google_gemma-3-27b-it-qat-GGUF/blob/main/google_gemma-3-27b-it-qat-Q4_0.gguf https://huggingface.co/bartowski/google_gemma-3-27b-it-qat-G... The "best Gemma 3 27b model to download" crown has taken a very roundabout path. After the initial Google release, it went from Unsloth Q4_K_M, to Google QAT Q4_0, to stduhpf Q4_0_S, to Bartowski Q4_0 now.
- codybontecou 2y agoCan you run the mlx-variation of this model through Ollama so that I can interact with it in Open WebUI?
- simonw 2y agoI haven't tried it yet but there's an MLX project that exposes an OpenAI-compatible serving endpoint that should work with Open WebUI: https://github.com/madroidmaq/mlx-omni-server https://github.com/madroidmaq/mlx-omni-server
- codybontecou 2y agoAppreciate the link. I'll try to tinker with it later today.
- deleted 2y ago[deleted]
- bobjordan 2y agoThanks for the call out on this model! I have 42gb usable VRAM on my ancient (~10yrs old) quad-sli titan-x workstation and have been looking for a model to balance large context window with output quality. I'm able to run this model with a 56K context window and it just fits into my 42gb VRAM to run 100% GPU. The output quality is really good and 56K context window is very usable. Nice find!
- ygreif 2y agoDo many consumer GPUs have >20 gigabytes RAM? That sounds like a lot to me
- mcintyre1994 2y agoI don't think so, but Apple's unified memory architecture makes it a possibility for people with Macbook Pros.