3 ms·
I have an M4 pro (48 GB ram) and I run Gemma 4 26b a4b at 52 tok/s and Qwen 3.5b a3b at 72 tok/s. Both 4bit quantized. These are enough for my needs and the per
by argee 24d ago
I have an M4 pro (48 GB ram) and I run Gemma 4 26b a4b at 52 tok/s and Qwen 3.5b a3b at 72 tok/s. Both 4bit quantized. These are enough for my needs and the performance is more than good enough. I'm not running the MLX version of the Gemma model, if I did the inference speed would likely be a bit better. I wouldn't use them for coding features though.
- dolebirchwood 24d ago> enough for my needs Which are...?
- argee 24d agoSome examples (keep in mind this is all indefinitely free for me, no burning quota away): 1. Getting information (such as information about hardware unfamiliar to me) when not connected to the internet, which happens occasionally in my case. 2. Continuing to learn Rust by way of toy examples, puzzles, and comparing aspects of various solutions, for example from LeetCode. 3. Reformatting data, for example from a PDF to a markdown table, or converting receipt images to text. 4. Simple translation/explanation (e.g. I'm teaching my wife one of the languages I speak but sometimes may not know/have the words to explain the full nuance of a translated word). 5. Summarization. One of the webnovels I'm reading has some very boring parts I don't want to slog through, in those cases I simply make the LLM summarize that part and move on. Etc., you get the idea. It's not unusable for coding, but it would make many mistakes when making a whole feature and the context lengths are limited to around 30k-40k tokens by my RAM. I could give it access to the web but I simply use an online model when I need that sort of thing, again partly due to the context limit. Edit: The MLX version of Gemma 4 26b a4b does about 62 tok/s.
- phazonoverload 24d agoMy perf sucks compared to yours. Added it to the post - same model averages 325 tok/s in processing prompts, and 34 tok/s in token generation. What am I doing wrong..?
- argee 24d agoWow, that's just about half the perf. I'm not sure what you're doing differently, though our hardware is a bit different: I am on a Macbook Pro M4 Pro, while you're on a Mac Mini. I would try a different version of the model from HuggingFace while ensuring it's MLX. I'm also using LM Studio, not oMLX, and I've seen some threads like these: https://www.reddit.com/r/LocalLLaMA/comments/1spuwir/omlx_10ts_slowlier_than_lm_studio_qwen36_35ba3_on/ https://www.reddit.com/r/LocalLLaMA/comments/1spuwir/omlx_10...
- dd8601fn 24d agoI have a (now discontinued) 64gb mini pro and I’ve found the same qwen model to be almost unusable unless I kill Thinking on each turn. What are you using them with/for?
- argee 24d agoI do turn thinking off most of the time for both models. I made a separate comment detailing my use cases.
- pwthornton 23d agoWhat kinds of tasks are you using this for?