5 ms·

My local model setup on an M4 Pro Mac Mini

336 points by raybb 24d ago

amanzi 24d ago
No mention of the performance of the models? I'm able to load a bunch of different models on my little mini-PC with 16GB RAM, but the performance is terrible. I always wonder what performance people are getting with local models that they find is acceptable?
- argee 24d ago
  I have an M4 pro (48 GB ram) and I run Gemma 4 26b a4b at 52 tok/s and Qwen 3.5b a3b at 72 tok/s. Both 4bit quantized. These are enough for my needs and the performance is more than good enough. I'm not running the MLX version of the Gemma model, if I did the inference speed would likely be a bit better. I wouldn't use them for coding features though.
  - dolebirchwood 24d ago
    > enough for my needs Which are...?
    - argee 24d ago
      Some examples (keep in mind this is all indefinitely free for me, no burning quota away): 1. Getting information (such as information about hardware unfamiliar to me) when not connected to the internet, which happens occasionally in my case. 2. Continuing to learn Rust by way of toy examples, puzzles, and comparing aspects of various solutions, for example from LeetCode. 3. Reformatting data, for example from a PDF to a markdown table, or converting receipt images to text. 4. Simple translation/explanation (e.g. I'm teaching my wife one of the languages I speak but sometimes may not know/have the words to explain the full nuance of a translated word). 5. Summarization. One of the webnovels I'm reading has some very boring parts I don't want to slog through, in those cases I simply make the LLM summarize that part and move on. Etc., you get the idea. It's not unusable for coding, but it would make many mistakes when making a whole feature and the context lengths are limited to around 30k-40k tokens by my RAM. I could give it access to the web but I simply use an online model when I need that sort of thing, again partly due to the context limit. Edit: The MLX version of Gemma 4 26b a4b does about 62 tok/s.
  - phazonoverload 24d ago
    My perf sucks compared to yours. Added it to the post - same model averages 325 tok/s in processing prompts, and 34 tok/s in token generation. What am I doing wrong..?
    - argee 24d ago
      Wow, that's just about half the perf. I'm not sure what you're doing differently, though our hardware is a bit different: I am on a Macbook Pro M4 Pro, while you're on a Mac Mini. I would try a different version of the model from HuggingFace while ensuring it's MLX. I'm also using LM Studio, not oMLX, and I've seen some threads like these: https://www.reddit.com/r/LocalLLaMA/comments/1spuwir/omlx_10ts_slowlier_than_lm_studio_qwen36_35ba3_on/ https://www.reddit.com/r/LocalLLaMA/comments/1spuwir/omlx_10...
  - dd8601fn 24d ago
    I have a (now discontinued) 64gb mini pro and I’ve found the same qwen model to be almost unusable unless I kill Thinking on each turn. What are you using them with/for?
    - argee 24d ago
      I do turn thinking off most of the time for both models. I made a separate comment detailing my use cases.
  - pwthornton 23d ago
    What kinds of tasks are you using this for?
- pcarolan 24d ago
  It’s not. Do it as a hobby or for privacy but for performance just use a frontier model api. You’re paying less than cost for something that would take tens of thousands to set up locally.
  - Gigachad 24d ago
    It does make me wonder how the hosted stuff is so cheap. For pretty much everything else, hosted/rented is more expensive but offers better convenience and flexibility. But for AI, even if you consider the total lifetime cost and are utilizing it heavily. You never break even by buying.
    - api 24d ago
      There are economies of scale but there’s also a data center bubble (probably) so there might be some selling dollars for fifty cents going on.
      - mrngld 24d ago
        Here's the thing that's a little different about data centers; we can tell from Anthropic and OpenAI that they're capacity constrained. Inference demand is there. I notice Cerebras doesn't offer much directly any more, all their capacity is getting completely sucked up by B2B sales. Grok did overbuild, but Anthropic was so desperate for more compute they ate their pride and leased the excess capacity. That means all these data centers are being heavily utilized by actual end user inference demand. Well, some is research on new models, but a lot is actual end user demand. No one has given an explanation of why peoples usage would decline. On top of that, margin on inference appears to be decent. It's model training that's a serious financial burden. And maybe that's where there will be a slowdown, maybe the market doesn't justify spending as much on R&D as it does, but the end demand for inference is there. Does that justify these stock prices? That's a different question. But the housing boom left behind endless rows of empty homes because demand disappeared. The 'dot com' boom left behind thousands of miles of dark fiber that'd been built out well ahead of demand for bandwidth. I can see the stock market having a giant sell off, but I don't see data centers sitting idle in that same fashion.
- hkchad 24d ago
  I run a similar setup to the one he described on similar hardware. I run bifrost and llama swap though (tailscale rocks). My local model usage is for some out of band batch processing one of my personal apps uses. Basically a personalized recommender for media, it curates stuff for me based on a database i've compiled over years, so non-interactive. For that use case, I don't really care that it might take a few minutes to run. It's free. The machine is just sitting there anyway. I have tried using qwen-coder and opencode on my M5 Max 128gb and compared to claude code it's painful. I did setup a workflow where claude plans, qwen executes (unattended overnight, again b/c it's slow) and then claude reviews. I benchmarked this several times and I ended up using MORE tokens with claude because it had to 'fix' all the qwen issues. While the code it produced was 'good enough' the fixes were worth it so I just stick to coding task using API models (codex and claude).
  - deleted 24d ago
    [deleted]
  - brettdav 24d ago
    Can you share a bit more about your bifrost and llama swap setup? I’m facing memory constraints and am looking for a managed model solution that will help with hot swapping loaded models and stay-warm concurrency. Ideally with prioritization.
    - hkchad 24d ago
      What do you want to know? Just start llama-swap with the models i have downloaded, add llama-swap as a provider in bifrost, expose the models you want and they become available in one single endpoint you can use in anything like opencode, openwebui or anything that speaks openai.
      - brettdav 22d ago
        Yes, that makes sense. Some of my models currently run in ollama while others require their own inference servers. I’m curious about custom inference servers in bifrost and the ability to orchestrate keeping some models warm in memory while evicting and cycling other models. All of which span different providers in bifrost (I think). Obviously I can get the mechanics from an agent, more wondering about any experience with something similar.
- ericd 24d ago
  I honestly wouldn’t bother with local models right now unless I either had a 5090 and was happy with running Qwen 3.8 27B, or a pair of DGX Sparks running DSv4 flash, or better, 2x6000 RTX Blackwells. Those are the kinds of rigs that the local model enthusiasts are running. With the GPU setups, you’re looking at generally >100tps generation in single stream, and >10k tps of prefill, so it’s snappier than Claude code, which somewhat makes up for it being dumber. That said, it is really cool to be able to run an LLM on eg a Mac laptop. Just not a better experience on almost any metric for interactive use than eg Claude Code, beside privacy and guardrails.
  - gruez 24d ago
    >I honestly wouldn’t bother with local models right now unless I either had a 5090 and was happy with running Qwen 3.8 27B How's the actual performance of Qwen 3.8 27B? On deepswe it supposedly performs slightly worse than gpt 5.6 luna high[1], but I can't help but think they've been benchmaxxed. [1] https://deepswe.datacurve.ai/ https://deepswe.datacurve.ai/, https://unsloth.ai/docs/models/qwen3.8#benchmarks https://unsloth.ai/docs/models/qwen3.8#benchmarks
    - ericd 24d ago
      Not sure, I haven't run it, I've just been running DS V4 Flash non-stop since it came out, and that's replaced a lot of my Claude Code usage. People seem very impressed, though, it seems like it trades vram/world knowledge for extra thinking time, which I think is a good trade for local. tbf, I've heard luna's not great at coding. Fast and good for things like classifiers, summarization, though. A friend and I were actually discussing today how benches show Luna Max at about par on coding with Sol Medium, but how it's nowhere near in reality. We were speculating that maybe it's because a lot of benches are best-of-n, and should probably be worst-of-n, because variance in performance is killer with large coding projects. Consistency is what lets you actually build on this stuff.
      - bel8 24d ago
        > We were speculating that maybe it's because a lot of benches are best-of-n, and should probably be worst-of-n. Thank you. I just changed my opinion on this thanks to you. I agree now, since we tend to execute LLM tasks once instead of N times anyway. I suspect models like Fable executes the same task N times in parallel and picks best answer or merges them to for a better answer.
- whatsThisBtn4 24d ago
  I can't imagine using CPU... Oh I did twice. If you are work from home and do dishes between prompts you can get a gpt3-like result. I found it useful when I was... Well I didn't find it useful. But an Nvidia 3060 let me ask unethical questions pretty fast.
  - ramgine 24d ago
    With which model. I have a 3060 with a bunch of system ram
    - whatsThisBtn4 24d ago
      Old school Berkeley Sterling or an abliterared model.
  - pjmlp 24d ago
    That is a lot of dishes.
- cdnsteve 24d ago
  These are estimates: https://tokenstead.ai/find/results?hardware_id=23&use_case=coding https://tokenstead.ai/find/results?hardware_id=23&use_case=c...
  - amanzi 24d ago
    That's useful thanks. Also, looks painfully slow!
- taylorhou 24d ago
  i have a 512gb ram m3 ultra mac studio setup with a gas city that runs one of my companies. today was the first time ever that a local model (GLM5.3 8-bit) was able to match fable5 in our tests. GLM-5.3-Flash at true 8-bit: 341 GB on disk, 328 GB resident, 288 experts across 46 layers, loads in 65 seconds. • 18.7 tokens/s generation, 35 tokens/s prompt, on a desk, on a $0 per-token bill. • Runs beside our whole agent city on one box with ~130 GB to spare. • Review test: caught 6 of 6 planted P1 defects, zero false positives, same score as the frontier model we pay for. • CRM test: 11 of 11 required records extracted, zero wrong writes, 45 minutes, first local model to clear the bar. • Serving a 131k-token window today; the model itself supports 1,048,576. Widened to 4 concurrent slots and still have 50gb+ of excess ram. granted my cto still isn't moving all of our inference to glm5.3 but we've identified 40%+ that is currently handled by fable that we're routing locally instead and will do concurrent requests to verify/compare responses for a while.
  - aa-jv 24d ago
    What sort of business can you run with this setup?
  - Normal_gaussian 24d ago
    $0 per-token bill You still have electricity and capital investment. Envelope math suggests cheap electricity is costing you something like $0.50/mtok and the opportunity cost on the capital tied up and lost in the unit purchase and resale is going to cost you something like $2/mtok at 100% utilization (so, frontier model prices or higher at real utilization), and you don't benefit from any elasticity. Hosted GLM 5.3 flash is like $0.15/mtok in $0.50/mtok out
    - icedchai 24d ago
      Time to completion also must be considered. If I have to wait around for hours for a prompt to complete locally and I’ll need to iterate quickly, I’m better off hosted than local. If it’s “free” and slow it may just not be worth it.
  - icedchai 24d ago
- madduci 24d ago
  Are you using the right configuration for your own CPU? On a Laptop with 32 GB RAM and Iris Xe integrated graphic card, I get between 11-18 Tokens/Second with Qwen 3.8 27B and llama.cpp with sysl Intel optimisations. Same results with the vulkan back end, although sometimes it ends in weird segmentation faults due to the memory consumption.
- traceroute66 24d ago
  > I'm able to load a bunch of different models on my little mini-PC with 16GB RAM, but the performance is terrible With all due respect, I'm not clear why you are so surprised ? By your own admission its a little mini-PC with 16GB RAM, I'm not sure what miracles you were expecting ? Its a bit like complaining Rasperry Pi performance is terrible when trying to compile the Linux kernel.
  - AdamN 24d ago
    That's their point
  - amanzi 24d ago
    Not surprised at all - just making the point that getting a model to run isn't that impressive if it runs at a few tokens per second.
- c16 24d ago
  Qwen3.8:27b-mlx on 64GB MBP M4, I can get up to 42tok/s, more often than not in the ~30 range.
- visarga 24d ago
  I took this thread and summarized it with Qwen3.6-35B-A3B, it had 1400 tps prefix and 60 tps completion. Very good performance. Using oMLX on MacBook M5 Pro 64GB.
- phazonoverload 24d ago
  I'm the author - hello! Added to the post! Qwen averages 325 tok/s in processing prompts, and 34 tok/s in token generation. That isn't instant, but it's quick enough that I never really think about it.
xydac 24d ago
yes, share performance, numbers if you can, also i wonder if you figured out a way to do a 2way audio with local models, or even explored that. I have a very similar setup but not too happy with the token speed, will try omlx though !!!
- phazonoverload 24d ago
  I haven't, but that's just because it really isn't my personal usage pattern.
alexgoodhart 24d ago
I have an m1 Mac 64gb and look forward to trying this out Not many people share setup with actual setup handholding so that was very G of you
mkagenius 24d ago
I tried the 1 bit model of Qwen3.6 27B on my M1 pro (16G) and got 13 tok/s with only 5G of ram usage. https://x.com/mkagenius/status/2093730391429685732 https://x.com/mkagenius/status/2093730391429685732 (xcancel seems to have received a cease and desist)
jumploops 24d ago
My biggest problem with running local LLMs on my M4 Max/128GB RAM is the prefill latency. I've since acquired two DGX Sparks, and it feels so much snappier.
- c0rruptbytes 24d ago
  m5 max really fixed pp with the better matmul support, im sure the m5 ultra will be even crazier the sparks have much slower memory bandwidth is the trade off
  - jumploops 24d ago
    I believe the dgx spark is still twice as fast at prefill as the m5 max, but the ultra should get closer to parity. Another benefit of the 2x spark setup is that you can parallelize to ~6 streams pretty efficiently. All depends on the workflows you’re using it for. I’m quite excited for the M7 class machines.
- shell0x 24d ago
  Would you mind sharing your local Mac setup and which models you currently use and whether it’s GGUF or MLX? I’ve the hardware same specs.
  - jumploops 24d ago
    Primarily used ds4[0] by antirez [0]https://github.com/antirez/ds4 https://github.com/antirez/ds4
    - shell0x 24d ago
      Thanks, that’s what I’m currently using too.
mintflow 24d ago
Have a macmini m4 32G, not the pro version, previously everytime I tried local LLM is a bit disappointing, and I finally decide to not waste time and perhaps in the future invest a better hardware to server more modern and dense model I am curious is what is the 80% request served by this setup, I was using it for OpenClaw which run serveral cron jobs that discover stuffs over the wide internet, check my support system's unanswered tickets, browser X and some social media for me to filter the valued ones(though I have to say even with GPT 5.6 sol, the quality is low for the timeline X sent to me) Btw, Tailscale is quite cool and did a good job, I was using it to serve the local LLM and connct the openclaw on a Linux Machine to it.
- arcanemachiner 24d ago
  You have tried Qwen 3.8 27B before coming to this conclusion, I hope? It's an incremental improvement over 3.6, but I mostly want to make sure you didn't just try running some old junker before coming to this conclusion.
- Arya_xiaofan 24d ago
  [dead]
- dnugget 17d ago
  [dead]
miles_io 24d ago
M4 Pro has been a solid performer for iterating on smaller local models. Much more convenient than spinning up cloud instances for dev.
- whatsThisBtn4 24d ago
  If you just want chat. Agents require at least DeepSeek pro and even that is the minimum. You might be able to get a good model to write instructions and run it in smaller models. Otherwise, cool your AI got the current weather.
  - EagnaIonat 24d ago
    > Agents require at least DeepSeek pro and even that is the minimum. The Granite 4.2 models which are just recently out, are optimized to handle agentic workflows. For local models, it's about using the right model for the right job.
    - whatsThisBtn4 24d ago
      Waste of time when I can pay $20 a month for sol.
      - EagnaIonat 24d ago
        versus $0 with local models. There will always be a reason to run frontier models, but local models are well at levels that assist with stuff that don't need that level of complexity.
        whatsThisBtn4 24d ago
        You could buy a $150 refurbished 16gb i5 and use that model until openAI does its IPO and has to become sane again. But I guess a $2000 Mac is probably better if you don't care about cost or quality.
        23d ago
max979 24d ago
That M4 Pro is probably a beast for quantised models. My M2 Pro handles 34B just barely; what speeds are you seeing?
whatsThisBtn4 24d ago
Apple did great work convincing people their unified memory was good at AI. Even AI says Apple is the best of all time at marketing. Meanwhile the stock market has Nvidia at the top... Until everyone gets cuda.
- AdamN 24d ago
  Apple is working from the 'desktop' up to beefy servers with 64GB+ RAM. Nvidia is working from the 'datacenter' down to beefy racks with terabytes of RAM. There isn't really an overlap yet. Individual Nvidia cards exist on desktops but they're not really oriented for regular inference so individual developers are left with Macs or datacenter resources as their options.
  - whatsThisBtn4 24d ago
    Uh... Even my $700 laptop with a 3060 can run 9b models.
shell0x 24d ago
[dead]
crossroadsguy 24d ago
> <a href="https://omlx.app">oMLX</a> Is that supposed to be hallucination? The human or other kind. Feels like a made up URL. It's .ai, isn't it?
- phazonoverload 24d ago
  A literal manual typo. Good catch, will fix.
tukHelix 24d ago
[flagged]
brainless 24d ago
I experiment a lot with local LLMs, particularly small ones like Qwen3.5 4B and 9B. I have build multiple experiments to make harnesses that use these models for code generation, planning, local search, etc. These are really good models but the harness has to be built around them. I have a ton of generated system prompts for specific purposes. Even parts of a SolidJS stack, for example Route management, has its own prompt. These are experiments but the results are real. If we build harnesses around small models, we can build a locally running WYSIWYG editor which works on plain text prompts. The performance, in simple tokens/second, is not the most important factor. For many private data points, like emails, I would rather have a local graph based search and LLM on top where the harness is specific to problems like calendar, contacts, finance, etc. I run all experiments on an 16GB M4 Mac Mini but coding agents building the harness are a mix of Codex, Claude Code and opencode.
yeasin-arafat 24d ago
[flagged]
heliskyr2 24d ago
[flagged]
ttul 24d ago
Most people running local models would probably love to run larger models if only they had access to big enough hardware. I'm curious: to those of you running models locally, if there was a way to inference the model of your choice at a reasonable cost by effectively time-sharing a B300 rack through some privacy-protecting intermediary, would you consider that? If there was a "Mullvad of GPU clouds", would that solve the privacy concerns?
- wilj 24d ago
  runpod.io is essentially this. You can rent the hardware for cheap in small time slices. I do this whenever I need to do a lot of embeddings, fast. I have an agent skill that will estimate the optimum hardware to reserve for the time/price constraints of the job, and you can spin up temporary inference for cheap via their API as well.
- strobe 24d ago
  yes, and it's already some offerings like that but they all cost a lot because they only good for "I have some idea of workload for N hours or days" lets rent it and run. That fine for some experimentation but if you think about renting something 24/7 even for example to share it with the friends that will cost at least 4x from any API prices as result (something like rtx 6000 48gb will cost ~$470/m).
- Youden 24d ago
  Chutes, Near AI, Phala and Tinfoil all offer various privacy assurances around inference. Some of the bigger providers also offer "zero data retention". The problem I have with these is that the guarantees aren't strong enough (Phala, Near) or the models are old (Tinfoil). Chutes is mostly pretty good (cryptographic security all the way to the GPU) but I'm not sure it's possible to cryptographically verify the precise source code they run on the mode.
  - ljlolel 24d ago
    These are all on my router TrustedRouter, and more providers coming. Tinfoil has some newer ones too like GLM 5.3 now. Phala isn't verifying all the way down but NEAR is and I know the CEO
gigatexal 24d ago
I really like these show and tell style posts. I’m always curious how people have their setups and what tools they use. Also the blog has a nice theme and is easy to read. I wanna get a desktop Mac for local ai so that I don’t turn my laptop into a delta 15k rpm fan when I run things. I guess I’ll get in line for one hah.
- ukd1 24d ago
  if you like this style and want more r/LocalLLaMA and r/homelabs have a bunch
thenthenthen 24d ago
Would love to see a tutorial on this setup =D
akg_67 24d ago
Recent performance data on my M1 Max 32GB MacBook using oMLX. I have been working on identifying suitable model and config for my use case and system. Using a refactor and suggest improvements prompt for a specific Django code block using VSCode Cline extension. --- Qwen3.8-27B-4bit, Prompt Processing (PP) 66.3 tok/s, Token Generation (TG) 11.8 tok/s Ornith-1.5-35B-A3B-MLX-4bit, PP 379.7, TG 45.8 Ornith-1.5-35B-A3B-MLX-4bit, PP 381.5, TG 46.4 Qwen3.6-35B-A3B-mxfp4, PP 389.6, TG 47.6 Qwen3.6-35B-A3B-OptiQ-4bit, PP 342.6, TG 44.4 --- Qwen3.8-27B-4bit generally runs out of output token before completing the task though excellent partial results. Ornith-1.5-35B-A3B-MLX-4bit seems to get in the loop often specially with tool calls. Qwen3.6-35B-A3B-mxfp4 seems to be optimal with speed and quality output. I am going to test Qwen3.6-35B-A3B-4bit soon with same code block just to check my intuition that any derivatives don't seem to perform better than the originals.
- madduci 24d ago
  Interesting, what's your Context Window?
  - akg_67 24d ago
    The above tests were done with 24k context window. Testing was mostly driven by ChatGPT analyzing oMLX server logs and suggesting changes. Finally, I settled on Qwen3.6-35B-A3B-4bit with 32,768 context window and 16,384 max tokens. --- Additional results from Qwen3.6-35B-A3B-4bit (Can't edit previous comment) Qwen3.6-35B-A3B-4bit, 329.7 PP, 41.3 TG
- visarga 24d ago
  > Prompt Processing (PP) 66.3 tok/s I got 400 pp tps on a 10k token input. Your numbers seem suspiciously low, maybe the input was too short to measure properly? And this dense 27B is slow, the MoE A3B models get to 1000 tps.
  - akg_67 24d ago
    What system? If on *M1 Max 32GB* or weaker, I will be interested in learning more about your setup.
willtemperley 24d ago
> You do not know what these companies do with your data once they have it. They might limit how it gets used, they might sell it, they might expose it. This is the burning question for me, what are they doing with our hard work. I'd have thought that sherlocking a user's $10M business would be too high risk, given the billions at stake if real evidence of this happening was found. However, OpenAI are currently being sued by Apple for trade secret theft, and the way it was done seems to be abundantly idiotic. So I'm torn.
hoistway 24d ago
Been curious about the M4 Pro for local models. My M3 Max machine still chugs on larger LLMs, definitely eyeing an upgrade.
kelt_row 24d ago
Similar setup here with an M3 Max, it's surprisingly capable. Curious what models you're running on that M4 Pro.
amelius 24d ago
> The main reason to run local: cloud APIs are rented land. They can change their pricing, hit your usage limits, or swap the model being served behind the scenes whenever they feel like it. Yes but it's easy to replace them. The main reason should be privacy.
juggle73 24d ago
[flagged]
workletterco 24d ago
[flagged]
c16 24d ago
> Running a large model locally comes down to one thing: how much RAM it actually needs in memory. Not completely true. It's memory AND memory bandwidth. You can have 1tb of memory but if you have awful memory-bandwidth you'll also have slow tok/s. A3B helps with this, but so does MTP. From my experience, you'd be better off running the dense 27b-mlx with MTP than the 3.6 version with A3B. You say your model is ~20GB of ram, but the 3.8:27b-mlx is 18GB and gets me very reasonable tok/s, and greater speed if you disable thinking when not required.
- Kayou 24d ago
  The dense 27b Qwen on M4 Pro has a prompt processing speed of around 125tok/s which makes it ok to ask a quick question but impossible to use in an agent, as processing the first prompt of the agent with the tools and instruction can easily be 10 000 tokens In this case the 35b a3b makes sense as it has a PP speed of around 800tok/s
  - c16 24d ago
    True. This then boils down to a quality vs speed decision. the 3.8 27b is far better than 3.6 A3B from my experience. I'm happy taking the speed hit, given local models aren't as intelligent as frontier models. Anything that can get me closer to my CC experience both in reasonable speed and intelligence is worth it. With that said CC can also be slow at times, so it's locally the difference in experience is not always noticeable.
- phazonoverload 24d ago
  Running it depends on RAM, which is what I wrote, bandwidth is important for speed. I chose my words carefully, but you are absolutely right.
  - phazonoverload 24d ago
    Coming back to this a few hours later, I've decided to add a section to explain this to the blog post. Thank you for flagging it.
tom_wang007 24d ago
[flagged]
bambax 24d ago
Nice setup; but, for simple tasks or questions, AI is currently free? And it will probably stay free, as I don't see Google starting to charge for using AI on its search engine? So costs can't be a motivation for running small models locally? For more complex or important tasks, costs, autonomy and privacy matter, but then so does performance/quality. So I'm not completely convinced it's really worth it; but it's tempting!
- Den_VR 24d ago
  Only “Free as in free beer.”
  - gwd 24d ago
    Maybe, "Free as in free WiFi?" Like WiFi, the models you can use for free online aren't the highest quality, and can be pulled any time. The models used in TFA are halfway in between the traditional "free as in beer" software. Open weight means once you download it, it continues to work forever; and you can also do your own RL on them; but you can't really see what went into their training, nor train a new one yourself from scratch.
- RugnirViking 24d ago
  I figure most free AI is free as in free electricity in the coffee shop. You're welcome to use it for small reasonable loads, but try to build anything off of it and you'll soon find yourself barred from the establishment. And that's probably good, otherwise the free ai would just be unavailable for everyone else
- ajb 24d ago
  It's free like ads are free. Or certain kinds of advice.
- nobodyandproud 24d ago
  At some point and for some tasks, predictability is important if not critical. I’d rather use a tool where I know the limitations, over a tool where the limitations and strengths keep changing. This way I know where in the process I ought to step in and pay attention.
thrw93747572007 24d ago
Quite a lot of "local doesn't work" in here - unfortunately, often with not much details about what the people actually want to use their models for. Which I'd be curious about. I, personally, do use frontier models in the cloud for a lot of (meta-)cognitive analyses that are heavy enough to have me run against the limits of payed accounts regularly - so I'm neither a Luddite nor stingy with cash in this case. However: I have pretty good experiences with local models as well. My solid but hardly extreme desktop (with one RX 9070 XT 16GB) mostly serves gemma4:12b and specialized models (embedding) to my local network. This is for general use like simple queries, simple code, reformatting and the like but also for two specific tasks that are permanently running: a) It's connected to Home Assistant (as a second stage after very simple "turn light XY on" commands which get processed without LLM). So, I can mumble into my smartwatch "computer, how much gas do we have in the warp core and how much energy did the bussard collectors make from the cosmic dust today?" (or describe a more complex light scene or create an automation I want or whatever). The phone transcribes that - with a local model on device - and fires it to the desktop who has agentic access to HA, looks through the sensors and data, sees that I've tagged my solar panels and battery with nerd vocabulary. It makes the right conclusion, converts a few units and gives me back a nice overview. All hands-free while I'm sitting on the toilet. b) It's the LLM backend for a personal radio station run by a fleet of nerdy/quirky AI DJs who's archetypes are represented more than well enough in the latent space of the "small" model to produce funny results. The DJs can produce consistent, individual segments and programs, run a playlist that works well for me (based on multi-layered audio analysis that also uses local LLMs), respond to song wishes and generally produce much better recommendations than Spotify ever could for me. And you can also put multiple of them in the "studio" to create hilarious crossovers that you would not get from a commercial entity because the IP owners would rather shoot each other in the face. All of this doesn't even max the available resources, so I can shovel F5-TTS into the VRAM as well and have all my DJs have good, locally created voices (or voice clones of Captain Picard and Han Solo, if I wanted to) based on zero-shot voice cloning. --> Far from "unusable". It just depends on the task. And I neither have to hand my keys to the Navidrome server nor to my Smart Home to any entity outside my local network.
stub_out 24d ago
Oh man, an M4 Pro. My old M1 is really starting to show its age trying to run anything bigger than 7B.
altern8 24d ago
Is this another marketing paid post from Apple..?
- phazonoverload 24d ago
  It is not
wila 24d ago
Is there anything one can reasonably run on a mac mini M2 with just 24GB RAM or should I not even try?
- ch_sm 24d ago
  It depends on your use case, but the smaller Gemma 4 models or qwen3.6:9b would probably run OK on that. I recommend trying it, even just for fun. It‘s easy with omlx.
  - wila 24d ago
    Ok, will see if I can find some time for play. Thanks.
- pornel 24d ago
  Not enough for coding. 48GB is minimum for a non-lobotomized coding model like qwen, and you'd likely want 64GB to have long context and not kernel panic when Chrome opens. You could run one of the smaller Gemma models to have a chatty Wikipedia.
  - wila 24d ago
    Figured that would be the case. Thanks.
sdevonoes 24d ago
Tip: include pictures of your machine. Not sure if I’m alone in this, but I love to see other people’s setups
claud_ia 24d ago
[flagged]
apexalpha 24d ago
I have the exact same stack (macbook pro in stead of mini) and swapped from qwen to: https://huggingface.co/ornith-ai/Ornith-1.5-35B-A3B https://huggingface.co/ornith-ai/Ornith-1.5-35B-A3B Literally today, but it feels like an improvement.
- phazonoverload 24d ago
  I will give it a look!
SipitenoMK 24d ago
Great setup! Well done!
t1E9mE7JTRjf 24d ago
Maybe a silly question, but is there a reason/advantage to using mac minis over any other kind of small computer/laptop, ie running linux? My understanding of using a mac mini for ai (ie running a claw bot or whatever) is to have it 'always on' and a better price/performance profile than a cheap vps. Is there performance (silicon processor?) so unique? As I guess it's not their graphics units. I see tonnes of people using mac minis for AI, to the point it almost became a meme. Edit: yes I know this article is about local models, my question is a bit more general.
- bbatha 24d ago
  Unified memory is the answer, it’s one of the cheapest ways to get >64GB of gpu addressable memory.
- wuschel 24d ago
  The main difference total other laptops of non-Apple make is/was unified memory (graphic VRAM + RAM) architecture. No need for an extra dedicated graphics card to get 64+ GB VRAM.
- em500 24d ago
  There are probably three main reasons: 1) unified memory (but you can also that with DGS Spark / Strix Halo), 2) access to your Apple account, so you can have a bot handle your iMessages, email, calendars 3) energy efficiency. Apart from that, if it doesn't work out you still have a Mac Mini, which in itself is more desirable for many than a DGX Spark or Strix Halo if you have no AI use case.
- numpad0 24d ago
  No, it's just hype and people cargo culting local LLM guys buying maxed out Mac Studio for its massive and relatively fast GPU-assignable RAM.
JKCalhoun 24d ago
I like the in-depth description. Everything from the naming convention of the models (and how much RAM they require) as well as all the components needed underscores just how complicated this all still is. I suppose I am waiting for AI-in-a-Box to come along so I can (painlessly) join in. (I'm sure wrangling with all these esoteric aspects of LLMs though is fun for some people.)
- phazonoverload 24d ago
  Hehe I really am just working it out as I go along - I promise it is fairly painless. Hugging Face allow you to specify your machine and then browse models that fit. And then you can just vibe out 'oh this one is a bit slow let me try another' etc
- maherbeg 24d ago
  I've seen https://www.lucebox.com/ https://www.lucebox.com/ as an interesting option
  - manueltgomes 23d ago
    It’s $7900! “Interesting” and “option” are doing a lot of heavy lifting there :).
    - Bluestein 23d ago
      ... if they are doing heavy lifting, are they ... load-bearing? :)
  - JKCalhoun 23d ago
    I was hoping for a software "LLM in a Box".
jimbobthemighty 24d ago
I just use Gemini Pro which comes bundled with a Chromebook; when the 'free' year runs out I buy another, initiate the free years Gemini again and then put the 'as new' Chromebook on eBay and get most of my money back. So Gemini Pro costs me less than £2 a month. And for my needs (investment research) that works well and is a pretty cheap compromise.
- VCFundedGenYer 24d ago
  You didn't read the article nor understand the purpose. The way you are using it uses the internet and datacenters. It is costly to the environment. Running locally is a significant cost savings in comparison.
manmal 24d ago
If author runs the local model for privacy reasons, then I don't understand why they give Telegram access to all their conversations. It's well known that Telegram doesn't end-to-end encrypt bot accounts.
- phazonoverload 24d ago
  The beauty of this is that you can just swap out the platform and everything remains as it's the same backend. You make a really good point, one that I haven't really considered, but I also only have so many hours in the day to be fiddling around with stuff.
  - manmal 24d ago
    I think I'd use Discord instead. Clankers are happy to set it all up for you.
    - sunaookami 23d ago
      Discord is even worse than Telegram in the privacy aspect though!?
      - manmal 22d ago
        Depends what actors you trust. They are equally bad but personally I‘m way more comfortable with the Discord story.
NikhilChowdaryG 24d ago
[flagged]
saejox 23d ago
Performance or quailty of the llm is not the draw for me. I read those blog posts to remind of wealth gap i have with the average hackernews user. Thank you for the encouragement, i will work harder to reach your level.

336 points by raybb 24d ago

No mention of the performance of the models? I'm able to load a bunch of different models on my little mini-PC with 16GB RAM, but the performance is terrible. I always wonder what performance people are getting with local models that they find is acceptable?

argee 24d ago
I have an M4 pro (48 GB ram) and I run Gemma 4 26b a4b at 52 tok/s and Qwen 3.5b a3b at 72 tok/s. Both 4bit quantized. These are enough for my needs and the performance is more than good enough. I'm not running the MLX version of the Gemma model, if I did the inference speed would likely be a bit better. I wouldn't use them for coding features though.
- dolebirchwood 24d ago
  > enough for my needs Which are...?
  - argee 24d ago
    Some examples (keep in mind this is all indefinitely free for me, no burning quota away): 1. Getting information (such as information about hardware unfamiliar to me) when not connected to the internet, which happens occasionally in my case. 2. Continuing to learn Rust by way of toy examples, puzzles, and comparing aspects of various solutions, for example from LeetCode. 3. Reformatting data, for example from a PDF to a markdown table, or converting receipt images to text. 4. Simple translation/explanation (e.g. I'm teaching my wife one of the languages I speak but sometimes may not know/have the words to explain the full nuance of a translated word). 5. Summarization. One of the webnovels I'm reading has some very boring parts I don't want to slog through, in those cases I simply make the LLM summarize that part and move on. Etc., you get the idea. It's not unusable for coding, but it would make many mistakes when making a whole feature and the context lengths are limited to around 30k-40k tokens by my RAM. I could give it access to the web but I simply use an online model when I need that sort of thing, again partly due to the context limit. Edit: The MLX version of Gemma 4 26b a4b does about 62 tok/s.
- phazonoverload 24d ago
  My perf sucks compared to yours. Added it to the post - same model averages 325 tok/s in processing prompts, and 34 tok/s in token generation. What am I doing wrong..?
  - argee 24d ago
    Wow, that's just about half the perf. I'm not sure what you're doing differently, though our hardware is a bit different: I am on a Macbook Pro M4 Pro, while you're on a Mac Mini. I would try a different version of the model from HuggingFace while ensuring it's MLX. I'm also using LM Studio, not oMLX, and I've seen some threads like these: https://www.reddit.com/r/LocalLLaMA/comments/1spuwir/omlx_10ts_slowlier_than_lm_studio_qwen36_35ba3_on/ https://www.reddit.com/r/LocalLLaMA/comments/1spuwir/omlx_10...
- dd8601fn 24d ago
  I have a (now discontinued) 64gb mini pro and I’ve found the same qwen model to be almost unusable unless I kill Thinking on each turn. What are you using them with/for?
  - argee 24d ago
    I do turn thinking off most of the time for both models. I made a separate comment detailing my use cases.
- pwthornton 23d ago
  What kinds of tasks are you using this for?
pcarolan 24d ago
It’s not. Do it as a hobby or for privacy but for performance just use a frontier model api. You’re paying less than cost for something that would take tens of thousands to set up locally.
- Gigachad 24d ago
  It does make me wonder how the hosted stuff is so cheap. For pretty much everything else, hosted/rented is more expensive but offers better convenience and flexibility. But for AI, even if you consider the total lifetime cost and are utilizing it heavily. You never break even by buying.
  - api 24d ago
    There are economies of scale but there’s also a data center bubble (probably) so there might be some selling dollars for fifty cents going on.
    - mrngld 24d ago
      Here's the thing that's a little different about data centers; we can tell from Anthropic and OpenAI that they're capacity constrained. Inference demand is there. I notice Cerebras doesn't offer much directly any more, all their capacity is getting completely sucked up by B2B sales. Grok did overbuild, but Anthropic was so desperate for more compute they ate their pride and leased the excess capacity. That means all these data centers are being heavily utilized by actual end user inference demand. Well, some is research on new models, but a lot is actual end user demand. No one has given an explanation of why peoples usage would decline. On top of that, margin on inference appears to be decent. It's model training that's a serious financial burden. And maybe that's where there will be a slowdown, maybe the market doesn't justify spending as much on R&D as it does, but the end demand for inference is there. Does that justify these stock prices? That's a different question. But the housing boom left behind endless rows of empty homes because demand disappeared. The 'dot com' boom left behind thousands of miles of dark fiber that'd been built out well ahead of demand for bandwidth. I can see the stock market having a giant sell off, but I don't see data centers sitting idle in that same fashion.
hkchad 24d ago
I run a similar setup to the one he described on similar hardware. I run bifrost and llama swap though (tailscale rocks). My local model usage is for some out of band batch processing one of my personal apps uses. Basically a personalized recommender for media, it curates stuff for me based on a database i've compiled over years, so non-interactive. For that use case, I don't really care that it might take a few minutes to run. It's free. The machine is just sitting there anyway. I have tried using qwen-coder and opencode on my M5 Max 128gb and compared to claude code it's painful. I did setup a workflow where claude plans, qwen executes (unattended overnight, again b/c it's slow) and then claude reviews. I benchmarked this several times and I ended up using MORE tokens with claude because it had to 'fix' all the qwen issues. While the code it produced was 'good enough' the fixes were worth it so I just stick to coding task using API models (codex and claude).
- deleted 24d ago
  [deleted]
- brettdav 24d ago
  Can you share a bit more about your bifrost and llama swap setup? I’m facing memory constraints and am looking for a managed model solution that will help with hot swapping loaded models and stay-warm concurrency. Ideally with prioritization.
  - hkchad 24d ago
    What do you want to know? Just start llama-swap with the models i have downloaded, add llama-swap as a provider in bifrost, expose the models you want and they become available in one single endpoint you can use in anything like opencode, openwebui or anything that speaks openai.
    - brettdav 22d ago
      Yes, that makes sense. Some of my models currently run in ollama while others require their own inference servers. I’m curious about custom inference servers in bifrost and the ability to orchestrate keeping some models warm in memory while evicting and cycling other models. All of which span different providers in bifrost (I think). Obviously I can get the mechanics from an agent, more wondering about any experience with something similar.
ericd 24d ago
I honestly wouldn’t bother with local models right now unless I either had a 5090 and was happy with running Qwen 3.8 27B, or a pair of DGX Sparks running DSv4 flash, or better, 2x6000 RTX Blackwells. Those are the kinds of rigs that the local model enthusiasts are running. With the GPU setups, you’re looking at generally >100tps generation in single stream, and >10k tps of prefill, so it’s snappier than Claude code, which somewhat makes up for it being dumber. That said, it is really cool to be able to run an LLM on eg a Mac laptop. Just not a better experience on almost any metric for interactive use than eg Claude Code, beside privacy and guardrails.
- gruez 24d ago
  >I honestly wouldn’t bother with local models right now unless I either had a 5090 and was happy with running Qwen 3.8 27B How's the actual performance of Qwen 3.8 27B? On deepswe it supposedly performs slightly worse than gpt 5.6 luna high[1], but I can't help but think they've been benchmaxxed. [1] https://deepswe.datacurve.ai/ https://deepswe.datacurve.ai/, https://unsloth.ai/docs/models/qwen3.8#benchmarks https://unsloth.ai/docs/models/qwen3.8#benchmarks
  - ericd 24d ago
    Not sure, I haven't run it, I've just been running DS V4 Flash non-stop since it came out, and that's replaced a lot of my Claude Code usage. People seem very impressed, though, it seems like it trades vram/world knowledge for extra thinking time, which I think is a good trade for local. tbf, I've heard luna's not great at coding. Fast and good for things like classifiers, summarization, though. A friend and I were actually discussing today how benches show Luna Max at about par on coding with Sol Medium, but how it's nowhere near in reality. We were speculating that maybe it's because a lot of benches are best-of-n, and should probably be worst-of-n, because variance in performance is killer with large coding projects. Consistency is what lets you actually build on this stuff.
    - bel8 24d ago
      > We were speculating that maybe it's because a lot of benches are best-of-n, and should probably be worst-of-n. Thank you. I just changed my opinion on this thanks to you. I agree now, since we tend to execute LLM tasks once instead of N times anyway. I suspect models like Fable executes the same task N times in parallel and picks best answer or merges them to for a better answer.
whatsThisBtn4 24d ago
I can't imagine using CPU... Oh I did twice. If you are work from home and do dishes between prompts you can get a gpt3-like result. I found it useful when I was... Well I didn't find it useful. But an Nvidia 3060 let me ask unethical questions pretty fast.
- ramgine 24d ago
  With which model. I have a 3060 with a bunch of system ram
  - whatsThisBtn4 24d ago
    Old school Berkeley Sterling or an abliterared model.
- pjmlp 24d ago
  That is a lot of dishes.
cdnsteve 24d ago
These are estimates: https://tokenstead.ai/find/results?hardware_id=23&use_case=coding https://tokenstead.ai/find/results?hardware_id=23&use_case=c...
- amanzi 24d ago
  That's useful thanks. Also, looks painfully slow!
taylorhou 24d ago
i have a 512gb ram m3 ultra mac studio setup with a gas city that runs one of my companies. today was the first time ever that a local model (GLM5.3 8-bit) was able to match fable5 in our tests. GLM-5.3-Flash at true 8-bit: 341 GB on disk, 328 GB resident, 288 experts across 46 layers, loads in 65 seconds. • 18.7 tokens/s generation, 35 tokens/s prompt, on a desk, on a $0 per-token bill. • Runs beside our whole agent city on one box with ~130 GB to spare. • Review test: caught 6 of 6 planted P1 defects, zero false positives, same score as the frontier model we pay for. • CRM test: 11 of 11 required records extracted, zero wrong writes, 45 minutes, first local model to clear the bar. • Serving a 131k-token window today; the model itself supports 1,048,576. Widened to 4 concurrent slots and still have 50gb+ of excess ram. granted my cto still isn't moving all of our inference to glm5.3 but we've identified 40%+ that is currently handled by fable that we're routing locally instead and will do concurrent requests to verify/compare responses for a while.
- aa-jv 24d ago
  What sort of business can you run with this setup?
- Normal_gaussian 24d ago
  $0 per-token bill You still have electricity and capital investment. Envelope math suggests cheap electricity is costing you something like $0.50/mtok and the opportunity cost on the capital tied up and lost in the unit purchase and resale is going to cost you something like $2/mtok at 100% utilization (so, frontier model prices or higher at real utilization), and you don't benefit from any elasticity. Hosted GLM 5.3 flash is like $0.15/mtok in $0.50/mtok out
  - icedchai 24d ago
    Time to completion also must be considered. If I have to wait around for hours for a prompt to complete locally and I’ll need to iterate quickly, I’m better off hosted than local. If it’s “free” and slow it may just not be worth it.
- icedchai 24d ago
madduci 24d ago
Are you using the right configuration for your own CPU? On a Laptop with 32 GB RAM and Iris Xe integrated graphic card, I get between 11-18 Tokens/Second with Qwen 3.8 27B and llama.cpp with sysl Intel optimisations. Same results with the vulkan back end, although sometimes it ends in weird segmentation faults due to the memory consumption.
traceroute66 24d ago
> I'm able to load a bunch of different models on my little mini-PC with 16GB RAM, but the performance is terrible With all due respect, I'm not clear why you are so surprised ? By your own admission its a little mini-PC with 16GB RAM, I'm not sure what miracles you were expecting ? Its a bit like complaining Rasperry Pi performance is terrible when trying to compile the Linux kernel.
- AdamN 24d ago
  That's their point
- amanzi 24d ago
  Not surprised at all - just making the point that getting a model to run isn't that impressive if it runs at a few tokens per second.
c16 24d ago
Qwen3.8:27b-mlx on 64GB MBP M4, I can get up to 42tok/s, more often than not in the ~30 range.
visarga 24d ago
I took this thread and summarized it with Qwen3.6-35B-A3B, it had 1400 tps prefix and 60 tps completion. Very good performance. Using oMLX on MacBook M5 Pro 64GB.
phazonoverload 24d ago
I'm the author - hello! Added to the post! Qwen averages 325 tok/s in processing prompts, and 34 tok/s in token generation. That isn't instant, but it's quick enough that I never really think about it.

xydac 24d ago

yes, share performance, numbers if you can, also i wonder if you figured out a way to do a 2way audio with local models, or even explored that. I have a very similar setup but not too happy with the token speed, will try omlx though !!!

phazonoverload 24d ago
I haven't, but that's just because it really isn't my personal usage pattern.

alexgoodhart 24d ago

I have an m1 Mac 64gb and look forward to trying this out Not many people share setup with actual setup handholding so that was very G of you

mkagenius 24d ago

I tried the 1 bit model of Qwen3.6 27B on my M1 pro (16G) and got 13 tok/s with only 5G of ram usage. https://x.com/mkagenius/status/2093730391429685732 https://x.com/mkagenius/status/2093730391429685732 (xcancel seems to have received a cease and desist)

jumploops 24d ago

My biggest problem with running local LLMs on my M4 Max/128GB RAM is the prefill latency. I've since acquired two DGX Sparks, and it feels so much snappier.

c0rruptbytes 24d ago
m5 max really fixed pp with the better matmul support, im sure the m5 ultra will be even crazier the sparks have much slower memory bandwidth is the trade off
- jumploops 24d ago
  I believe the dgx spark is still twice as fast at prefill as the m5 max, but the ultra should get closer to parity. Another benefit of the 2x spark setup is that you can parallelize to ~6 streams pretty efficiently. All depends on the workflows you’re using it for. I’m quite excited for the M7 class machines.
shell0x 24d ago
Would you mind sharing your local Mac setup and which models you currently use and whether it’s GGUF or MLX? I’ve the hardware same specs.
- jumploops 24d ago
  Primarily used ds4[0] by antirez [0]https://github.com/antirez/ds4 https://github.com/antirez/ds4
  - shell0x 24d ago
    Thanks, that’s what I’m currently using too.

mintflow 24d ago

Have a macmini m4 32G, not the pro version, previously everytime I tried local LLM is a bit disappointing, and I finally decide to not waste time and perhaps in the future invest a better hardware to server more modern and dense model I am curious is what is the 80% request served by this setup, I was using it for OpenClaw which run serveral cron jobs that discover stuffs over the wide internet, check my support system's unanswered tickets, browser X and some social media for me to filter the valued ones(though I have to say even with GPT 5.6 sol, the quality is low for the timeline X sent to me) Btw, Tailscale is quite cool and did a good job, I was using it to serve the local LLM and connct the openclaw on a Linux Machine to it.

arcanemachiner 24d ago
You have tried Qwen 3.8 27B before coming to this conclusion, I hope? It's an incremental improvement over 3.6, but I mostly want to make sure you didn't just try running some old junker before coming to this conclusion.
Arya_xiaofan 24d ago
[dead]
dnugget 17d ago
[dead]

miles_io 24d ago

M4 Pro has been a solid performer for iterating on smaller local models. Much more convenient than spinning up cloud instances for dev.

whatsThisBtn4 24d ago
If you just want chat. Agents require at least DeepSeek pro and even that is the minimum. You might be able to get a good model to write instructions and run it in smaller models. Otherwise, cool your AI got the current weather.
- EagnaIonat 24d ago
  > Agents require at least DeepSeek pro and even that is the minimum. The Granite 4.2 models which are just recently out, are optimized to handle agentic workflows. For local models, it's about using the right model for the right job.
  - whatsThisBtn4 24d ago
    Waste of time when I can pay $20 a month for sol.
    - EagnaIonat 24d ago
      versus $0 with local models. There will always be a reason to run frontier models, but local models are well at levels that assist with stuff that don't need that level of complexity.
      - whatsThisBtn4 24d ago
        You could buy a $150 refurbished 16gb i5 and use that model until openAI does its IPO and has to become sane again. But I guess a $2000 Mac is probably better if you don't care about cost or quality.
        23d ago

max979 24d ago

That M4 Pro is probably a beast for quantised models. My M2 Pro handles 34B just barely; what speeds are you seeing?

whatsThisBtn4 24d ago

Apple did great work convincing people their unified memory was good at AI. Even AI says Apple is the best of all time at marketing. Meanwhile the stock market has Nvidia at the top... Until everyone gets cuda.

AdamN 24d ago
Apple is working from the 'desktop' up to beefy servers with 64GB+ RAM. Nvidia is working from the 'datacenter' down to beefy racks with terabytes of RAM. There isn't really an overlap yet. Individual Nvidia cards exist on desktops but they're not really oriented for regular inference so individual developers are left with Macs or datacenter resources as their options.
- whatsThisBtn4 24d ago
  Uh... Even my $700 laptop with a 3060 can run 9b models.

shell0x 24d ago

[dead]

crossroadsguy 24d ago

> <a href="https://omlx.app">oMLX</a> Is that supposed to be hallucination? The human or other kind. Feels like a made up URL. It's .ai, isn't it?

phazonoverload 24d ago
A literal manual typo. Good catch, will fix.

tukHelix 24d ago

[flagged]

brainless 24d ago

I experiment a lot with local LLMs, particularly small ones like Qwen3.5 4B and 9B. I have build multiple experiments to make harnesses that use these models for code generation, planning, local search, etc. These are really good models but the harness has to be built around them. I have a ton of generated system prompts for specific purposes. Even parts of a SolidJS stack, for example Route management, has its own prompt. These are experiments but the results are real. If we build harnesses around small models, we can build a locally running WYSIWYG editor which works on plain text prompts. The performance, in simple tokens/second, is not the most important factor. For many private data points, like emails, I would rather have a local graph based search and LLM on top where the harness is specific to problems like calendar, contacts, finance, etc. I run all experiments on an 16GB M4 Mac Mini but coding agents building the harness are a mix of Codex, Claude Code and opencode.

yeasin-arafat 24d ago

[flagged]

heliskyr2 24d ago

[flagged]

ttul 24d ago

Most people running local models would probably love to run larger models if only they had access to big enough hardware. I'm curious: to those of you running models locally, if there was a way to inference the model of your choice at a reasonable cost by effectively time-sharing a B300 rack through some privacy-protecting intermediary, would you consider that? If there was a "Mullvad of GPU clouds", would that solve the privacy concerns?

wilj 24d ago
runpod.io is essentially this. You can rent the hardware for cheap in small time slices. I do this whenever I need to do a lot of embeddings, fast. I have an agent skill that will estimate the optimum hardware to reserve for the time/price constraints of the job, and you can spin up temporary inference for cheap via their API as well.
strobe 24d ago
yes, and it's already some offerings like that but they all cost a lot because they only good for "I have some idea of workload for N hours or days" lets rent it and run. That fine for some experimentation but if you think about renting something 24/7 even for example to share it with the friends that will cost at least 4x from any API prices as result (something like rtx 6000 48gb will cost ~$470/m).
Youden 24d ago
Chutes, Near AI, Phala and Tinfoil all offer various privacy assurances around inference. Some of the bigger providers also offer "zero data retention". The problem I have with these is that the guarantees aren't strong enough (Phala, Near) or the models are old (Tinfoil). Chutes is mostly pretty good (cryptographic security all the way to the GPU) but I'm not sure it's possible to cryptographically verify the precise source code they run on the mode.
- ljlolel 24d ago
  These are all on my router TrustedRouter, and more providers coming. Tinfoil has some newer ones too like GLM 5.3 now. Phala isn't verifying all the way down but NEAR is and I know the CEO

gigatexal 24d ago

I really like these show and tell style posts. I’m always curious how people have their setups and what tools they use. Also the blog has a nice theme and is easy to read. I wanna get a desktop Mac for local ai so that I don’t turn my laptop into a delta 15k rpm fan when I run things. I guess I’ll get in line for one hah.

ukd1 24d ago
if you like this style and want more r/LocalLLaMA and r/homelabs have a bunch

thenthenthen 24d ago

Would love to see a tutorial on this setup =D

akg_67 24d ago

Recent performance data on my M1 Max 32GB MacBook using oMLX. I have been working on identifying suitable model and config for my use case and system. Using a refactor and suggest improvements prompt for a specific Django code block using VSCode Cline extension. --- Qwen3.8-27B-4bit, Prompt Processing (PP) 66.3 tok/s, Token Generation (TG) 11.8 tok/s Ornith-1.5-35B-A3B-MLX-4bit, PP 379.7, TG 45.8 Ornith-1.5-35B-A3B-MLX-4bit, PP 381.5, TG 46.4 Qwen3.6-35B-A3B-mxfp4, PP 389.6, TG 47.6 Qwen3.6-35B-A3B-OptiQ-4bit, PP 342.6, TG 44.4 --- Qwen3.8-27B-4bit generally runs out of output token before completing the task though excellent partial results. Ornith-1.5-35B-A3B-MLX-4bit seems to get in the loop often specially with tool calls. Qwen3.6-35B-A3B-mxfp4 seems to be optimal with speed and quality output. I am going to test Qwen3.6-35B-A3B-4bit soon with same code block just to check my intuition that any derivatives don't seem to perform better than the originals.

madduci 24d ago
Interesting, what's your Context Window?
- akg_67 24d ago
  The above tests were done with 24k context window. Testing was mostly driven by ChatGPT analyzing oMLX server logs and suggesting changes. Finally, I settled on Qwen3.6-35B-A3B-4bit with 32,768 context window and 16,384 max tokens. --- Additional results from Qwen3.6-35B-A3B-4bit (Can't edit previous comment) Qwen3.6-35B-A3B-4bit, 329.7 PP, 41.3 TG
visarga 24d ago
> Prompt Processing (PP) 66.3 tok/s I got 400 pp tps on a 10k token input. Your numbers seem suspiciously low, maybe the input was too short to measure properly? And this dense 27B is slow, the MoE A3B models get to 1000 tps.
- akg_67 24d ago
  What system? If on *M1 Max 32GB* or weaker, I will be interested in learning more about your setup.

willtemperley 24d ago

> You do not know what these companies do with your data once they have it. They might limit how it gets used, they might sell it, they might expose it. This is the burning question for me, what are they doing with our hard work. I'd have thought that sherlocking a user's $10M business would be too high risk, given the billions at stake if real evidence of this happening was found. However, OpenAI are currently being sued by Apple for trade secret theft, and the way it was done seems to be abundantly idiotic. So I'm torn.

hoistway 24d ago

Been curious about the M4 Pro for local models. My M3 Max machine still chugs on larger LLMs, definitely eyeing an upgrade.

kelt_row 24d ago

Similar setup here with an M3 Max, it's surprisingly capable. Curious what models you're running on that M4 Pro.

amelius 24d ago

> The main reason to run local: cloud APIs are rented land. They can change their pricing, hit your usage limits, or swap the model being served behind the scenes whenever they feel like it. Yes but it's easy to replace them. The main reason should be privacy.

juggle73 24d ago

[flagged]

workletterco 24d ago

[flagged]

c16 24d ago

> Running a large model locally comes down to one thing: how much RAM it actually needs in memory. Not completely true. It's memory AND memory bandwidth. You can have 1tb of memory but if you have awful memory-bandwidth you'll also have slow tok/s. A3B helps with this, but so does MTP. From my experience, you'd be better off running the dense 27b-mlx with MTP than the 3.6 version with A3B. You say your model is ~20GB of ram, but the 3.8:27b-mlx is 18GB and gets me very reasonable tok/s, and greater speed if you disable thinking when not required.

Kayou 24d ago
The dense 27b Qwen on M4 Pro has a prompt processing speed of around 125tok/s which makes it ok to ask a quick question but impossible to use in an agent, as processing the first prompt of the agent with the tools and instruction can easily be 10 000 tokens In this case the 35b a3b makes sense as it has a PP speed of around 800tok/s
- c16 24d ago
  True. This then boils down to a quality vs speed decision. the 3.8 27b is far better than 3.6 A3B from my experience. I'm happy taking the speed hit, given local models aren't as intelligent as frontier models. Anything that can get me closer to my CC experience both in reasonable speed and intelligence is worth it. With that said CC can also be slow at times, so it's locally the difference in experience is not always noticeable.
phazonoverload 24d ago
Running it depends on RAM, which is what I wrote, bandwidth is important for speed. I chose my words carefully, but you are absolutely right.
- phazonoverload 24d ago
  Coming back to this a few hours later, I've decided to add a section to explain this to the blog post. Thank you for flagging it.

tom_wang007 24d ago

[flagged]

bambax 24d ago

Nice setup; but, for simple tasks or questions, AI is currently free? And it will probably stay free, as I don't see Google starting to charge for using AI on its search engine? So costs can't be a motivation for running small models locally? For more complex or important tasks, costs, autonomy and privacy matter, but then so does performance/quality. So I'm not completely convinced it's really worth it; but it's tempting!

Den_VR 24d ago
Only “Free as in free beer.”
- gwd 24d ago
  Maybe, "Free as in free WiFi?" Like WiFi, the models you can use for free online aren't the highest quality, and can be pulled any time. The models used in TFA are halfway in between the traditional "free as in beer" software. Open weight means once you download it, it continues to work forever; and you can also do your own RL on them; but you can't really see what went into their training, nor train a new one yourself from scratch.
RugnirViking 24d ago
I figure most free AI is free as in free electricity in the coffee shop. You're welcome to use it for small reasonable loads, but try to build anything off of it and you'll soon find yourself barred from the establishment. And that's probably good, otherwise the free ai would just be unavailable for everyone else
ajb 24d ago
It's free like ads are free. Or certain kinds of advice.
nobodyandproud 24d ago
At some point and for some tasks, predictability is important if not critical. I’d rather use a tool where I know the limitations, over a tool where the limitations and strengths keep changing. This way I know where in the process I ought to step in and pay attention.

thrw93747572007 24d ago

Quite a lot of "local doesn't work" in here - unfortunately, often with not much details about what the people actually want to use their models for. Which I'd be curious about. I, personally, do use frontier models in the cloud for a lot of (meta-)cognitive analyses that are heavy enough to have me run against the limits of payed accounts regularly - so I'm neither a Luddite nor stingy with cash in this case. However: I have pretty good experiences with local models as well. My solid but hardly extreme desktop (with one RX 9070 XT 16GB) mostly serves gemma4:12b and specialized models (embedding) to my local network. This is for general use like simple queries, simple code, reformatting and the like but also for two specific tasks that are permanently running: a) It's connected to Home Assistant (as a second stage after very simple "turn light XY on" commands which get processed without LLM). So, I can mumble into my smartwatch "computer, how much gas do we have in the warp core and how much energy did the bussard collectors make from the cosmic dust today?" (or describe a more complex light scene or create an automation I want or whatever). The phone transcribes that - with a local model on device - and fires it to the desktop who has agentic access to HA, looks through the sensors and data, sees that I've tagged my solar panels and battery with nerd vocabulary. It makes the right conclusion, converts a few units and gives me back a nice overview. All hands-free while I'm sitting on the toilet. b) It's the LLM backend for a personal radio station run by a fleet of nerdy/quirky AI DJs who's archetypes are represented more than well enough in the latent space of the "small" model to produce funny results. The DJs can produce consistent, individual segments and programs, run a playlist that works well for me (based on multi-layered audio analysis that also uses local LLMs), respond to song wishes and generally produce much better recommendations than Spotify ever could for me. And you can also put multiple of them in the "studio" to create hilarious crossovers that you would not get from a commercial entity because the IP owners would rather shoot each other in the face. All of this doesn't even max the available resources, so I can shovel F5-TTS into the VRAM as well and have all my DJs have good, locally created voices (or voice clones of Captain Picard and Han Solo, if I wanted to) based on zero-shot voice cloning. --> Far from "unusable". It just depends on the task. And I neither have to hand my keys to the Navidrome server nor to my Smart Home to any entity outside my local network.

stub_out 24d ago

Oh man, an M4 Pro. My old M1 is really starting to show its age trying to run anything bigger than 7B.

altern8 24d ago

Is this another marketing paid post from Apple..?

phazonoverload 24d ago
It is not

wila 24d ago

Is there anything one can reasonably run on a mac mini M2 with just 24GB RAM or should I not even try?

ch_sm 24d ago
It depends on your use case, but the smaller Gemma 4 models or qwen3.6:9b would probably run OK on that. I recommend trying it, even just for fun. It‘s easy with omlx.
- wila 24d ago
  Ok, will see if I can find some time for play. Thanks.
pornel 24d ago
Not enough for coding. 48GB is minimum for a non-lobotomized coding model like qwen, and you'd likely want 64GB to have long context and not kernel panic when Chrome opens. You could run one of the smaller Gemma models to have a chatty Wikipedia.
- wila 24d ago
  Figured that would be the case. Thanks.

sdevonoes 24d ago

Tip: include pictures of your machine. Not sure if I’m alone in this, but I love to see other people’s setups

claud_ia 24d ago

[flagged]

apexalpha 24d ago

I have the exact same stack (macbook pro in stead of mini) and swapped from qwen to: https://huggingface.co/ornith-ai/Ornith-1.5-35B-A3B https://huggingface.co/ornith-ai/Ornith-1.5-35B-A3B Literally today, but it feels like an improvement.

phazonoverload 24d ago
I will give it a look!

SipitenoMK 24d ago

Great setup! Well done!

t1E9mE7JTRjf 24d ago

Maybe a silly question, but is there a reason/advantage to using mac minis over any other kind of small computer/laptop, ie running linux? My understanding of using a mac mini for ai (ie running a claw bot or whatever) is to have it 'always on' and a better price/performance profile than a cheap vps. Is there performance (silicon processor?) so unique? As I guess it's not their graphics units. I see tonnes of people using mac minis for AI, to the point it almost became a meme. Edit: yes I know this article is about local models, my question is a bit more general.

bbatha 24d ago
Unified memory is the answer, it’s one of the cheapest ways to get >64GB of gpu addressable memory.
wuschel 24d ago
The main difference total other laptops of non-Apple make is/was unified memory (graphic VRAM + RAM) architecture. No need for an extra dedicated graphics card to get 64+ GB VRAM.
em500 24d ago
There are probably three main reasons: 1) unified memory (but you can also that with DGS Spark / Strix Halo), 2) access to your Apple account, so you can have a bot handle your iMessages, email, calendars 3) energy efficiency. Apart from that, if it doesn't work out you still have a Mac Mini, which in itself is more desirable for many than a DGX Spark or Strix Halo if you have no AI use case.
numpad0 24d ago
No, it's just hype and people cargo culting local LLM guys buying maxed out Mac Studio for its massive and relatively fast GPU-assignable RAM.

JKCalhoun 24d ago

I like the in-depth description. Everything from the naming convention of the models (and how much RAM they require) as well as all the components needed underscores just how complicated this all still is. I suppose I am waiting for AI-in-a-Box to come along so I can (painlessly) join in. (I'm sure wrangling with all these esoteric aspects of LLMs though is fun for some people.)

phazonoverload 24d ago
Hehe I really am just working it out as I go along - I promise it is fairly painless. Hugging Face allow you to specify your machine and then browse models that fit. And then you can just vibe out 'oh this one is a bit slow let me try another' etc
maherbeg 24d ago
I've seen https://www.lucebox.com/ https://www.lucebox.com/ as an interesting option
- manueltgomes 23d ago
  It’s $7900! “Interesting” and “option” are doing a lot of heavy lifting there :).
  - Bluestein 23d ago
    ... if they are doing heavy lifting, are they ... load-bearing? :)
- JKCalhoun 23d ago
  I was hoping for a software "LLM in a Box".

jimbobthemighty 24d ago

I just use Gemini Pro which comes bundled with a Chromebook; when the 'free' year runs out I buy another, initiate the free years Gemini again and then put the 'as new' Chromebook on eBay and get most of my money back. So Gemini Pro costs me less than £2 a month. And for my needs (investment research) that works well and is a pretty cheap compromise.

VCFundedGenYer 24d ago
You didn't read the article nor understand the purpose. The way you are using it uses the internet and datacenters. It is costly to the environment. Running locally is a significant cost savings in comparison.

manmal 24d ago

If author runs the local model for privacy reasons, then I don't understand why they give Telegram access to all their conversations. It's well known that Telegram doesn't end-to-end encrypt bot accounts.

phazonoverload 24d ago
The beauty of this is that you can just swap out the platform and everything remains as it's the same backend. You make a really good point, one that I haven't really considered, but I also only have so many hours in the day to be fiddling around with stuff.
- manmal 24d ago
  I think I'd use Discord instead. Clankers are happy to set it all up for you.
  - sunaookami 23d ago
    Discord is even worse than Telegram in the privacy aspect though!?
    - manmal 22d ago
      Depends what actors you trust. They are equally bad but personally I‘m way more comfortable with the Discord story.

NikhilChowdaryG 24d ago

[flagged]

saejox 23d ago

Performance or quailty of the llm is not the draw for me. I read those blog posts to remind of wealth gap i have with the average hackernews user. Thank you for the encouragement, i will work harder to reach your level.