5 ms·
I think its just marketing, and the marketing is working. Look how many people bought Minis and ended up just paying for API calls anyway. (Saw it IRL 2x, see i
by butILoveLife 8mo ago
I think its just marketing, and the marketing is working. Look how many people bought Minis and ended up just paying for API calls anyway. (Saw it IRL 2x, see it on reddit openclaw daily)
I don't mind it, I open Apple stock. But I'm def not buying into their rebranding of integrated GPU under the guise of Unified Memory.
- Hamuko 8mo agoI've tried to use a local LLM on an M4 Pro machine and it's quite painful. Not surprised that people into LLMs would pay for tokens instead of trying to force their poor MacBooks to do it.
- giancarlostoro 8mo agoWhat are the other specs and how's your setup look? You need a minimum of 24GB of RAM for it to run 16GB or less models.
- SV_BubbleTime 8mo agoThis is typically true. And while it is stupid slow, you can run models of hard drive or swap space. You wouldn’t do it normally, but it can be done to check an answer in one model versus another.
- Hamuko 8mo ago48 GB MacBook Pro. All of the models I've tried have been slow and also offered terrible results.
- giancarlostoro 8mo agoTry a software called TG Pro lets you override fan settings, Apple likes to let your Mac burn in an inferno before the fans kick in. It gives me more consistent throughput. I have less RAM than you and I can run some smaller models just fine, with reasonable performance. GPT20b was one.
- jazzyjackson 8mo agoTokens per second is abysmal no matter how much ram you have
- giancarlostoro 8mo agoSome models run worse than others but I have gotten reasonable performance on my M4 Pro with 24 GB of RAM
- atwrk 8mo agoLocal LLM inference is all about memory bandwidth, and an M4 pro only has about the same as a Strix Halo or DGX Spark. That's why the older ultras are popular with the local LLM crowd.
- freeone3000 8mo agoI’m super happy with it for embedding, image recog, and semantic video segmentation tasks.
- andoando 8mo agoLocal LLMs are useful for stuff like tool calling
- renewiltord 8mo agoWhat models are you using? I’ve found that SOTA Claudes outperform even gpt-5.2 so hard on this that it’s cheaper to just use Sonnet because num output tokens to solve problem is so much lower that TCO is lower. I’m in SF where home power is 54¢/kWh. Sonnet is so fast too. GPT-5.2 needs reasoning tuned up to get tool calling reliable and Qwen3 Coder Next wasn’t close. I haven’t tried Qwen3.5-A3B. Hearing rave reviews though. If you’re using successfully some model knowing that alone is very helpful to me.
- usagisushi 8mo agoQwen 3.5 35B-A3B and 27B have changed the game for me. I expect we'll see something comparable to Sonnet 4.6 running locally sometime this year.
- prettyblocks 8mo agoCould be, but it likely won't be able to support the massive context window required for performance on par with sonnet 4.6
- OtomotO 8mo agoThis would be an absolute game changer for me. I am dictating this text now on a local model and I think this is the way to go. I want to have everything locally. I'm not opposed to AI in general or LLMs in general, but I think that sending everything over the pond is a no-go. And even if it were European, I still wouldn't want to send everything to some data center and so on. So I think this is a good, it would be a good development and I think I would even buy an Apple device for the first time since the iPod just for that.
- jsheard 8mo ago> Look how many people bought Minis and ended up just paying for API calls anyway. (Saw it IRL 2x, see it on reddit openclaw daily) Aren't the OpenClaw enjoyers buying Mac Minis because it's the cheapest thing which runs macOS, the only platform which can programmatically interface with iMessage and other Apple ecosystem stuff? It has nothing to do with the hardware really. Still, buying a brand new Mac Mini for that purpose seems kind of pointless when a used M1 model would achieve the same thing.
- ErneX 8mo agoIt’s exactly that. They are buying the base model just for that. You are not going to do much local AI with those 16GB of ram anyway, it could be useful for small things but the main purpose of the Mini is being able to interact with the apple apps and services.
- chaostheory 8mo agoNo one is buying a base model Mac for local LLM. Everyone is forgetting the PC prices have drastically increased due to RAM and SSD. Meanwhile, Macs had no such price change… at least for the models that didn’t just drop today. Mac’s are just a good deal at the moment.
- briffle 8mo agothe new models cost $200 more for each 8GB of Ram you add.. Ouch...
- Forgeties79 8mo agoThat's been the case for years. Not new to the M5's
- jsheard 8mo ago> Meanwhile, Macs had no such price change Yeah because Mac upgrade prices were already sky high, long before the component shortage. 32GB of DDR5-6000 for a PC rocketed from $100 to $500, while the cost of adding 16GB to a Mac was and still is $400.
- mleo 8mo agoMy M4 MacBook Pro for work just came a few weeks ago with 128 GB of RAM. Some simple voice customization started using 90GB. The unified memory value is there.
- rafram 8mo agoWhy not? The integrated GPUs are quite powerful, and having access to 32+ GB of GPU memory is amazing. There's a reason people buy Macs for local LLM work. Nothing else on the market really beats it right now.
- threatofrain 8mo agoThe biggest problem with personal ML workflows on Mac right now is the software.
- cmdrmac 8mo agoI'm curious to know what software you're referring to.
- csullivannet 8mo agoYes
- lizknope 8mo agoJeff Geerling had a video of using 4 Mac Studios each with 512GB RAM connected by Thunderbolt. Each machine is around $10K so this isn't cheap but the performance is impressive. https://www.youtube.com/watch?v=x4_RsUxRjKU https://www.youtube.com/watch?v=x4_RsUxRjKU
- Greed 8mo agoIf 40k is the barrier to entry for impressive, that doesn't really sell the usecase of local LLMs very well. For the same price in API calls, you could fund AI driven development across a small team for quite a long while. Whether that remains the case once those models are no longer subsidized, TBD. But as of today the comparison isn't even close.
- ttoinou 8mo agoWith M3 Max with 64GB of unified ram you can code with a local LLM, so the bar is much lower
- Greed 8mo agoBut why? Spending several thousand dollars to run sub-par models when the break-even point could still be years away seems bizarre for any real usecase where your goal is productivity over novelty. Anyone who has used Codex or Opus can attest that the difference between those and a locally available model like Qwen or Codestral is night and day. To be clear, I totally get the idea of running local LLMs for toy reasons. But in a business context the sell on a stack of Mac Pros seems misguided at best.
- 0x457 8mo agoI started doing it to hedge myself for inevitable disappearance of cheap inference.
- robotresearcher 8mo agoSometimes you can't push your working data to third party service, by law, by contract, or by preference.
- tcmart14 8mo agoI'm not really into AI and LLMs. I personally don't like anything they output. But the people I know who are into it and into running their own local setups are buying Studios and Minis for their at home local LLM set ups. Really, everyone I personally know who is doing their build your own with local LLMs are doing this. I don't know anyone anymore buying other computers and NVIDIA graphics cards for it.
- 0x457 8mo agoI think people buying those don't realize requirements to run something as big as Opus, they think those gigabytes of memory on Mac studio/mini is a lot only to find out that its "meh" on context of LLMs. Plus most buy it as a gateway into Apple ecosystem for their Claws, iMessage for example. > But I'm def not buying into their rebranding of integrated GPU under the guise of Unified Memory. But it is Unified Memory? Thanks to Intel iGPU term is tainted for a long time.