12 ms·
It has A19 Pro. A19 Pro has matmul acceleration in its GPU, the equivalent of Nvidia's Tensor cores. This would make future Macs extremely viable for local LLMs
by aurareturn 1y ago
It has A19 Pro. A19 Pro has matmul acceleration in its GPU, the equivalent of Nvidia's Tensor cores. This would make future Macs extremely viable for local LLMs. Currently, Macs have high memory bandwidth and high VRAM capacity but low prompt processing speeds. Give it a large context and it'll take forever before the first token is generated.
If the M5 generation gets this GPU upgrade, which I don't see why not, then the era of viable local LLM inferencing is upon us.
That's the most exciting thing from this Apple's event in my opinion.
PS. I also like the idea of the ultra thin iPhone Air, the 2x better noise cancellation and live translation of Airpods 3, high blood pressure detection of the new Watch, and the bold sexy orange color of the iPhone 17 Pro. Overall, this is as good as it gets for incremental updates in Apple's ecosystem in a while.
- astrange 1y agoA19 supports MTE: https://news.ycombinator.com/item?id=45186265 https://news.ycombinator.com/item?id=45186265 Which is a very powerful feature for anyone who likes security or finding bugs in their code. Or other people's code. Even if you didn't really want to find them.
- rising-sky 1y agoMIE
- philodeon 1y agoMIE is a combination of enhanced MTE (EMTE) and some highly-overdue software allocator improvements.
- tucnak 1y agoIt certainly took them a while to introduce MTE! Pixel 8 came out in 2023. I wonder how it compares against hardened_malloc with 48-bit address space and 33-bit ASLR in Graphene. Apple's security team has reported that MIE could break all "known" exploit chains, but so does hardened_malloc. Hard to tell right now which one is best (most def MIE) but everything else included in Graphene is probably making the point moot anyway.
- rising-sky 1y agoYes, but it is not MTE, they are technically different. That's what I was attempting to point out but thought it may have been a typo
- baybal2 1y ago[dead]
- sercand 1y agoWhere did you see the matmul acceleration support? I couldn't find this detail online.
- aurareturn 1y agoApple calls it "Neural Accelerators". It's all over their A19 marketing.
- emchammer 1y agoDoes this mean that equivalent logic for what has been called Neural Engine is now integrated into each CPU core?
- rmccue 1y agoEach GPU core, but yes, this was part of what they announced today - it’s now integral rather than separate.
- kridsdale3 1y agoWhat a ridiculous way to market "linear algebra transistor array".
- jacquesm 1y agoHey man, it helps you think different. You just never knew your neurons needed accelerating.
- kridsdale1 1y agoI accelerate them every morning with an Americano.
- liamwire 1y agoI have to ask out of curiosity, why is your first comment made with one account, and the reply with a similarly-named alt?
- Nokinside 1y agoThe first SoC including Neural Engine was the A11 Bionic, used in iPhone 8, 8 Plus and iPhone X, introduced in 2017. Since then, every Apple A-series SoC has included a Neural Engine.
- runjake 1y agoThe matmul stuff is part of the Neural Accelerator marketing, which is distinct from the Neural Engine you're talking about. I don't blame you. It's confusing.
- Nokinside 1y agoIt's remaining and rearrangement of the same stuff. Not a new feature.
- runjake 1y ago1. It adds new features. Eg. see matmul and other to-be-detailed-soon features. 2. It moves some stuff from the external Neural Engine to the GPU, which substantially increases speeds for those workloads. That itself is a feature. Will any of this really matter much to the average consumer at this point? Probably not. Not until Apple Intelligence gets off the ground.
- aurareturn 1y agoThe NPU is still there. This adds matmul acceleration directly into each GPU core. It takes about ~10% more transistors to add these accelerators into the GPU so it's a significant investment for Apple.
- aurareturn 1y agoThe Neural Engine is its own block. Neural Engine is not used for local LLMs on Macs. Neural Engine is optimized for power efficiency while running small models. It's not good for LARGE language models. This change is strictly adding matmul acceleration into each GPU core where it is being used for LLMs.
- baby 1y agoIMO it's underwhelming considering folding phones have been out for many years now and we still don't have a folding iPhone. What are the PMs doing at Apple.
- boppo1 1y agoFolders seem gimmicky to me
- baby 1y agoUntil you use one
- dmix 1y agoI tested the Samsung one in the store and that groove thing in the screen would drive me crazy
- baby 1y agoI have a google fold 9 for a year and I've never noticed it unless I look at the phone from the side. It's interesting that this is the criticism that comes up the most here where it's already been a solved issue
- NetMageSCW 1y agoYou think it’s solved - many people disagree.
- baby 1y agoWho havent used one
- pwthornton 1y agoWhat’s the main use case of this?
- Aperocky 1y agoSo.. 6 hour batteries like the Apple Watch?
- mbirth 1y agoBut there’s this now: https://store.apple.com/uk/xc/product/MGPG4ZM/A https://store.apple.com/uk/xc/product/MGPG4ZM/A
- stephenlf 1y agoIt’s embarrassing how frequently they’ll yank out an important part and sell it as an add-on.
- mbirth 1y agoYes, and then don’t even make it compatible with other phones. I’m a big fan of their previous (discontinued) MagSafe battery as that supports reverse charging, charge state display on phone and has the perfect size. This new battery however is only compatible with the Air as other phones have a bigger camera bump.
- badc0ffee 1y agoExcuse me, that hump is the Iconic Plateau.
- numpad0 1y agoSignifying flatlining iPhone?
- justinator 1y agowhy wouldn't it be compatible with other phones?
- mbirth 1y agoSee this photo: https://store.storeimages.cdn-apple.com/1/as-images.apple.com/is/MGPG4_AV2?wid=1144&hei=1144&fmt=jpeg&qlt=90&.v=Y0p5KzBmSlV2enJ0MFZpdW9CUlhlVlZya2lKWlJmUEwrYndWOTJiVWJWQUYwVmtIbGRkS25RMVpBRlo0bk5DUXAvZmIyYWJNNGo4aFd6czRKNXFqMEE https://store.storeimages.cdn-apple.com/1/as-images.apple.co... The camera bump on other models protrudes more towards the centre of the body. And thus the battery wouldn't fit (flush) and the Qi charging wouldn't engage properly.
- commandersaki 1y agoHoping this budget macbook rumour based on A19/A19 Pro is real.
- cj 1y agoIsn’t the MacBook Air already pretty cheap at $999?
- toxican 1y ago$998, $997, etc.
- randallsquared 1y agoA couple more examples would have been nice; it took a while to understand your point.
- flkenosad 1y agoMade my day lol thanks
- triceratops 1y agoConsidering inflation $999 is cheaper.
- commandersaki 1y agoCompeting at high end budget PC laptop will increase market share, but more importantly it’d be an easy recommendation to make when people see a lower priced PC laptop. Rumours were $699.
- alwillis 1y agoSure but the one he's referring to is rumored to cost $599-$699.
- babl-yc 1y agoI've always been a bit confused about when to run models on the GPU vs the neural engine. The best I can tell, GPU is simpler to use as a developer especially when shipping a cross platform app. But an optimized neural engine model can run lower power. With the addition of NPUs to the GPU, this story gets even more confusing...
- avianlyric 1y agoIn reality you don’t much of a choice. Most of the APIs Apple exposes for running neural nets don’t let you pick. Instead some Apple magic in one of their frameworks decides where it’s going to host your network. At least from what I’ve read, these frameworks will usually distribute your networks over all available matmul compute, starting on the neural net (assuming your specific network is compatible) and spilling onto the GPU as needed. But there isn’t a trivial way to specifically target the neural engine.
- babl-yc 1y agoYou're right there is no way to specifically target the neural engine. You have to use it via CoreML which abstracts away the execution. If you use Metal / GPU compute shaders it's going to run exclusively on GPU. Some inference libraries like TensorFlow/LiteRT with backend = .gpu use this.
- scosman 1y agoExactly. And most folks are using a framework like llama.cpp which does control where it’s run.
- Uehreka 1y agoI will believe this when I see it. It’s totally possible that those capabilities are locked behind some private API or that there’s some weedsy hardware complication not mentioned that makes them non-viable for what we want to do with them.
- llm_nerd 1y agoThey might recommend using CoreML to leverage them, though I imagine it will be available to Metal. The whole point of CoreML is that your solution uses whatever hardware is available to you, including enlisting a heterogeneous set of units to conquer a large problem. Software written years ago would use the GPU matmul if deployed to a capable machine.
- aurareturn 1y agoAlready available via Metal: https://x.com/liuliu/status/1932158994698932505 https://x.com/liuliu/status/1932158994698932505
- deleted 1y ago[deleted]
- AdventureMouse 1y ago> If the M5 generation gets this GPU upgrade, which I don't see why not, then the era of viable local LLM inferencing is upon us. I don't think local LLMs will ever be a thing except for very specific use cases. Servers will always have way more compute power than edge nodes. As server power increases, people will expect more and more of the LLMs and edge node compute will stay irrelevant since their relative power will stay the same.
- rowanG077 1y agoThat's assuming diminishing returns won't hit hard. If a 10x smaller local model is 95%(Whatever that means) as good as the remote model it makes sense to use local models most of the time. It remains to be seen if that will happen but it's certainly not unthinkable imp.
- sigmar 1y agoIt's really task-dependent, text summarization and grammar corrections are fine with local models. I posit any tasks that are 'arms race-y' (image generation, creative text generation) are going to be offloaded to servers, as there's no 'good enough' bar above which they can't improve.
- MPSimmons 1y agoThe crux is how big the L is in the local LLMs. Depending on what it's used for, you can actually get really good performance on topically trained models when leveraged for their specific purpose.
- rickdeckard 1y agoThere's alot of L's in LLLM, so overall it's hard to tell what you're trying to say... Is it 'Local'?, 'Large?'...'Language?'
- touristtam 1y agoDo you see the C for Cheap in there? Me neither.
- deleted 1y ago[deleted]
- ActorNightly 1y agoGood luck actually getting access to ANE. There is a reason why Pytorch doesn't use it even if its been around for a while.
- aurareturn 1y agoNo luck needed. This is for the GPU and is already available via Metal. https://x.com/liuliu/status/1932158994698932505 https://x.com/liuliu/status/1932158994698932505
- ActorNightly 1y agoIll believe it when I see it supported in Pytorch
- mgerdts 1y agoIf you compare the specs of the 10 and 11 series watches you will see they both claim high blood pressure detection. https://www.apple.com/watch/compare/?modelList=watch-series-11,watch-series-10 https://www.apple.com/watch/compare/?modelList=watch-series-... In the past few weeks the oxymeter feature was enabled by a firmware update on series 10. Measurements are done on the watch, results are only reported on a phone.
- sgustard 1y agoGood to know! The fine print: As of September 9, 2025, hypertension notifications are currently under FDA review and expected to be cleared this month, with availability on Apple Watch Series 9 and later and Apple Watch Ultra 2 and later. The feature is not intended for use by people under 22 years old, those who have been previously diagnosed with hypertension, or pregnant persons.
- tartrate 1y ago> [hypertension notifications] is not intended for use by people [...] who have been previously diagnosed with hypertension Sounds a bit ironic but I guess it's for legal reasons.
- pixiemaster 1y agolegal, and also: if you already have been diagnosed, you should already be under medical professional supervision (meds, checkups,…) anyway. my guess is this is more like the heart irregularities feature: it’s for the first diagnosis. (a relative of mine actually got diagnosed that way)
- kalap_ur 1y agoI believe this is for the fringe cases where you have been diagnosed with hypertension, but your apple Watch does not tell you that you have hypertension risk, then you may decide to not take your drugs, since your watch told you all clear. This could trigger lawsuits if complications set in when you decide not to take your drugs because of "lack of alarm"
- zumu 1y ago> the bold sexy orange color of the iPhone 17 Pro The color line up reminds me of the au MEDIA SKIN phones (Japanese carrier) circa 2007. Maybe it's because I had one back in the day, but I can't help but think they took some influence.
- user_7832 1y ago> MEDIA SKIN phones Wow, thanks for sharing the name, these are really good! I don't know why I was surprised to realize that great designers have made fantastic products even in the past... Some sites with images, for anyone curious: 1. https://www.dezeen.com/2007/01/17/tokujin-yoshioka-launches-mobile-phone/ https://www.dezeen.com/2007/01/17/tokujin-yoshioka-launches-... 2. https://spoon-tamago.com/best-of-2007-part-iv/ https://spoon-tamago.com/best-of-2007-part-iv/
- zumu 1y agoIf you like these, check out the INFOBAR phones from a few years prior. https://platform.theverge.com/wp-content/uploads/sites/2/chorus/uploads/chorus_asset/file/6562591/iida-infobar-naoto-fukasawa-7.0.jpg https://platform.theverge.com/wp-content/uploads/sites/2/cho... People like the multi-colored one, but I've always been partial to the green. I believe there's been a few newer homages to these over the years.
- supportengineer 1y agoI was reminded of this today for no particular reason: "iPhone4 vs HTC Evo" https://www.youtube.com/watch?v=FL7yD-0pqZg https://www.youtube.com/watch?v=FL7yD-0pqZg
- SilverElfin 1y agodeleted
- WanderPanda 1y agoIt was 4x over the original version IIRC so should be ~ 2x over the previous
- apparent 1y agoAccording to this page, [1] it reduces unwanted noise 4x as much as the original AirPods Pro and 2x as much as the AirPods Pro 2. Though I do wonder, given the logarithmic nature of sound perception, are these numbers deceptive in terms of what the user will perceive? 1: https://www.apple.com/airpods-pro/ https://www.apple.com/airpods-pro/
- vasco 1y ago> bold sexy orange color Luckily they added the blood pressure check for when you get too excited about the color orange.
- formerly_proven 1y agoIt is almost strange, since iPhones were only available in ugly drab colors for several generations. And the Pro models in particular were previously never available in a decent color.
- steve_adams_86 1y agoThe 15s and 16s both had titanium bodies which (as I recall at least) don't take on colour as well when they're anodized, so that could be the cause of drab colour ways. edit: It was only the Pros and up which had titanium bodies. The 17s are all aluminum.
- formerly_proven 1y agoBut even the non-Pro phones had mostly ugly colors in the last couple years. Maybe to match the ugliness of the Pro models?
- jachee 1y agoA realization that I came to today is the fact that I’m still on my 12 because it’s (one of the?) last PRODUCT(RED) one. The Sage Air has my eye. Would match my AirPods Max. But that orange pro is also calling.
- mauvehaus 1y agoAnodizing titanium creates an oxide layer, the thickness of which varies with the voltage used. The thickness of that oxide layer determines which wavelengths of light it refracts [0]. In practical terms, your choices for color are pretty limited[1]. I'm not a chemist, but I looked into this years back when I was wondering why everything titanium is offered in the same couple of colors. Personally, I like the plain gray. [0] https://wisensemachining.com/titanium-anodizing-guide/ https://wisensemachining.com/titanium-anodizing-guide/ [1] https://www.snowpeak.com/collections/cups/products/ti-single-450-cup?variant=32400810442796 https://www.snowpeak.com/collections/cups/products/ti-single...
- whyenot 1y agoI wish they would offer the 17 pro in some lighter colors (like the new sage green for the regular 17). Not everyone wants bold, and the color selection for pro is always so limited. They don't even have white with this generation, just silver.
- amelius 1y ago> It has A19 Pro. But it's not general purpose. Broken by design. I'll pass. Not going to support this. We need less of this crap not more.
- atcon 1y agoViable may be already here: demo of smollm3/3b <https://news.ycombinator.com/item?id=44501413 https://news.ycombinator.com/item?id=44501413> on iphone with asr + tts: <https://x.com/adrgrondin/status/1965097304995889642 https://x.com/adrgrondin/status/1965097304995889642> Intrigued to explore with a19/m5 and test energy efficiency.
- chisleu 1y agoBecause of the prompt processing speed, small models like Qwen 3 coder 30b a3b are the sweet spot for mac platform right now. Which means a 32 or 64GB mac is all you need to use Cline or your favorite agent locally.
- DrAwdeOccarim 1y agoYes, I use LM Studio daily with Qwen 3 30b a3b. I can't believe how good it is locally.
- paool 1y agoCan you use your Qwen instance in CLIs like Claude code, codex, or whatever open source coding agent? Or do you have to copy paste into LM studio?
- DrAwdeOccarim 1y agoLM Studio lets you run a model as a local API (OpenAI-compatible REST server).
- evilduck 1y agoYeah you can, so long as you're hosting your local LLM through something with an OpenAI-compatible API (which is a given for almost all local servers at this point, including LM Studio). https://opencode.ai https://opencode.ai and https://github.com/QwenLM/qwen-code https://github.com/QwenLM/qwen-code both allow you to configure any API as the LLM provider. That said, running agentic workloads on local LLMs will be a short and losing battle against context size if you don't have hardware specifically bought for this purpose. You can get it running and it will work for several autonomous actions but not nearly as long as a hosted frontier model will work.
- TrajansRow 1y agoUnfortunately, IDE integration like this tends to be very prefill intensive (more math than memory). That puts Apple Silicon at a disadvantage without the feature that we’re talking about. Presumably the upcoming M5 will also have dedicated matmul acceleration in the GPU. This could potentially change everything in favor of local AI, particularly on mobile devices like laptops.
- aagha 1y agoApple is playing 3D chess while every other PC maker is learning how to play checkers.
- bendoy 1y agoI'm most excited about the heart rate sensor in Airpods Pro 3!
- ottah 1y agoNah, memory is still the bottleneck. Kernel performance is already pretty good, but cpu memory is still dramatically slower than gpu memory.
- SirMaster 1y agoThe live translation is software. It works on the AirPods Pro 2 and the AirPods 4 with AND. So is the high blood pressure detection. It's not from the new watch, it works also in the series 10 and series 9 watches.