12 ms·
Apple wants AI to run directly on its hardware instead of in the cloud
- deleted 3y ago[deleted]
- tech_ken 3y ago> “As those things get scaled up, they start to make a meaningful change in the user experience and enable new innovation which has the potential to create a new upgrade cycle in smartphones.” Interesting read. "Create a new upgrade cycle" is a phrase that's almost too on the nose, but will be interesting to see if the AI-powered feature set genuinely justifies an upgrade. I have a hard time imagining someone wanting to shell out 4 digits to be able to run ex. stable diffusion locally if they weren't already in the market for a new phone, but maybe I'm underestimating the AI hype wave right now. I could see predictive typing or improved voice assistants being the two biggest opportunities for a killer feature, but even then not sure if the value proposition would change that much, especially if the battery life takes any hit whatsoever.
- msoad 3y agoSiri that actually works sounds very appealing to be honest. All the integration and UX is there. It just doesn’t work today. So an LLM based Siri that can understand my request is worth an upgrade (for me)
- merpnderp 3y agoExactly, just figure out how to run Whisper on my phone and it would be the best phone on the market. Whisper plus Mixtral on my phone and it could do anything.
- rollcat 3y agoOn the contrary, Siri already works perfectly (or rather, used to work) for a lot of things... It's just Apple keeps on randomly changing stuff, and every time they change something, shit breaks. "Hey Siri, good evening" - "I turned on the `good morning` scene." "Hey Siri, timer five minutes" - "I set an alarm for seventeen hundred hours" "Hey Siri, weather" - turns on the lights in the bathroom I wish I was joking or exaggerating... The last thing I need is an LLM, just please fix the obvious problems and then maybe iterate on that
- ceejayoz 3y agoYeah, I'll get random weeks in which "text <wife>" works fine, and other weeks where it says "which one?" and gives me her email and phone as different contacts. No changes on my end that'd cause it.
- kstrauser 3y agoMy wife is "Jen". I used to work with "Joe Chan". To this day, if I say "text Jen ...", there's about a 20% chance it'll reply with "OK, saying ... to Joe Chan." Dude. How about optimizing for the option that I've actually communicated with this decade, and 4 times already today?
- raccoonDivider 3y agoMakes me wonder how voice assistants handle people with multiple partners. Maybe "which one" is actually the right answer for some users?
- oidar 3y agoAdditionally, in a multi-person household adding things to a shopping list should take any authentication. Not everybody has an iDevice or needs to have an iDevice - my petsitter, babysitter, maid and random family members should be able to add stuff to the shopping list just because they are in the room with the homePod. On Alexa, it works - but we have to deal with all the advertisements for crappy AliExpress junk. As soon as Apple figures out shopping lists for whoever is in the room, I'll be dropping all my Alexa devices.
- VogonPoetry 3y agoNot sure exactly which way round you are asking -- should requests be tied to the person or be generic? In the HomePod settings there is a section for "Personal Requests". When enabled, adding reminders will then go to the device / account of the person speaking. I'm not sure what the behavior is when set to off. You can change the setting for each HomePod.
- n9 3y agoNot sure what you're needing that you're note getting -- there is certainly a lot that it doesn't do -- but Siri works like a champ for me these days for maybe 70% of the things that I would do with my phone or ever computer without it. I initiate all calls with it, do tons of texting with it, cue up all my music in the car with it, constantly add things to reminders lists ('hey siri... add soy sauce to the grocery list' results in soy sauce on my grocery list... in the condiments sublist) I hadn't noticed the improvements until I started using my apple watch a lot... I had given up on it a few years back, I guess.
- ForkMeOnTinder 3y agofwiw Google is attempting the same thing, but they're not doing much better. https://www.notebookcheck.net/MrWhosetheboss-video-reveals-Google-s-Pixel-8-Pro-Tensor-G3-off-loads-all-generative-AI-tasks-to-the-cloud.760215.0.html https://www.notebookcheck.net/MrWhosetheboss-video-reveals-G... > Google's Pixel 8 Pro Tensor G3 off-loads all generative AI tasks to the cloud
- rainclouds 3y agoCritical. I’d like my data to stay local. I probably don’t need cloud processing for any regular usage. The privacy to life benefit equation is broken in society atm. This will change.
- tech_ken 3y agoIt didn't seem like training will occur locally, just serving prompt responses from a GPT model (maybe I misunderstood the article). My personal privacy calculus places a lot of less value on keeping prompts local, especially if some kind of E2EE can be provided between me and the serving model. I understand that not everyone feels this way, but from the perspective of market adoption I don't see the average user caring too much whether a prompt is kept local or sent to the cloud (if encrypted or similar). To me it seems like the most critical determinants would be response latency and battery life, plus the basic performance of the thing.
- mitjam 3y agoNVIDIA‘s newest GPUs support confidential computing which can be used for E2EE but also for intellectual property protection. https://www.nvidia.com/en-us/data-center/solutions/confidential-computing/ https://www.nvidia.com/en-us/data-center/solutions/confident...
- tech_ken 3y agoIf it's just serving prompt responses I'm not sure if the IP protection is a huge dealbreaker for me as a downstream user. The most critical use case I can currently imagine is better predictive text for my keyboard, and that one seems pretty light on the IP violations (for me personally). The biggest thing for me here is E2EE if the text is leaving my phone.
- hx8 3y agoThere aren't many more single new features that are going to justify upgrading that will exist on the mobile platform. Apple is playing a game where they try to add enough features to prevent the upgrade cycle from becoming too long. Encourage people with 2 or 3 year old phones to upgrade this cycle instead of delaying a generation. Honestly, Apple's business model is harder if everyone wants to upgrade to a certain new version. The M1 Macs were so good, everyone upgraded their Intel Macs and then mac sales dropped. It's better logistically if the sales are more predictable and consistent year-to-year.
- tech_ken 3y agoThat's an interesting point, so more of a game of amortizing upgrades over a few years rather than instigating a massive one-off purchasing craze. I could see how a drip of sufficiently useful AI features and improvements over like '24-'27 would bolster the market. Even then though, this seems like an opportunity to hold rather than really reverse the trend; at some point I think Apple is going to have to reckon with the fact that the market is more or less saturated, though I could definitely be wrong here.
- hx8 3y agoApple knows the cell phone market is saturated. That's why they have iPads, AirPods, Apple Watches, MacBook Airs, etc. Their plan is to create new accessories (and services) that pair with the iPhone. This generates a moat around existing iPhone users.
- tech_ken 3y agoI guess if they start layering LLMs or similar into their service or accessory interop stack I could see how having the phone running the model could be a big win for latency and availability. Like if Apple Calendar uses a GPT to create a .ics from the invitation text my friend sent me then it could be very handy to do that locally, moreso if I'm using my Apple Watch to actually signal that I want the .ics created.
- n9 3y ago
- yieldcrv 3y ago7B parameter language models running on metal work really well Other kinds of transformers could too Fix Apple autocorrect and detect replacements for misspelled words better
- deleted 3y ago[deleted]
- TechRemarker 3y agoOf course they “want” it to just like they want Siri and everything else to be computed locally since best for privacy and offline performance. Whether they compete with a cloud AI time will tell. Will presumably be far more limited than ChatGPT 4 but hopefully light years better than Siri which is what is most important.
- dade_ 3y agoAlso light years ahead when the cloud access isn’t available or spotty in an elevator, the middle of the lake, a parkade, travelling with cellular turned off and wifi is on.
- GenericPoster 3y ago> Whether they compete with a cloud AI time will tell. But they're not competing with cloud AI. Why would a person need to go to the cloud to give you a reminder or download an app? They're competing against the current local assistant, Siri. Large models are great but they can't fit on 8 or 16gb of ram. And that's a very big deal.
- theshrike79 3y agoThey don't need to put all of the world's information locally, just the relevant bits. It doesn't need to know every celebrity's full history for example You can have the basic stuff on-device with the "smarts" of an LLM that can have conversations with the user and have context to previous questions. The other stuff can be fetched from the cloud (with the user's permission OFC) and optionally saved locally.
- theshrike79 3y agoYou can make a local model contact the cloud for specific questions like current events. You can't make a cloud model work 100% locally for privacy or during connection issues.
- mg 3y agoThis brings up the question how efficient the human brain is. As AI keeps catching up more and more with human intelligence, will it also need more and more hardware? Can it achieve human and superhuman intelligence and still run on a device like the iPhone, which weights about 170g? The human brain is 8 times heavier.
- fumar 3y agoDid our brains optimize for size, weight, and or power efficiency?
- acchow 3y agoThe iPhone 15 neural engine does 35Tflops, which is about on par with the fastest supercomputer from 2002. Maybe in 24 years the iPhone 39 will have the computing power that OpenAI has today? Hopefully our compression gets much better in the meantime.
- filterfiber 3y ago> still run on a device like the iPhone, which weights about 170g [1]? The human brain is 8 times heavier. Why are you comparing the weight? I'm sorry but this is a bizarre comparison. This isn't even apples to oranges, this is apples to a telephone pole. I'll also throw in that a single Nvidia H100 is 1200 grams. Unless you have a "Bracket with screws" which will add 20 grams (who wouldn't want an extra 20grams of intelligence?). Like 70%+ of the human brain is water. The human brain needs a massive network of systems to transport nutrients/oxygen which is irrelevant to logical processing. Similarly the majority of the iphone weight is the battery and frame. The weight of the processing chip is _grams_. Besides the conflicting variables with the weight, the way that ML works on a physical level is completely different from the human brain.
- PaulKeeble 3y agoThey are going to want to start putting more RAM in their machines then or they are going to find it a bit tricky to compete given how big even moderate AI is.
- freedomben 3y agoThey will ship hardware (a new iphone for example) that has a co-processor and flash storage dedicated to the Apple model (which many are calling Apple GPT). They aren't going to make it easy to run your own model. That's completely against their style
- MBCook 3y agoThey already allow that. There are apps in the store doing that.
- freedomben 3y agoThere are apps in the app store that are allowed to use the new co-processor that I'm predicting they'll develop? That seems physically impossible since it doesn't exist yet.
- MBCook 3y agoThe Neural Engine was added in the like the A11 a few years ago.
- MBCook 3y agoThey did publish that paper the other day explaining how to use less RAM, but you’re right it’s still a big chunk.
- rgbrgb 3y agoI'm totally aligned with this line of thought and I think running local is the best possible thing to do with all of the privacy risks AI presents. In the future I want my cognition 10x amplified by AI and the only way I can imagine that working is if info is being fluidly exchanged with my brain at the speed of thought. No way am I going to plug into a cloud data sink for advertisers and spy agencies to snoop my unfiltered thoughts. I want my cognition air-gapped on hardware I own.
- tkems 3y agoI think that running AI on device is the clear next step in the world of super powerful portable phones/devices. I have a 2021 model phone with very powerful specs (granted it is the high end model that I got used). With nearly the same RAM and storage as some of my laptops from a few years ago, I see that Apple might want to offload some of their expensive AI apps to these powerful user devices. I see this also as a privacy win as I don't want any of my data being used for training. I also like to have control over what models I use and the ethics the models follow.
- itake 3y agoI am cautiously optimistic, but the reality is these models are huge. LLama 7b is 13g. Even if there is enough cpu cycles, the phone's storage can't support multiple models (and there will need to be multiple models). Compare that to cloud, you can run the latest and greatest model and effectively unlimited storage.
- mejutoco 3y agoBut the cloud would cost apple money, while if ran in your own device they can sell you the hardware for a healthy margin, and market privacy. I think a single useful model on device or many tiny ones are feasible. Even today, the photos app on my iphone knows what pictures are dog pictures, can remove backgrounds, etc.
- josho 3y agoIt would simply become a part of the iCloud+ bundle and contribute to service revenue. Hell, that’s likely what will happen anyway. The models run on device with cloud assistance.
- ynniv 3y agoA reasonable quantization (Q4_K_M) is 4GB on disk and uses 6.5 GB of RAM. It will run at reasonable speed on an iPhone 15 Pro today. Using a smaller model or a more efficient quantization on a phone two years from now (aka whatever's in their lab today) will be easier than running the latest First Person Shooter. Welcome to the future. Edit: I see a few comments about hundreds of GB. That works for storage, but DRAM consumes battery just by existing, and that's been keeping RAM sizes from increasing the way storage has.
- jelled 3y agoI expect that locally installed models will quickly resemble how we think about drivers today. In the short term you'll install a notes app that requires Mixtral-8x7B so you pull that in through ollama. Down the line all operating systems will ship with various models preinstalled.
- rvz 3y agoAs predicted in: [0] [1] [0] https://news.ycombinator.com/item?id=35711680 https://news.ycombinator.com/item?id=35711680 [1] https://news.ycombinator.com/item?id=35581777 https://news.ycombinator.com/item?id=35581777
- conjecTech 3y agoI've noticed that despite Apple's stated on-device aspirations, they've put very little incremental die space toward the neural engine in the most recent generations. I think there's a number of possible explanations for the disconnect. Curious if anyone with a better perspective knows which are true. 1) They think the existing hardware is powerful enough. 2) The utilization of the ANE doesn't justify increased resources. 3) They plan to re-generalize AI compute through things like vector operations. 4) (the most pessimistic) They are saving large increases for future releases when they need to force upgrades. I could see the math for on-device just not working for things like massive LLMs. The amount of silicon you'd need to make it possible would be large, and the frequency of use low. The same silicon for one person's phone could likely support dozens if it were in a datacenter instead.
- TillE 3y agoI'm not clear on what precisely the Neural Engine can do, but Apple has put plenty of resources into improving their GPUs, which can definitely do compute well (see eg, MacWhisper).
- conjecTech 3y agoYeah, it could be that they plan to do #3 through use of their chunky GPUs. There is definitely a lot of focus on the whole APU concept. From look at some of the dies and what projects like ggml have been able to do through mixed computation, I'm guessing the neural engine is ~3-5x the throughput/area vs GPU for the kinds of operations its doing. Trying to map neural nets onto graphics seems reasonable, but personally I would bet on neuralizing graphics to be the better strategy long term.
- ip26 3y agoThere wasn't a "killer app" for on-die AI accelerators until roughly this March. Until about nine months ago, the only widely-discussed ideas were re-aiming pupils to make your video call feel more natural, and applying background filters. So today's on-die AI accelerators were, as far as I can tell, a cautious bet. It turns out caution was warranted, because LLM's need large amounts of memory bandwidth and capacity far more than they need a specialized neural compute engine.
- simonw 3y agoIf you haven't tried it yet, and you have a modern enough phone, I strongly recommend trying out Mistral 7B on your mobile device. I've been running it on an iPhone 15 using this app: https://llm.mlc.ai/#ios https://llm.mlc.ai/#ios - App Store link: https://apps.apple.com/us/app/mlc-chat/id6448482937 https://apps.apple.com/us/app/mlc-chat/id6448482937 The same team have an Android version too which I've not yet tried myself: https://llm.mlc.ai/#android https://llm.mlc.ai/#android On my iPhone it works, and provides very decent performance. The only catch is that it needs pretty much the entire phone's memory to work - so if you switch out of it to another app and back again it resets the state and has to load the model from scratch.
- ComputerGuru 3y ago> so if you switch out of it to another app and back again it resets the state and has to load the model from scratch. This has been my experience anyway ever since upgrading to iOS 16 (and now 17) anyway. Everything is always paging, even on the latest hardware.
- doug_durham 3y agoThere is no paging in iOS. At least paging in the Linux sense. Your apps memory will get evicted when you switch to another app if the system needs it.
- dcdc123 3y agoIt is up to each app to store their state and resume. One of the most annoying apps for not doing this well is Google Maps. Holy crap I hate when I search something and get a route set and tab out to check something and com eback to a fresh Google Maps like 5 seconds later. They will fix it for a while then break it again several times a year. Once you start navigating it seems pretty reliable but until you hit start it is a shitshow.
- kergonath 3y agoThe Google Maps app is atrocious on iOS. So is their website.
- yellow_postit 3y agoApple letting Siri stagnate for so many years is such a shame. I do wonder what foundation model Apple will eventually use. Both training their own as well as licensing a 3P one seem at odds with their claimed data privacy stance. My bet though is a licensed 3p model that’s fine tuned locally with on device data.
- freedomben 3y agoDon't have a link right now, but I read earlier they have a model developed internally already called colloquially "Apple GPT" that (supposedly) rivals OpenAI in parameter size, etc. They'll release a hardware co-processor similar to Google's when they're ready to ship this.
- yellow_postit 3y agoI saw that too but what data did they train on? iMessage, apple photos, Siri inputs? These all seem problematic beyond the standard open crawl type datasets. They don’t have a web search corpus.
- MBCook 3y agoThey could’ve gotten a corpus somehow. Maybe they licensed it from someone, maybe they do have a crawler. It’s a very good question though. That could make all the difference, huh.
- kraig911 3y agoI think it's a great idea but... local will always not be as good as the an AI connected to the cloud. I find myself going from conversations I left off on google/chatgpt/notes between my desktop, phone. Would that it could work offline great but then what of the divergent thoughts brought up in our mutual thinking?
- TillE 3y agoApple's biggest problem here is that they've been consistently stingy with RAM, with only the latest iPhone 15 Pro featuring 8GB. If they want to make a really gutsy move for on-device AI, they could up that to 16GB at the very least. That eats into Tim's precious margins, but it seems like a price worth paying if they want to do exciting, useful things with this tech.
- MBCook 3y agoI do wonder how useful significantly more RAM would be to the majority of iPhone owners until now. I’m sure Apple has tons of metrics collect to decide with. Adding more would certainly be a price/margin issue. Plus DRAM uses battery even if sitting unused, and battery is precious on a phone. Unless they came up with some sort of hot-plug style option to be able to turn off unused RAM chips completely, which would be cool.
- jlokier 3y agoDRAM that is unused according to the OS doesn't need to use battery, as it doesn't need to be refreshed. If this selective no-refresh feature isn't already implemented, I would expect Apple to be working on implementing it in their SoCs. But it's such an obvious and simple power saving feature that I would be surprised and disappointed if it wasn't implemented many years ago.
- w10-1 3y agoIt's crazy to say Apple is late to AI or missing the boat. Yes, OpenAI gets a lot of attention to the extent LLM's could displace search - except search is based on ads from links and selling data, which are even less promising for AI. "Big" AI now is stuck with organization-defining cloud bills for training and huge players are scrambling to push software into hardware. They are barely getting started on the integration. OpenAI has been talking about AGI as it lines up commercial partners across the globe and up and down the stack. But AGI is no more realistic than crypto taking over from central banks. It's possible in theory, but not in fact. Meanwhile, Apple has had neural processors on devices for 4+ years; AI features have figured in every subsequent marketing campaign. And VisionOS augmented reality provides a whole new domain for AI utility, explicitly targeted not just for play, but also for work, as remote work becomes the rule instead of the exception. Plus, Apple is the only safety- and privacy- enforcing ecosystem in existence.
- ErneX 3y agoYes, 1st neural engine was in the A11 Bionic, 2017.
- parineum 3y ago> OpenAI has been talking about AGI as it lines up commercial partners across the globe and up and down the stack. But AGI is no more realistic than crypto taking over from central banks. It's possible in theory, but not in fact. At some level it feels like the pearl clutching about GPT4 as it relates to AGI is astroturfed.
- jakderrida 3y agoI'm inclined to agree with you on that to an extent. When I hear Sam Altman talking about it like it's inevitable and he carries the weight of the world on his shoulders, just seems a little cringey. Don't get me wrong. I like the guy, but it is cringey.
- MBCook 3y agoThe idea that they could create such a thing both enhances their reputation and is something they can use to try and get legislation to prevent anyone from catching up with them. “How could you trust random people to do this safely? It needs to be illegal and something only we can be trusted to make money o… I mean use responsibly.” It’s an attempt at regulatory capture.
- ssijak 3y agoCan they just give an option "I'm OK with my data going to Apple cloud for better LLM inference and better Photo library search, etc". I'm all for local inference, but until it gets good enough on limited devices, I would like to share you my data, please.
- Kon-Peki 3y agoKeep in mind that Apple is charging you money for the hardware that you are using for this inference. You are paying for the electricity that it uses, and you have paid for the storage space and memory. Who pays (and how) if Apple starts doing this for you, on their equipment? Would you pay $5/mo for it?
- MBCook 3y agoIf they did that likely the same as many other Apple services behind how the iPhone currently works: it’s built into the price.
- notaustinpowers 3y agoI'm sure it's part of their ML with the newer iOS but this does make me happy to think of the AI running on the hardware. Yesterday when driving I got a message and usually it would just say "Jane says 'blah blah' and sent a photo." But last night Siri in CarPlay said "Jane says 'blah blah' and sent a photo of a dog in a blanket on a couch". And when I checked the photo they sent it was exactly that. If it's E2E encrypted, I don't think it'd be able to identify that during transit, so it had to have been locally done. Thought it was interesting, and a nice QoL update. Excited to see how this can grow if they introduce more things to be processed with AI on the hardware.
- bayindirh 3y agoI love how Apple obsessively does things on the edge, on the device itself rather than on the cloud. Sometimes my lowly iPhone X gets super hot during charging. Next morning I either get the facial recognition updated, a new memory generated or something else has happened. It's nice, confidence evoking even. The features you like are run on your device, with your data not leaving your device, on its free time. This is the AI I like. Personal, confined and private.
- chatmasta 3y agoThey're in the best possible position to innovate on the literal edge, because they've got a fleet of billions of devices that mostly auto-update. As hardware advances, Apple will have the clearest path to deploying truly personal and locally hosted AI.
- consumer451 3y ago> truly personal and locally hosted AI. If we can combine this with a neuro bill of privacy/rights [0] then the future might not be as bleak as it seems. [0] https://www.cirsd.org/en/horizons/horizons-winter-2021-issue-no-18/its-time-for-neuro--rights https://www.cirsd.org/en/horizons/horizons-winter-2021-issue... > Sean Carroll & Nita Farahany on Ethics, Law, and Neurotechnology https://www.preposterousuniverse.com/podcast/2023/03/13/229-nita-farahany-on-ethics-law-and-neurotechnology/ https://www.preposterousuniverse.com/podcast/2023/03/13/229-...
- dang 3y agoRecent and related: LLM in a Flash: Efficient LLM Inference with Limited Memory - https://news.ycombinator.com/item?id=38704982 https://news.ycombinator.com/item?id=38704982 - Dec 2023 (51 comments)
- madrox 3y agoThis is the possibility I'm most excited for in the AI space. Cloud models are great, but a lot will be unlocked once we get decent models running on consumer devices. If I never have to look at a web page again and use an LLM as my browser, it will be a game changer for the industry.
- nerdbert 3y agoIf nobody is looking at web pages, and therefore nobody is making web pages, then where is the training material for LLMs going to come from?
- tech_ken 3y agoI think that no matter what you're still going to need to connect over WAN to interact with remote services. GPTs are super capable but I don't see them generating arbitrary webpages without at least fetching some basic resources from another machine somewhere. Like if a restaurant updates its menu, that data needs to make its way to the model in one way or another before it can generate their webpage. At the very least I think the "website" might change from an HTML page to an engineered prompt that gets served to you and which generates the page locally. But even in this case it's not clear to me why I'd do that generation on my phone rather than have the server gen the page for me and send the finished product.
- sovietmudkipz 3y agoI kinda wonder if it’s a derisking maneuver. If the AI runs on the client hardware you gain certain characteristics: the compute happens on the client hardware (saving apple $$) and if a bad actor succeeds at jail breaking from the AI controls then only the client hardware is owned (instead of a multi-tenant apple server). Just my naive thoughts
- thih9 3y agoI guess this looks moat shaped. And seems to be aligned well with Apple too: their focus on privacy, their hardware performance gains, their ai assistant products. Apple is good at launching new product categories, ahead of competition. The apple watch had no competition for the first couple of years after launch, ipads are a class of its own, I wonder if they’re planning a similar move with ai too.
- MBCook 3y agoIt could end up a very big advantage. Apple has been using their Neural Engine for a couple of years now, and it’s obviously been noticed. Other companies are now starting to include similar blocks in their CPUs. They could be a match or even better, or they could end up looking like an old integrated graphics card against a discrete GPU. Time will tell us how much of a difference all of this makes.
- mcintyre1994 3y ago> While Microsoft and Google have largely focused on delivering chatbots and other generative AI services over the Internet from their vast cloud computing platforms, Apple’s research suggests that it will instead focus on AI that can run directly on an iPhone. Isn’t there supposed to be a version of Gemini specifically for Pixel phones? It seems like Google see this as a USP for their hardware as well. I wonder how companies like Samsung are thinking about it.
- cmarschner 3y agoThe metal support for pytorch is a real sh*t show at this point - all APIs are either legacy or unfinished. Also, app memory is about 3GB even for the largest phones. It really still feels that it‘s all in its infancies.
- MBCook 3y agoApple already has their own environment in CoreML and the neural engine. They have ways to train/run models on it in highly optimized formats and unified memory. PyTorch came from PCs running on GPUs with their own pool of dedicated memory and a slower link to main memory. I know people have been speeding it up a lot on Apple processors, but I’m not sure it’s a good indicator of how ready the iPhone is for running models for end user programs.
- zmmmmm 3y agoWell of course they do, they have 0% of the cloud hardware market and a gigantic percentage of the phone market. They like money, after all.
- samstave 3y agoPersonally, I believe this is the future. You cant have really cool (tm) space suits with all the telemetry, comms, life support, situ, interfaces, etc... etc... etc... without on-bio AI hardware. obviously its augmented by any Galaxync.
- mark_l_watson 3y agoApple's CoreML libraries have existed for several years, including a BERT model that is useful. I have been expecting Apple to bring more advanced Transformer models than BERT to iOS and iPadOS. My iPad Pro has 16G RAM and I run a 13B model on it. Things are moving fast. About 6 weeks ago I bought a 32G Mac Mini and the models that I can now run 6 weeks later are a huge improvement (e.g., mistral, solar, mixtral:8x7b-instruct, Phi, Yi, etc.) and for local RAG applications and various experiments I feel like I could live with what I have now for a while. My expectations for the future are high!
- KaiserPro 3y agoI think the world and it's dog want to run AI on consumer hardware instead of in the cloud. Its sure as shit cheaper to do (if you exclude the RnD costs.)
- thorum 3y agoApart from the advantages for privacy and Siri, imagine if the OS included a shared system AI that any app could query. That would unlock so many new features that are either impossible or impractical today.
- MBCook 3y agoThat seems kind of like table stakes to me. While individual apps could always include their own model (you can do it today) having a highly optimized pre-trained one in shared storage would certainly be a big benefit compared to everyone trying to stick their own copy of a similar five gig model in their app. Plus it would mean Apple could improve it with OS updates without app makers having to update and ship app updates every single time.
- smrtinsert 3y agoSeems like a sort of win/win. Great for pushing new devices regularly and also good for reliability of the AI based functionality as well as privacy.
- ardel95 3y agoI think Apple is fully capable of getting there too. They have a high enough price point, and a loyal enough customer base, that even if getting there required additional hardware or shipping with more RAM by default, they could make it work.
- 1-6 3y agoNothing is better than inferencing at the edge. For example, Apple now processes Siri locally on the Series 9 and Ultra 2 watches. That’s remarkable given the small size and power of these devices.
- 1-6 3y agoI thought more people already knew about this. It’s quite evident when you see their product announcements.
- contrarian1234 3y agoI don't really get why Apple doesn't go all-in on privacy. They probably make peanuts on advertisement and it's not their core business at all. They have such good PR potential. They can easily paint Google/Meta as blood sucking data vampires that read all your emails and look at all your dick pics. They could paint Android as a data hoovering OS that only works for the benefit of advertisers. Meanwhile Apple.. The good guys.. Encrypt all your data, don't inject JS on every web page , protect you from ads with Safari, and do compute on-device
- troupo 3y agoThat is almost literaly how they have been advertising themselves for the past almost 10 years.
- contrarian1234 3y agoThey don't make any commitments about user data, and they engage in varying degrees of advertising
- MBCook 3y agoI think you can turn off targeted advertising from Apple, although there’s still untargeted advertising. As a user you can choose to lock (almost) everything in iCloud so that Apple can’t see it, but that comes with the trade-off that they can’t help you if you lose your access. It will be permanently gone.
- paxys 3y agoGemini Nano is on it way. Various other 3B-7B models are getting small and efficient enough to run on consumer hardware. It has been clear for a while that the "edge" is the new frontier for LLMs.
- J_Shelby_J 3y agoThat’s nice. Anyways, i had fun building a Linux server to stick an nvidia GPU in. I’m sure it’s going to be pretty common over the next few years as devs realize they can’t just do everything with a MacBook. (Or maybe apple will add support for 3rd party GPUs like they had with the intel MacBooks) This was my take away from the apple AR headset; it doesn’t matter how nice the UX is if it lacks the compute to do anything interesting… and AR/VR takes a LOT of compute.
- MBCook 3y agoSure no one‘s going to be running 16 gig models on their iPhone anytime soon. And my $1500 MacBook is not gonna compete with a dedicated $2000 Nvidia card drawing hundreds of watts. But you don’t need all that. There’s must be tons of great uses that would benefit me as a user that don’t have requirements that steep. My devices will never compete with a data center, there will always be things that have to be hosted elsewhere. But there’s still benefit to be had.
- carabiner 3y agoIt's still amazing how much ML already does on iPhone. OCR in photos, subject identification in photos work incredibly well, whereas Android phones have to do it via cloud.
- MBCook 3y agoThe on device speech recognition has gotten significantly better too.
- deleted 3y ago[deleted]
- olliej 3y ago"AI in the cloud" with user data requires users to give up any pretense of privacy so it's amazing to me that people are ok with this? If you build "AI" into an OS, and that "AI" is on the cloud that means you are requiring your users to upload their data in plaintext.
- brap 3y agoI wonder what companies like OpenAI will do if we see on-device AI becoming the mainstream. The entire business model is based on API access. Will they start selling offline models? I guess then piracy becomes an issue…
- BudaDude 3y agoOpenAI will always be in a different league. I know Apple is making some amazing advancements with LLMs, but OpenAI basically recreated Google Search as a chatbot. I'm not saying Apple couldn't do the same thing, but the model alone would take up so much space on the phone that it would not be practical. I can see Apple making something closer to a very fine tuned mistral 7B than GPT4. But I'm sure in a year this comment will be wrong.
- brap 3y agoI’m not saying on-device performance will ever be comparable to cloud, it’s probably a different use case and market at this point. But 1, it’s still a profitable business to be in, so why shouldn’t OpenAI go there? And 2, more importantly, as hardware and software become more efficient, in the future on-device might be good enough to put up a fight and start eating the cloud. Seems risky not to go there.
- MBCook 3y agoOpenAI could always leverage their expertise to license out their model, or a cut down/customized version, to companies who want to use it locally on end user devices.
- summerlight 3y agoThis got to be one of the major goals for the next gen generative models. Those "big" models cannot be reliably used on critical tasks thanks to various failure modes. Even for those non-critical tasks, the latency is typically high enough to be unusable for many potential applications. If it can be 99.9% done in-device, this has a potential to drastically change the user experience in many directions.
- movedx 3y agoThis is very clever of Apple because it solves a lot of problems in one go. Namely, it lowers their costs as they're not running infrastructure to power the AI/ML options available to iPhone users; it distributes the compute of that to the edge, again saving them money and taking advantage of the costs they've already laid out for the manufacturer of the device; and it gives consumers the privacy they want - the data never leaves the phone.
- moandcompany 3y agoPart of the narrative from both Google and Apple is that "on-device" is better aligned with preserving user privacy. That's true, and it also has performance rationale when network latency makes online/cloud-inference non-viable for some use cases. Google and Apple have been promoting on-device inference/hardware accelerated AI/ML processing for several years, and with Google this goes back as early as Pixel 2. However, a big factor motivating these companies to promote "on-device" inference is the operating cost. While the compute and data requirements for training many models exceed the resources of almost everyone but these titans, it is still expected that when these models will be used for inference, and the total costs of inference over time, will far exceed the cost of training.
- fennecbutt 3y agoPretty sure every device manufacturer wants this too. Cloud services for AI are expensive and are realistically loss leaders to sell devices.