8 ms·
I really hope Apple releases an iPhone with a good on-device private LLM assistant, perhaps next year. Their hardware is well-positioned for it. It could make
by yreg 3y ago
I really hope Apple releases an iPhone with a good on-device private LLM assistant, perhaps next year. Their hardware is well-positioned for it.
It could make me get a new phone outside of my usual ~4 year cycle. Siri is almost unusable for me.
- CaptainOfCoit 3y agoYou're unlikely to get a better experience with Siri if she becomes equipped with a 7B or 13B LLM, unless Apple figured out something revolutionary.
- s3p 3y agoWhy would that be?
- ghqst 3y agoHave you ever actually used Siri?
- yreg 3y agoYes, I've been trying it out regularly ever since it was released. Last time I've talked to it for like 30 minutes while driving in October (just to test it again). It simply doesn't work for me.
- jurmous 3y agoReleased 2 days ago by Apple, a research paper on methods to run larger llms on iPhones. https://www.macrumors.com/2023/12/21/apple-ai-researchers-run-llms-iphones/ https://www.macrumors.com/2023/12/21/apple-ai-researchers-ru... https://arxiv.org/pdf/2312.11514.pdf https://arxiv.org/pdf/2312.11514.pdf
- reissbaker 3y agoThe paper was definitely cool but doesn't allow you to run particularly large LLMs on iPhones. It allows you to run a certain kind of LLM (sparse ReLU based LLMs) whose weights are somewhere less than 2x RAM. So, 7b Falcon works, but the competitive-with-gpt-3.5-turbo LLMs are still out of reach (and aren't ReLU based, although maybe that could change in the future). And nothing is competitive with GPT-4 right now. Of course in the long run I think it will happen — smaller and more efficient models are getting better regularly, and Apple can also just ship their new iPhones with larger amounts of RAM. But I'd be very surprised if there was GPT-4 level intelligence running locally on an iPhone within the next couple years — that sized model is so big right now even with significant memory optimizations, and I think distilling it down to iPhone size would be very hard even if you had access to the weights (and Apple doesn't). More likely there will be small models that run locally, but that fall back to large models running on servers somewhere for complex tasks, at least for the next couple years.
- SpaceManNabs 3y ago> And nothing is competitive with GPT-4 right now. You mean nothing available? Or you mean nothing that public knows exists? The answers to those two questions are different. There are definitely products that aren't available but the public knows exist and are upcoming that are in GPT-4's ballpark.
- olddustytrail 3y agoEasy to claim but harder to prove. Name one.
- schleck8 3y agoI heard rumours of these claims a few weeks ago, I assume they are talking about the same thing. Nothing concrete but from a reputable person and honestly with how well mixtral performs on the chatbot arena elo board I wouldn't be surprised if it's true.
- reissbaker 3y agoI mean nothing that is able to be benchmarked and validated by third parties is GPT-4 quality. I know there are upcoming releases that are hyped as being equal to GPT-4, e.g. Gemini Ultra, which I am very excited to get my hands on — but regardless, Ultra is not small enough to run on phones, even using the sparse ReLU flash memory optimization. And we'll see how it benchmarks once it's released; according to some benchmarks Gemini Pro has somewhat underperformed GPT-3.5-Turbo [1], despite Google's initial claims. (Although there are criticisms of that benchmarking, and it does beat the current 1106 version of GPT-3.5-Turbo on the Chatbot Arena leaderboard [2], although it slightly underperforms the previous 0613 version.) 1: https://arxiv.org/pdf/2312.11444.pdf https://arxiv.org/pdf/2312.11444.pdf 2: https://huggingface.co/spaces/lmsys/chatbot-arena-leaderboard https://huggingface.co/spaces/lmsys/chatbot-arena-leaderboar...
- mikhailt 3y agoYea but it's likely to be better than the current iteration of Siri even in that state. They can still outsource to a much larger LLMs on their servers for anything that can't be done locally like they do now.
- bbor 3y agoNote that “using an LLM” doesn’t just mean “plugging user queries straight into an LLM”. Enhancing Siri will probably be an ensemble project.
- dbish 3y agoThis is part of the mismatch between comparing Alexa/Siri/Cortana to a chat based LLM right now. If you just want chat and info retrieval, today’s LLMs are way better then the conversational dialogue, search, and q&a capabilities any of those assistants have. But, if you want relatively predicable task completion for things like smart home control, timers and alarms, real time weather updates, etc. (basic but frequently used interactions) or any integration with your phone or computer directly, there’s a lot to do that isn’t solved yet in the LLM space and is more of a system integration and action choice problem that the existing assistants have been hammering away at for years.
- bbor 3y agoI would argue that “info retrieval” is also something the LLM space has yet to yet to solve to a human level of reliability, but I think your comment is right on. I see this all as part of the greater symbolic vs. stochastic (/neat v scruffy) dynamics
- spookthesunset 3y agoI would hope that it would at last get you out of the hell that is “what the heck did I name that stupid light?”. Device naming is, in my opinion, the worst part of any of the voice based home assistant things. Is it “porch led”, “porch light” or “outdoor light”? Or is “outdoor light” actually the one in the front yard? What is the one by my kids nightstand? And what routine name do I use to set the mood for watching a movie? I would hope a properly trained llm with an awareness of my devices and their locations would allow for more verbal ambiguity when controlling things.
- dbish 3y agoMaybe, seems like overkill to solve a relatively straight forward problemt that is about resolution to real entities with a generative model that has no entity grounding (and may introduce other problems). It's really just not solved because Alexa/Siri leadership doesn't actually care enough about that use case to solve it. Device entity resolution and disambiguation does not require an LLM to solve for a limited number of devices, it just requires people to prioritze that over other things, and smart home device owners are the kind of early adopter that would not be the current market focus for growth (my guess, haven't worked on Alexa in many years). I know a half dozen different ways to improve that today without LLMs.
- nexuist 3y agoSiri is really quite dumb. I am confident that a 7B model would be able to provide better responses in over 90% of user queries. I can't even get Siri to reliably set a timer.
- CaptainOfCoit 3y agoYes, Siri is really dumb. But so is every 7B/13B model out there too.
- kossTKR 3y agoEh no, 7B Mistral / Deepseek would certainly almost already be able to function as a super Siri, but probably something closer to PHI-2 + the new MLX apple optimisations. Have you tried those? https://twitter.com/awnihannun/status/1735355067673526360 https://twitter.com/awnihannun/status/1735355067673526360 If trained on an iPhone API + documentation and given a little web access it would blow absolutely everything out of the water. If they can already create -basic- Python/Swift/JS/rust apps that sets timers, save things, create lists, how's that too dumb for being a Siri replacement? They just have to give it access to an iPhone/Web Api like ChatGPT's code analysis tool. So if you ask it "hey siri do this, this and this", it will create a script, then run it on the internal API, or fetch an article then work on that etc. I know it's still logically "dumb" but i'm not trying to play game theoretical scenarios with my phone or do logic tests or advanced math (yet).
- skygazer 3y agoThat sounds amazing and also the jailbreak of it via adversarial voice prompting sounds like a horrific vulnerability.
- kossTKR 3y agoTrue but you could make the api restricted, having certain routes completely locked, some requiring double checks, some requiring on screen approval or face-id, throttling outside fetches, only being able to run get and not etc, no financial app control etc. But yeah "hey siri transfer all of my funds to eric", or "hey siri group all of my photos where i'm nude and send them to jack" are new almost sci fi vectors.
- aaronbrethorst 3y agoRumors suggest they’re gearing up to make iOS 18 an AI focused release. It’ll be interesting to see if they offer different capabilities for online/offline scenarios, or if their offerings are strictly offline. Here’s one story to offer some context. There are others. https://archive.is/en3VL https://archive.is/en3VL
- para_parolu 3y agoI really hope they will make siri usable. In the current state it’s only good for fixed phrases. And even then it fails time to time
- behnamoh 3y agoI hope they get rid of it completely. People on r/locallama and others have made much better assistants using GPT which use iOS APIs to control the phone. It's ridiculous that Apple still hasn't done anything useful regarding Siri.
- michelb 3y agoIt's been in an 'AI' rewrite for a while now. Pretty sure we'll see something next year.
- astrange 3y agoIt's been rewritten several times, people largely don't notice because they don't try using it in different languages. And of course because they want to seem savvy so they repeat impressions from other people's posts, not realizing those posts are years old.
- ben_w 3y agoI'd be (pleasantly) surprised; look at how long it takes them to allow system apps to get replaced with downloadable 3rd party. And even then, the activation keyword is likely to be whatever Apple says. Similar logic as 3rd party keyboards, don't want user input to get stuck on even merely potentially untrustworthy or buggy code.
- bbor 3y agoI really, really doubt it for one reason: I’m convinced Apple is still terrified of that “Amazon Alexa tells child to stick a penny in a socket” story, and will hamstring themselves in an attempt to have their agential cake and eat it too
- thebruce87m 3y agoThey are right to be careful, they are held to a much higher standard than their competitors. Pixel phones have had emergency call issues for years across multiple models but they just get a pass. Apple would be crucified for this.
- astrange 3y agoSounds like a regulator issue. Doing emergency calls is a phone's #1 job, they shouldn't be allowing them to be sold if they don't work.
- Baldbvrhunter 3y agoyet in 15+ years I have never used any of mine for that
- astrange 3y agoGood for you. It's the only thing phones are required to be able to do in the US even if they don't have a working SIM or you haven't paid the phone bill. Well, I guess they're also not allowed to cause RF interference or randomly catch fire.
- Baldbvrhunter 3y agosure, but that doesn't make it their #1 job my pyjamas are regulated to be fireproof too
- behnamoh 3y agoApple is all about a controlled pleasant experience, it doesn't matter if it doesn't give you shiny new things; most Apple customers don't even know those shiny new things exist, so they keep spreading the word that "Apple is so easy and simple." The idea of having an unpredictable LLM in the ecosystem is Apple's worst nightmare. I bet they will overly restrict it to the point that it stops being a general purpose LLM and becomes a neutered obedient LLM that always acts according to Apple's rules. Also, it doesn't help that ALL the authors of this Apple paper are chinese. It raises questions about how Apple will handle political debates with its LLM.
- hmottestad 3y agoWith iOS 17 they added a tiny little LLM to the predictive typing. I have the newest and greatest iPhone but I feel that I very rarely see it in action. I must assume that it’s just too slow at to keep up with my typing at the moment. Or it’s just not large enough to give very many useful suggestions.
- KMnO4 3y agoIs tiny LLM an oxymoron? I believe Apple has told us it’s a transformer language model, but not specifically a LLM.
- catoc 3y agoYeah, they meant a TLM
- 0x1ceb00da 3y agoThat's print('Hello, world!')
- ben_w 3y agoIt's like how the "New Forest" is really old now: even small LLMs are (from what I've seen which isn't exhaustive) large compared to Markov language models.
- astrange 3y agoThere's no difference. An LLM is just a transformer language model that's "large".
- hmottestad 3y agoAccording to this article[1] it has about 34 million parameters. https://jackcook.com/2023/09/08/predictive-text.html https://jackcook.com/2023/09/08/predictive-text.html
- dontlaugh 3y agoIt’s probably why autocomplete got drastically worse, to the point I’m considering turning it off entirely. Most “AI” features are so incredibly fragile they’re not worth deploying.
- deleted 3y ago[deleted]
- cedws 3y agoWhat are you going to do with it?
- fnordpiglet 3y agoThe auto correct is already backed by a smallish LLM, FYI. https://jackcook.com/2023/09/08/predictive-text.html https://jackcook.com/2023/09/08/predictive-text.html
- blululu 3y agoAnd it is a serious quality regression IMO. The dictionary is too small and misses/messes up a ton of basic words.
- fnordpiglet 3y agoIt’s a place holder for iOS 18’s expansion. It’s 0.1 of the LLM in iOS. And the other prior implementation was so ducking bad that I’m not sure how you would observe such a regression.
- scosman 3y agoSLM? :)
- zitterbewegung 3y agoNext year releases of macOS / iOS are rumored to have LLMs as a feature .
- spaceman_2020 3y agoGPT-4 voice is so, so good. Really what you would want a voice tool to be like. I can talk to it like a normal human being, unlike issuing specific commands loudly as with Siri.
- klabb3 3y agoBut no matter the Siri shittiness (which I agree with) an LLM can only interact with the outside world – ie run commands – that exist and have a reasonable API surface, no? Apple has had automation for ages with Automator, Shortcuts etc but nothing that actually integrates well with day to day flow. So.. setting a timer when my hands are wet already works ok, and that’s about what I need. I honestly wonder what type of voice interactions people want with their phones. I can see transcribing/crafting chat messages I guess? But even so, it feels like it would mess up and use iMessage instead of WhatsApp, will it narrate my memes, open links and read “subscribe for only 4.99 to read this article”, cookie consents etc etc. if everything sucks how is narrating it gonna help? Maybe I’m old but I still don’t see the major value-add of voice interfaces, despite massively improved tech and potential.
- spaceman_2020 3y agoI would be happy if it does my morning routine for me. Give me a brief summary of my emails over the night, tell me the headlines from the news outlets I follow, summarize tweets from my favorited accounts, and give me an overview of the market based on my investments and watchlists.
- schleck8 3y agoYes, their hardware is positioned phenomenally with little RAM even by phone standards which is what you'd hack around with for inference on mobile architectures.
- fennecbutt 3y agoNobody can tame LLM models yet not even Apple. I can still get chatgpt to say the most vile things and if Apple release something on device I'll get that to be a bad, baaaad robot, too. LLMs are not yet safe for public facing production use,imo.