5 ms·

I really hope Apple doesn't mess this up and includes a solid on device LLM in iOS in the near future. They have amazing chips, but Siri has been a subpar assi

by yreg 3y ago

I really hope Apple doesn't mess this up and includes a solid on device LLM in iOS in the near future. They have amazing chips, but Siri has been a subpar assistent since forever. Now is the time to redeem her.

gnicholas 3y ago
I'd love to have my iPhone communicate with a Mac Studio at my house, for the heavy lifting. I realize this would be slower than having on-device processing, but it would be much better for battery life. And although I trust Apple's privacy more than Google/FB, I'd still rather keep my AI interactions separate from anyone's cloud.
- jayd16 3y ago
  > it would be much better for battery life. I wonder what the numbers actually are for local compute on custom hardware compared to firing up the wifi antennae to make the remote request.
  - gnicholas 3y ago
    Yeah I have wondered about this. But seeing how an LLM hammers my M2 MBA CPU for many seconds per request, I’m guessing this would have a significant impact on a smartphone battery.
    - gcr 3y ago
      ANE power usage for always-on Hey Siri wake word detection is impressively low. The language models we have are orders of magnitude way too big, but they won’t be for long. I think we’ll be surprised. For comparison, a quantized mobilenetv2 takes 3.7MB disk space to solve the same image understanding task as the old VGG Caffe models which take 550MB. We’ve come a long way in six years.
  - circuit10 3y ago
    Good language models are NOT easy to run, so I’d imagine local compute would take many orders of magnitude more power
- smoldesu 3y ago
  You might be pleased to hear that nothing really stops you from doing this today. If you ran Serge[0] on a Mac with Tailscale, you could hack together a decently-accelerated Llama chatbot. [0] https://github.com/serge-chat/serge https://github.com/serge-chat/serge
  - gnicholas 3y ago
    I'm not technical enough to be able to hack this together, but I do hope that enough other people have the same itch, and are able to scratch it!
- bob-09 3y ago
  I’d love to see both options and a seamless transition between them. An option tuned for home use that utilizes the processing power of home devices and local networks, and an option tuned for on-the-go use utilizing the iPhone/iPad processors and mobile networks.
viraptor 3y ago
> Siri has been a subpar assistent since forever They don't need a massive shift in the tech to make it better though. It would be cool if they improved it that way, but they have low-hanging fruit they could've addressed long time ago. Switching from one underutilized tech to another underutilized tech may not solve much without taking the whole feature seriously.
baby 3y ago
Why would it accomplish vs doing request to a remote Apple server? I'm not asking from a privacy standpoint, or a "it'd be cool if it works online", but from an Apple standpoint.
- yreg 3y ago
  Privacy standpoint is the Apple standpoint.
  - baby 3y ago
    not really, only when it makes sense with their main mantras of locking you in
- pornel 3y ago
  * Lower latency, especially p99 (networks can be crappy) * User pays for the silicon and the energy used to service requests. OTOH Cook probably would love to sell you an AppleAI+ subscription…
  - baby 3y ago
    lower latency is definitely a great point. The second point I subscribe less to it, this would mean more complexity and harder constraints on the battery
sp332 3y ago
It's going to chew up at least 1 GB of storage space and RAM, right? And probably kill the battery life to boot.
- gmerc 3y ago
  Once the foundational tech has stabilised, post transformers likely, the hardware people will finally get to do their thing - at which point the battery drain will go the way the battery drain for HD video went
- refulgentis 3y ago
  Yeah. People are playing tons of word games with this stuff, ex. Apple is saying its shipping an LLM for the iOS 17 keyboard, and who knows what that means: it sounds great and plausible unless you're familiar w/the nuts and bolts.
  - Tagbert 3y ago
    Apple is calling their typing correction a “transformer”. That is a component of LLMs, but Apple may not be using a full LLM in that case. This feature seems like a sandbox for them to try out some of this tech in the field while they do work internally on more ambitious implementations. Apple is also dogfooding an LLM AI tool internally also likley to gain a better understanding of how this works in practice and how people are using them. https://forums.macrumors.com/threads/apple-experimenting-with-apple-gpt-ai-tool-no-launch-planned-yet.2396336/ https://forums.macrumors.com/threads/apple-experimenting-wit...
    - astrange 3y ago
      An LLM is made entirely out of transformers; you could just call it a "large transformer model". In this case it's a transformer model that is not "large". So, an LM.
  - shwaj 3y ago
    Apple's not playing word games, because they didn't say "LLM". They said that autocorrect will use "a transformer language model, a state-of-the-art on-device machine learning language model for word prediction", which is a much more precise statement than what you attributed to them. This sounds totally plausible. It will be a much smaller transformer model than ChatGPT, probably much smaller than even GPT-2. https://www.apple.com/newsroom/2023/06/ios-17-makes-iphone-more-personal-and-intuitive/ https://www.apple.com/newsroom/2023/06/ios-17-makes-iphone-m...
asadm 3y ago
They are adding their first LLM in iOS 17's keyboard.
- gnicholas 3y ago
  This will be great. Hopefully it will be able to figure out that it should capitalize my last name, which is on my Contact card, and that of a dozen of my relatives. When I went to the Apple Store, they told me that I could either add a custom autocorrect to my library, or reset all settings. They did admit this was some sort of bug, and that it would be massive overkill to reset all settings (lose all wifi passwords, etc.).
  - coder543 3y ago
    > or reset all settings > that it would be massive overkill to reset all settings (lose all wifi passwords, etc.). I don't think that's what anyone was recommending... Settings -> General -> Transfer or Reset iPhone -> Reset -> Reset Keyboard Dictionary is almost certainly what they were recommending. What does resetting your keyboard dictionary have to do with your wifi passwords?
    - gnicholas 3y ago
      Nope, the two employees I spoke to were talking about a full reset (which affects network settings). Regardless of what the Keyboard Dictionary says, the iPhone should be autocompleting/capitalizing the last name of contacts, and especially the owner's name.
      - coder543 3y ago
        Why would that implicitly be "regardless" of what the keyboard dictionary says? I would expect the learned dictionary to be prioritized over other sources of information, just as a practical matter, even if someone might reasonably assume there are other things that should be prioritized over it. None of that explains how resetting everything would have any affect on capitalization of names if resetting the keyboard dictionary wouldn't, and you didn't say whether you tried resetting keyboard dictionary.
rvz 3y ago
Of course they will add an on-device LLM and can afford to. It doesn't cost them anything to integrate or train a AI whether if it is a ConvNet or a LLM and jump into the race to zero with Meta with on-device machine learning. They have done it before and they will certainly do it again, especially with Apple Silicon and CoreML. The one that needs to worry the most is O̶p̶e̶n̶AI.com as they rushed to stop the adoption of downloadable and powerful AI models for free to the regulators. That shows that O̶p̶e̶n̶AI.com does not have any sort of 'moat' at all.
- yreg 3y ago
  The question is whether the iPhones released in September are going to be already ready for it. They haven't mentioned LLMs at WWDC beyond keyboard autocorrect (mentioned already by your sibling comment).
  - baq 3y ago
    No chance unless they were prescient early 2022. Hardware cycles are too long for that otherwise.
    - yreg 3y ago
      Is it even necessary to do changes to the neural engine though? Maybe something like increasing RAM (which is rumoured) is enough.
      - baq 3y ago
        If there’s one company which can wrangle its suppliers to deliver 4x RAM capacity in the same form factor, performance and thermals it’s Apple, but they aren’t sorcerers, just ruthlessly efficient. I’ll be queueing at midnight for the first time ever if I’m wrong.
cscurmudgeon 3y ago
Otoh, Apple’s monopolistic behavior implies that it will be good for society if they mess up.

by yreg 3y ago

gnicholas 3y ago

I'd love to have my iPhone communicate with a Mac Studio at my house, for the heavy lifting. I realize this would be slower than having on-device processing, but it would be much better for battery life. And although I trust Apple's privacy more than Google/FB, I'd still rather keep my AI interactions separate from anyone's cloud.

jayd16 3y ago
> it would be much better for battery life. I wonder what the numbers actually are for local compute on custom hardware compared to firing up the wifi antennae to make the remote request.
- gnicholas 3y ago
  Yeah I have wondered about this. But seeing how an LLM hammers my M2 MBA CPU for many seconds per request, I’m guessing this would have a significant impact on a smartphone battery.
  - gcr 3y ago
    ANE power usage for always-on Hey Siri wake word detection is impressively low. The language models we have are orders of magnitude way too big, but they won’t be for long. I think we’ll be surprised. For comparison, a quantized mobilenetv2 takes 3.7MB disk space to solve the same image understanding task as the old VGG Caffe models which take 550MB. We’ve come a long way in six years.
- circuit10 3y ago
  Good language models are NOT easy to run, so I’d imagine local compute would take many orders of magnitude more power
smoldesu 3y ago
You might be pleased to hear that nothing really stops you from doing this today. If you ran Serge[0] on a Mac with Tailscale, you could hack together a decently-accelerated Llama chatbot. [0] https://github.com/serge-chat/serge https://github.com/serge-chat/serge
- gnicholas 3y ago
  I'm not technical enough to be able to hack this together, but I do hope that enough other people have the same itch, and are able to scratch it!
bob-09 3y ago
I’d love to see both options and a seamless transition between them. An option tuned for home use that utilizes the processing power of home devices and local networks, and an option tuned for on-the-go use utilizing the iPhone/iPad processors and mobile networks.

viraptor 3y ago

> Siri has been a subpar assistent since forever They don't need a massive shift in the tech to make it better though. It would be cool if they improved it that way, but they have low-hanging fruit they could've addressed long time ago. Switching from one underutilized tech to another underutilized tech may not solve much without taking the whole feature seriously.

baby 3y ago

Why would it accomplish vs doing request to a remote Apple server? I'm not asking from a privacy standpoint, or a "it'd be cool if it works online", but from an Apple standpoint.

yreg 3y ago
Privacy standpoint is the Apple standpoint.
- baby 3y ago
  not really, only when it makes sense with their main mantras of locking you in
pornel 3y ago
* Lower latency, especially p99 (networks can be crappy) * User pays for the silicon and the energy used to service requests. OTOH Cook probably would love to sell you an AppleAI+ subscription…
- baby 3y ago
  lower latency is definitely a great point. The second point I subscribe less to it, this would mean more complexity and harder constraints on the battery

sp332 3y ago

It's going to chew up at least 1 GB of storage space and RAM, right? And probably kill the battery life to boot.

gmerc 3y ago
Once the foundational tech has stabilised, post transformers likely, the hardware people will finally get to do their thing - at which point the battery drain will go the way the battery drain for HD video went
refulgentis 3y ago
Yeah. People are playing tons of word games with this stuff, ex. Apple is saying its shipping an LLM for the iOS 17 keyboard, and who knows what that means: it sounds great and plausible unless you're familiar w/the nuts and bolts.
- Tagbert 3y ago
  Apple is calling their typing correction a “transformer”. That is a component of LLMs, but Apple may not be using a full LLM in that case. This feature seems like a sandbox for them to try out some of this tech in the field while they do work internally on more ambitious implementations. Apple is also dogfooding an LLM AI tool internally also likley to gain a better understanding of how this works in practice and how people are using them. https://forums.macrumors.com/threads/apple-experimenting-with-apple-gpt-ai-tool-no-launch-planned-yet.2396336/ https://forums.macrumors.com/threads/apple-experimenting-wit...
  - astrange 3y ago
    An LLM is made entirely out of transformers; you could just call it a "large transformer model". In this case it's a transformer model that is not "large". So, an LM.
- shwaj 3y ago
  Apple's not playing word games, because they didn't say "LLM". They said that autocorrect will use "a transformer language model, a state-of-the-art on-device machine learning language model for word prediction", which is a much more precise statement than what you attributed to them. This sounds totally plausible. It will be a much smaller transformer model than ChatGPT, probably much smaller than even GPT-2. https://www.apple.com/newsroom/2023/06/ios-17-makes-iphone-more-personal-and-intuitive/ https://www.apple.com/newsroom/2023/06/ios-17-makes-iphone-m...

asadm 3y ago

They are adding their first LLM in iOS 17's keyboard.

gnicholas 3y ago
This will be great. Hopefully it will be able to figure out that it should capitalize my last name, which is on my Contact card, and that of a dozen of my relatives. When I went to the Apple Store, they told me that I could either add a custom autocorrect to my library, or reset all settings. They did admit this was some sort of bug, and that it would be massive overkill to reset all settings (lose all wifi passwords, etc.).
- coder543 3y ago
  > or reset all settings > that it would be massive overkill to reset all settings (lose all wifi passwords, etc.). I don't think that's what anyone was recommending... Settings -> General -> Transfer or Reset iPhone -> Reset -> Reset Keyboard Dictionary is almost certainly what they were recommending. What does resetting your keyboard dictionary have to do with your wifi passwords?
  - gnicholas 3y ago
    Nope, the two employees I spoke to were talking about a full reset (which affects network settings). Regardless of what the Keyboard Dictionary says, the iPhone should be autocompleting/capitalizing the last name of contacts, and especially the owner's name.
    - coder543 3y ago
      Why would that implicitly be "regardless" of what the keyboard dictionary says? I would expect the learned dictionary to be prioritized over other sources of information, just as a practical matter, even if someone might reasonably assume there are other things that should be prioritized over it. None of that explains how resetting everything would have any affect on capitalization of names if resetting the keyboard dictionary wouldn't, and you didn't say whether you tried resetting keyboard dictionary.

rvz 3y ago

Of course they will add an on-device LLM and can afford to. It doesn't cost them anything to integrate or train a AI whether if it is a ConvNet or a LLM and jump into the race to zero with Meta with on-device machine learning. They have done it before and they will certainly do it again, especially with Apple Silicon and CoreML. The one that needs to worry the most is O̶p̶e̶n̶AI.com as they rushed to stop the adoption of downloadable and powerful AI models for free to the regulators. That shows that O̶p̶e̶n̶AI.com does not have any sort of 'moat' at all.

yreg 3y ago
The question is whether the iPhones released in September are going to be already ready for it. They haven't mentioned LLMs at WWDC beyond keyboard autocorrect (mentioned already by your sibling comment).
- baq 3y ago
  No chance unless they were prescient early 2022. Hardware cycles are too long for that otherwise.
  - yreg 3y ago
    Is it even necessary to do changes to the neural engine though? Maybe something like increasing RAM (which is rumoured) is enough.
    - baq 3y ago
      If there’s one company which can wrangle its suppliers to deliver 4x RAM capacity in the same form factor, performance and thermals it’s Apple, but they aren’t sorcerers, just ruthlessly efficient. I’ll be queueing at midnight for the first time ever if I’m wrong.

cscurmudgeon 3y ago

Otoh, Apple’s monopolistic behavior implies that it will be good for society if they mess up.