4 ms·

This seems really cool. They said the models can scale to "private cloud compute" based on Apple Silicon which will be ensured by your device to run "publicly

This seems really cool. They said the models can scale to "private cloud compute" based on Apple Silicon which will be ensured by your device to run "publicly verifiable software" in order to guarantee no misuse of your data. I wonder if their server-side code will be open-source? That'd be positively surprising. Curious to see how this evolves. Anyway, overall looks really really cool. If it works as marketed, then it will be an easy "shut up and take my money". Siri seems to finally be becoming what it was meant to be (I wonder if they're piggy-backing on top of the Shortcuts Actions catalogue to have a wide array of possible actions right away), and the image and emoji generation features that integrate with Apple Photos and other parts of the system look _really_ cool. It seems like it will require M1+ on Macs/iPads, or an iPhone 15 Pro.

deleted 2y ago
[deleted]
deleted 2y ago
[deleted]
zitterbewegung 2y ago
You don't even have to buy a new device since it's backwards compatible with A17 Pro and M1, M2, M3 and M4. It feels like the integration of the services are using existing models and integrating the API used traditionally originally from AppleScript but, extending it to LLM or stable diffusion systems. It seems that they want the M4 as soon as possible though for the gaming and cloud pushes.
cube2222 2y ago
For those curious, there is in fact a ChatGPT integration. The way it works is that when the on-device model decides "this could better be answered by chatgpt" then it will ask you if it should use that. They described it in a way which seems to indicate that it will be pluggable for other models too over time. Notably, ChatGPT 4o will be available for free without creating an OpenAI account.
- ladams 2y ago
  I don't think that 4o will actually be available for free. It seemed like they were quite careful in choosing their words. My guess is 3.5 is free without an account, and accessing 4o requires linking your OpenAI account.
  - skygazer 2y ago
    They only mentioned 4o, but they mentioned it explicitly at the start, well before they mentioned one can also tie in to their openAI account, if you have one, at the end of the presentation. To me that implies 4o by default, but I guess we'll find out.
    - kokon 2y ago
      I think it's going to be free: https://openai.com/index/openai-and-apple-announce-partnership/ https://openai.com/index/openai-and-apple-announce-partnersh...
      - ukuina 2y ago
        Apple must be paying OpenAI a pretty penny to give the best LLM tokens away for free.
        ukuina 2y ago
        It seems no money is exchanging hands!
deleted 2y ago
[deleted]
losvedir 2y ago
I'm really curious about this. Framing it as "running a large language model in the cloud" is almost burying the lede for me. Is this saying that in general the client will be able to cryptographically ascertain somehow the code that the server is running? That sounds incredibly interesting and useful outside of this.
localhost 2y ago
It seems like this is an orchestration layer that runs on Apple Silicon, given that ChatGPT integration looks like an API call from that. It's not clear to me what is being computed on the "private cloud compute"?
- cube2222 2y ago
  If I understand correctly there's three things here: - on-device models, which will power any tasks it's able to, including summarisation and conversation with Siri - private compute models (still controlled by apple), for when it wants to do something bigger, that requires more compute - external LLM APIs (only chatgpt for now), for when the above decide that it would be better for the given prompt, but always asks the user for confirmation
  - localhost 2y ago
    The second point makes sense. It gives Apple optionality to cut off the external LLMs at a later date if they want to. I wonder what % of requests will be handled by the private cloud models vs. local. I would imagine TTS and ASR is local for latency reasons. Natural language classifiers would certainly run on-device. I wonder if summarization and rewriting will though - those are more complex and definitely benefit from larger models.
deleted 2y ago
[deleted]

deleted 2y ago

[deleted]

deleted 2y ago

[deleted]

zitterbewegung 2y ago

You don't even have to buy a new device since it's backwards compatible with A17 Pro and M1, M2, M3 and M4. It feels like the integration of the services are using existing models and integrating the API used traditionally originally from AppleScript but, extending it to LLM or stable diffusion systems. It seems that they want the M4 as soon as possible though for the gaming and cloud pushes.

cube2222 2y ago

For those curious, there is in fact a ChatGPT integration. The way it works is that when the on-device model decides "this could better be answered by chatgpt" then it will ask you if it should use that. They described it in a way which seems to indicate that it will be pluggable for other models too over time. Notably, ChatGPT 4o will be available for free without creating an OpenAI account.

ladams 2y ago
I don't think that 4o will actually be available for free. It seemed like they were quite careful in choosing their words. My guess is 3.5 is free without an account, and accessing 4o requires linking your OpenAI account.
- skygazer 2y ago
  They only mentioned 4o, but they mentioned it explicitly at the start, well before they mentioned one can also tie in to their openAI account, if you have one, at the end of the presentation. To me that implies 4o by default, but I guess we'll find out.
  - kokon 2y ago
    I think it's going to be free: https://openai.com/index/openai-and-apple-announce-partnership/ https://openai.com/index/openai-and-apple-announce-partnersh...
    - ukuina 2y ago
      Apple must be paying OpenAI a pretty penny to give the best LLM tokens away for free.
      - ukuina 2y ago
        It seems no money is exchanging hands!

deleted 2y ago

[deleted]

losvedir 2y ago

I'm really curious about this. Framing it as "running a large language model in the cloud" is almost burying the lede for me. Is this saying that in general the client will be able to cryptographically ascertain somehow the code that the server is running? That sounds incredibly interesting and useful outside of this.

localhost 2y ago

It seems like this is an orchestration layer that runs on Apple Silicon, given that ChatGPT integration looks like an API call from that. It's not clear to me what is being computed on the "private cloud compute"?

cube2222 2y ago
If I understand correctly there's three things here: - on-device models, which will power any tasks it's able to, including summarisation and conversation with Siri - private compute models (still controlled by apple), for when it wants to do something bigger, that requires more compute - external LLM APIs (only chatgpt for now), for when the above decide that it would be better for the given prompt, but always asks the user for confirmation
- localhost 2y ago
  The second point makes sense. It gives Apple optionality to cut off the external LLMs at a later date if they want to. I wonder what % of requests will be handled by the private cloud models vs. local. I would imagine TTS and ASR is local for latency reasons. Natural language classifiers would certainly run on-device. I wonder if summarization and rewriting will though - those are more complex and definitely benefit from larger models.

deleted 2y ago

[deleted]