5 ms·

I have a hard time believing this since everything Apple has been releasing in the AI space points towards running models locally.

dannyw 2y ago
Few thoughts here: 1. You can't fit a XXX billion param model on an iPhone. It's just impossible. A mixed approach where most ML is local; but some are cloud, fits with their existing Siri approach. 2. OpenAI won't share model weights, but if there's one company they'd make an exception for, it'd be Apple. Certainly not their big models, but perhaps they have a distilled, SOTA 3-7B model that they're happy to license, especially with Apple's Secure Enclave and ML model encryption.
- MisterBiggs 2y ago
  For 1. Apple is certainly trying to fit models onto an iPhone with OpenELM and I think the M4 debuting on an iPad is also posturing for local compute as well. A mixed approach is a great point though and I think I could mostly be happy with that.
  - ffgjgf1 2y ago
    > M4 debuting on an iPad is also posturing for local compute as well. Wouldn’t memory be a much bigger bottleneck? You don’t really need a particularly fast a CPU/GPU to run most basic models that couldn’t even fit into the amount of memory that Apple is offering on their devices (especially if you still want to run other apps).
  - lolinder 2y ago
    > For 1. Apple is certainly trying to fit models onto an iPhone with OpenELM and I think the M4 debuting on an iPad is also posturing for local compute as well. You misunderstand point 1. Three-digit billion parameter models like GPT-4 (or even 3.5) aren't going to fit on modern iPhones, not even close. Even Llama 70b requires 35GB of RAM at q4 quantization, and that's just for the model. Compare that to the iPhone 15's 6GB and you'll see the problem. Apple isn't about to announce an iPhone with 6x as much RAM as any previous model. I'd be shocked if they even doubled it. Their local inference models are going to be tiny and limited, which is fine, but that means they have to go to the cloud to provide all the features people are expecting.
- a_wild_dandan 2y ago
  You can fit a Q5 Llama-3-8b or many other models on new iPhones.
  - ffgjgf1 2y ago
    It’s not very good though and with 6 GB of memory iPhones would struggle anything else alongside it so forget multitasking or even running the full OS while it’s loaded.. Almost all Apple devices (including most Macs) have very low amounts of memory so Apple hadn’t really positioned themselves that well if their goal is running LLMs locally.
    - d13 2y ago
      Llama 3 8b Q5 is within GPT4 territory. I’ve been using it daily, it’s astonishing.
      - ffgjgf1 2y ago
        I found it to be not even remotely close outside of a few tasks/queries
        theshrike79 2y ago
        You don't need a "phone AI" to solve general relativity. You need it to understand what you want to do within the limited scope of the phone. Basically I want Apple's AI to do what Shortcuts is capable of today, but instead of a janky Scratch programming style, I want it to respond to speech.
- dareal 2y ago
  Gemini nano is already in Google pixel and runs locally.
  - ffgjgf1 2y ago
    Pixels have a lot more RAM than iPhones though, only low-end Androids devices have as little.
- mritchie712 2y ago
  > OpenAI won't share model weights Microsoft has 'em [0] 0 - plenty of sources on this, but here's one https://x.com/elonmusk/status/1639138603371491329 https://x.com/elonmusk/status/1639138603371491329
Tagbert 2y ago
I suspect that they will be running AI locally but also calling out to server-side services where necessary. I don't expect a ChatGPT integration to be the whole of what Apple has planned for AI on their devices.
capybara_2020 2y ago
This is what it says in the article > The majority of iOS 18’s AI features, however, will be powered entirely on-device, allowing Apple to tout privacy and speed benefits.
Xenoamorphous 2y ago
I see a lot of talk in HN about how Apple running models on the device is such a great idea, for privacy reasons. However I feel like outside of a few niches like HN, people don’t give a damn about privacy (for example, the company I work for has a website with millions of daily unique visitors; GDPR consent rates are usually around 98%). So if other phones use models running on the cloud and they’re higher quality than the local running ones in iPhones (which will prob be smaller) that might be more important to the average user.
- jwells89 2y ago
  A big differentiator is that local models have access to more user data without sending anything over the wire (which ignoring privacy implications, isn’t free). Local models can also more easily plug into existing macOS/iOS facilities like AppleScript, Automator, Services, Shortcuts, and NSUserActivity which means they’ll be a lot more functional, with the ability to wire together apps that haven’t explicitly added AI support. In other words there’s a lot of potential here. If implemented well, LLM-powered Siri would have vastly more context and integration than anything else right out of the gate.
- kfarr 2y ago
  They’re not mutually exclusive. Local model can provide immediate answer with medium quality, remote can provide higher quality but longer round trip But anything is better than old school Siri
- whatindaheck 2y ago
  Apple is primarily a hardware company. Privacy claims aside, on-device gives Apple a reason to keep making, improving, and selling hardware. I also don’t think the GDPR consent is a good measure. Those suck to opt out of manually and I doubt most people consider finding an extension. Apple sells its privacy as zero to low configuration. You “buy” privacy, you don’t configure it. Whether their claims are misleading or not. That’s extremely appealing. A few other thoughts: * Local models would be a requirement if Advanced Data Protection[0] were enabled * Local models could potentially do some interesting work with hefty data (something media related maybe) that isn’t fit for cloud workflows [0]: https://support.apple.com/guide/security/advanced-data-protection-for-icloud-sec973254c5f/web https://support.apple.com/guide/security/advanced-data-prote...
sillywalk 2y ago
It looks like both, and they're apparently using Apple Silicon in their own datacenters. From Bloomberg[0] (archived)[1]: "Apple Inc. will deliver some of its upcoming artificial intelligence features this year via data centers equipped with its own in-house processors, part of a sweeping effort to infuse its devices with AI capabilities. The company is placing high-end chips — similar to ones it designed for the Mac — in cloud-computing servers designed to process the most advanced AI tasks coming to Apple devices, according to people familiar with the matter. Simpler AI-related features will be processed directly on iPhones, iPads and Macs ... The first AI server chips will be the M2 Ultra, which was launched last year as part of the Mac Pro and Mac Studio computers, though the company is already eyeing future versions based on the M4 chip..." ... Relatively simple AI tasks — like providing users a summary of their missed iPhone notifications or incoming text messages — could be handled by the chips inside of Apple devices. More complicated jobs, such as generating images or summarizing lengthy news articles and creating long-form responses in emails, would likely require the cloud-based approach — as would an upgraded version of Apple’s Siri voice assistant. ... For now, Apple is planning to use its own data centers to operate the cloud features, but it will eventually rely on outside facilities — as it does with iCloud and other services. The Wall Street Journal reported earlier on some aspects of the server plan." --- And from [2], Apple has been working on Apple Chips in Data Center (ACDC) for few years, focusing on AI inference, not training. [0] https://www.bloomberg.com/news/articles/2024-05-09/apple-to-power-ios-18-ai-features-with-in-house-server-mac-chips-this-year https://www.bloomberg.com/news/articles/2024-05-09/apple-to-... [1] https://archive.ph/MUswk https://archive.ph/MUswk [2] https://www.medianama.com/2024/05/223-apple-launches-project-develop-ai-chips-data-centers/ https://www.medianama.com/2024/05/223-apple-launches-project...
ein0p 2y ago
Por que no los dos? A large LLM won’t fit, but you don’t need a large LLM to invoke functions or answer texts. And when you need chat or search, that can go to the server, same as Siri does today.
gnicholas 2y ago
It would be really great if they could respond immediately using locally-generated information, and then seamlessly weave in data that comes back from the cloud.

dannyw 2y ago

Few thoughts here: 1. You can't fit a XXX billion param model on an iPhone. It's just impossible. A mixed approach where most ML is local; but some are cloud, fits with their existing Siri approach. 2. OpenAI won't share model weights, but if there's one company they'd make an exception for, it'd be Apple. Certainly not their big models, but perhaps they have a distilled, SOTA 3-7B model that they're happy to license, especially with Apple's Secure Enclave and ML model encryption.

MisterBiggs 2y ago
For 1. Apple is certainly trying to fit models onto an iPhone with OpenELM and I think the M4 debuting on an iPad is also posturing for local compute as well. A mixed approach is a great point though and I think I could mostly be happy with that.
- ffgjgf1 2y ago
  > M4 debuting on an iPad is also posturing for local compute as well. Wouldn’t memory be a much bigger bottleneck? You don’t really need a particularly fast a CPU/GPU to run most basic models that couldn’t even fit into the amount of memory that Apple is offering on their devices (especially if you still want to run other apps).
- lolinder 2y ago
  > For 1. Apple is certainly trying to fit models onto an iPhone with OpenELM and I think the M4 debuting on an iPad is also posturing for local compute as well. You misunderstand point 1. Three-digit billion parameter models like GPT-4 (or even 3.5) aren't going to fit on modern iPhones, not even close. Even Llama 70b requires 35GB of RAM at q4 quantization, and that's just for the model. Compare that to the iPhone 15's 6GB and you'll see the problem. Apple isn't about to announce an iPhone with 6x as much RAM as any previous model. I'd be shocked if they even doubled it. Their local inference models are going to be tiny and limited, which is fine, but that means they have to go to the cloud to provide all the features people are expecting.
a_wild_dandan 2y ago
You can fit a Q5 Llama-3-8b or many other models on new iPhones.
- ffgjgf1 2y ago
  It’s not very good though and with 6 GB of memory iPhones would struggle anything else alongside it so forget multitasking or even running the full OS while it’s loaded.. Almost all Apple devices (including most Macs) have very low amounts of memory so Apple hadn’t really positioned themselves that well if their goal is running LLMs locally.
  - d13 2y ago
    Llama 3 8b Q5 is within GPT4 territory. I’ve been using it daily, it’s astonishing.
    - ffgjgf1 2y ago
      I found it to be not even remotely close outside of a few tasks/queries
      - theshrike79 2y ago
        You don't need a "phone AI" to solve general relativity. You need it to understand what you want to do within the limited scope of the phone. Basically I want Apple's AI to do what Shortcuts is capable of today, but instead of a janky Scratch programming style, I want it to respond to speech.
dareal 2y ago
Gemini nano is already in Google pixel and runs locally.
- ffgjgf1 2y ago
  Pixels have a lot more RAM than iPhones though, only low-end Androids devices have as little.
mritchie712 2y ago
> OpenAI won't share model weights Microsoft has 'em [0] 0 - plenty of sources on this, but here's one https://x.com/elonmusk/status/1639138603371491329 https://x.com/elonmusk/status/1639138603371491329

Tagbert 2y ago

I suspect that they will be running AI locally but also calling out to server-side services where necessary. I don't expect a ChatGPT integration to be the whole of what Apple has planned for AI on their devices.

capybara_2020 2y ago

This is what it says in the article > The majority of iOS 18’s AI features, however, will be powered entirely on-device, allowing Apple to tout privacy and speed benefits.

Xenoamorphous 2y ago

I see a lot of talk in HN about how Apple running models on the device is such a great idea, for privacy reasons. However I feel like outside of a few niches like HN, people don’t give a damn about privacy (for example, the company I work for has a website with millions of daily unique visitors; GDPR consent rates are usually around 98%). So if other phones use models running on the cloud and they’re higher quality than the local running ones in iPhones (which will prob be smaller) that might be more important to the average user.

jwells89 2y ago
A big differentiator is that local models have access to more user data without sending anything over the wire (which ignoring privacy implications, isn’t free). Local models can also more easily plug into existing macOS/iOS facilities like AppleScript, Automator, Services, Shortcuts, and NSUserActivity which means they’ll be a lot more functional, with the ability to wire together apps that haven’t explicitly added AI support. In other words there’s a lot of potential here. If implemented well, LLM-powered Siri would have vastly more context and integration than anything else right out of the gate.
kfarr 2y ago
They’re not mutually exclusive. Local model can provide immediate answer with medium quality, remote can provide higher quality but longer round trip But anything is better than old school Siri
whatindaheck 2y ago
Apple is primarily a hardware company. Privacy claims aside, on-device gives Apple a reason to keep making, improving, and selling hardware. I also don’t think the GDPR consent is a good measure. Those suck to opt out of manually and I doubt most people consider finding an extension. Apple sells its privacy as zero to low configuration. You “buy” privacy, you don’t configure it. Whether their claims are misleading or not. That’s extremely appealing. A few other thoughts: * Local models would be a requirement if Advanced Data Protection[0] were enabled * Local models could potentially do some interesting work with hefty data (something media related maybe) that isn’t fit for cloud workflows [0]: https://support.apple.com/guide/security/advanced-data-protection-for-icloud-sec973254c5f/web https://support.apple.com/guide/security/advanced-data-prote...

sillywalk 2y ago

It looks like both, and they're apparently using Apple Silicon in their own datacenters. From Bloomberg[0] (archived)[1]: "Apple Inc. will deliver some of its upcoming artificial intelligence features this year via data centers equipped with its own in-house processors, part of a sweeping effort to infuse its devices with AI capabilities. The company is placing high-end chips — similar to ones it designed for the Mac — in cloud-computing servers designed to process the most advanced AI tasks coming to Apple devices, according to people familiar with the matter. Simpler AI-related features will be processed directly on iPhones, iPads and Macs ... The first AI server chips will be the M2 Ultra, which was launched last year as part of the Mac Pro and Mac Studio computers, though the company is already eyeing future versions based on the M4 chip..." ... Relatively simple AI tasks — like providing users a summary of their missed iPhone notifications or incoming text messages — could be handled by the chips inside of Apple devices. More complicated jobs, such as generating images or summarizing lengthy news articles and creating long-form responses in emails, would likely require the cloud-based approach — as would an upgraded version of Apple’s Siri voice assistant. ... For now, Apple is planning to use its own data centers to operate the cloud features, but it will eventually rely on outside facilities — as it does with iCloud and other services. The Wall Street Journal reported earlier on some aspects of the server plan." --- And from [2], Apple has been working on Apple Chips in Data Center (ACDC) for few years, focusing on AI inference, not training. [0] https://www.bloomberg.com/news/articles/2024-05-09/apple-to-power-ios-18-ai-features-with-in-house-server-mac-chips-this-year https://www.bloomberg.com/news/articles/2024-05-09/apple-to-... [1] https://archive.ph/MUswk https://archive.ph/MUswk [2] https://www.medianama.com/2024/05/223-apple-launches-project-develop-ai-chips-data-centers/ https://www.medianama.com/2024/05/223-apple-launches-project...

ein0p 2y ago

Por que no los dos? A large LLM won’t fit, but you don’t need a large LLM to invoke functions or answer texts. And when you need chat or search, that can go to the server, same as Siri does today.

gnicholas 2y ago

It would be really great if they could respond immediately using locally-generated information, and then seamlessly weave in data that comes back from the cloud.