3 ms·

Few thoughts here: 1. You can't fit a XXX billion param model on an iPhone. It's just impossible. A mixed approach where most ML is local; but some are cloud,

by dannyw 2y ago

Few thoughts here: 1. You can't fit a XXX billion param model on an iPhone. It's just impossible. A mixed approach where most ML is local; but some are cloud, fits with their existing Siri approach. 2. OpenAI won't share model weights, but if there's one company they'd make an exception for, it'd be Apple. Certainly not their big models, but perhaps they have a distilled, SOTA 3-7B model that they're happy to license, especially with Apple's Secure Enclave and ML model encryption.

MisterBiggs 2y ago
For 1. Apple is certainly trying to fit models onto an iPhone with OpenELM and I think the M4 debuting on an iPad is also posturing for local compute as well. A mixed approach is a great point though and I think I could mostly be happy with that.
- ffgjgf1 2y ago
  > M4 debuting on an iPad is also posturing for local compute as well. Wouldn’t memory be a much bigger bottleneck? You don’t really need a particularly fast a CPU/GPU to run most basic models that couldn’t even fit into the amount of memory that Apple is offering on their devices (especially if you still want to run other apps).
- lolinder 2y ago
  > For 1. Apple is certainly trying to fit models onto an iPhone with OpenELM and I think the M4 debuting on an iPad is also posturing for local compute as well. You misunderstand point 1. Three-digit billion parameter models like GPT-4 (or even 3.5) aren't going to fit on modern iPhones, not even close. Even Llama 70b requires 35GB of RAM at q4 quantization, and that's just for the model. Compare that to the iPhone 15's 6GB and you'll see the problem. Apple isn't about to announce an iPhone with 6x as much RAM as any previous model. I'd be shocked if they even doubled it. Their local inference models are going to be tiny and limited, which is fine, but that means they have to go to the cloud to provide all the features people are expecting.
a_wild_dandan 2y ago
You can fit a Q5 Llama-3-8b or many other models on new iPhones.
- ffgjgf1 2y ago
  It’s not very good though and with 6 GB of memory iPhones would struggle anything else alongside it so forget multitasking or even running the full OS while it’s loaded.. Almost all Apple devices (including most Macs) have very low amounts of memory so Apple hadn’t really positioned themselves that well if their goal is running LLMs locally.
  - d13 2y ago
    Llama 3 8b Q5 is within GPT4 territory. I’ve been using it daily, it’s astonishing.
    - ffgjgf1 2y ago
      I found it to be not even remotely close outside of a few tasks/queries
      - theshrike79 2y ago
        You don't need a "phone AI" to solve general relativity. You need it to understand what you want to do within the limited scope of the phone. Basically I want Apple's AI to do what Shortcuts is capable of today, but instead of a janky Scratch programming style, I want it to respond to speech.
dareal 2y ago
Gemini nano is already in Google pixel and runs locally.
- ffgjgf1 2y ago
  Pixels have a lot more RAM than iPhones though, only low-end Androids devices have as little.
mritchie712 2y ago
> OpenAI won't share model weights Microsoft has 'em [0] 0 - plenty of sources on this, but here's one https://x.com/elonmusk/status/1639138603371491329 https://x.com/elonmusk/status/1639138603371491329

by dannyw 2y ago

MisterBiggs 2y ago

For 1. Apple is certainly trying to fit models onto an iPhone with OpenELM and I think the M4 debuting on an iPad is also posturing for local compute as well. A mixed approach is a great point though and I think I could mostly be happy with that.

ffgjgf1 2y ago
> M4 debuting on an iPad is also posturing for local compute as well. Wouldn’t memory be a much bigger bottleneck? You don’t really need a particularly fast a CPU/GPU to run most basic models that couldn’t even fit into the amount of memory that Apple is offering on their devices (especially if you still want to run other apps).
lolinder 2y ago
> For 1. Apple is certainly trying to fit models onto an iPhone with OpenELM and I think the M4 debuting on an iPad is also posturing for local compute as well. You misunderstand point 1. Three-digit billion parameter models like GPT-4 (or even 3.5) aren't going to fit on modern iPhones, not even close. Even Llama 70b requires 35GB of RAM at q4 quantization, and that's just for the model. Compare that to the iPhone 15's 6GB and you'll see the problem. Apple isn't about to announce an iPhone with 6x as much RAM as any previous model. I'd be shocked if they even doubled it. Their local inference models are going to be tiny and limited, which is fine, but that means they have to go to the cloud to provide all the features people are expecting.

a_wild_dandan 2y ago

You can fit a Q5 Llama-3-8b or many other models on new iPhones.

ffgjgf1 2y ago
It’s not very good though and with 6 GB of memory iPhones would struggle anything else alongside it so forget multitasking or even running the full OS while it’s loaded.. Almost all Apple devices (including most Macs) have very low amounts of memory so Apple hadn’t really positioned themselves that well if their goal is running LLMs locally.
- d13 2y ago
  Llama 3 8b Q5 is within GPT4 territory. I’ve been using it daily, it’s astonishing.
  - ffgjgf1 2y ago
    I found it to be not even remotely close outside of a few tasks/queries
    - theshrike79 2y ago
      You don't need a "phone AI" to solve general relativity. You need it to understand what you want to do within the limited scope of the phone. Basically I want Apple's AI to do what Shortcuts is capable of today, but instead of a janky Scratch programming style, I want it to respond to speech.

dareal 2y ago

Gemini nano is already in Google pixel and runs locally.

ffgjgf1 2y ago
Pixels have a lot more RAM than iPhones though, only low-end Androids devices have as little.

mritchie712 2y ago

> OpenAI won't share model weights Microsoft has 'em [0] 0 - plenty of sources on this, but here's one https://x.com/elonmusk/status/1639138603371491329 https://x.com/elonmusk/status/1639138603371491329