5 ms·

I think that running AI on device is the clear next step in the world of super powerful portable phones/devices. I have a 2021 model phone with very powerful sp

by tkems 3y ago

I think that running AI on device is the clear next step in the world of super powerful portable phones/devices. I have a 2021 model phone with very powerful specs (granted it is the high end model that I got used). With nearly the same RAM and storage as some of my laptops from a few years ago, I see that Apple might want to offload some of their expensive AI apps to these powerful user devices. I see this also as a privacy win as I don't want any of my data being used for training. I also like to have control over what models I use and the ethics the models follow.

itake 3y ago
I am cautiously optimistic, but the reality is these models are huge. LLama 7b is 13g. Even if there is enough cpu cycles, the phone's storage can't support multiple models (and there will need to be multiple models). Compare that to cloud, you can run the latest and greatest model and effectively unlimited storage.
- mejutoco 3y ago
  But the cloud would cost apple money, while if ran in your own device they can sell you the hardware for a healthy margin, and market privacy. I think a single useful model on device or many tiny ones are feasible. Even today, the photos app on my iphone knows what pictures are dog pictures, can remove backgrounds, etc.
  - josho 3y ago
    It would simply become a part of the iCloud+ bundle and contribute to service revenue. Hell, that’s likely what will happen anyway. The models run on device with cloud assistance.
- ynniv 3y ago
  A reasonable quantization (Q4_K_M) is 4GB on disk and uses 6.5 GB of RAM. It will run at reasonable speed on an iPhone 15 Pro today. Using a smaller model or a more efficient quantization on a phone two years from now (aka whatever's in their lab today) will be easier than running the latest First Person Shooter. Welcome to the future. Edit: I see a few comments about hundreds of GB. That works for storage, but DRAM consumes battery just by existing, and that's been keeping RAM sizes from increasing the way storage has.
- deleted 3y ago
  [deleted]
- wayfinder 3y ago
  13 GB a lot? What year is it? 1995? If Apple or Google wanted to add 1 TB to their phones, it wouldn’t cost them anywhere near what they charge us to add 256 GB. A copy of GTA V takes like 100 GB.
- furyofantares 3y ago
  Inference/RAM is 100% the bottleneck, not storage, and it isn't close. These models all have to be fully loaded in memory to run. I think the max RAM on an iPhone is 8GB, max storage is 1TB, so we're talking 2 orders of magnitude more ability to store models vs actually running them. And of course you'd really rather not use the full RAM for inference.
  - BryantD 3y ago
    You're 100% correct on the technical challenges here, but the fully loaded requirement is what Apple is trying to address as per the paper they just published. I'm somewhat skeptical that they've solved it since it's a hard problem. We'll see!
- theshrike79 3y ago
  I'm currently running dolphin 2.2.1 mistral on an M1 Mac Mini with 16GB of RAM. Zero optimisations, just running base LM Studio, I haven't even bothered with GPU acceleration. It thinks about 5-10 seconds before answering and writes the answer about the speed I can read it. The current top of the line iPads are more powerful than this and an optimised modern iPhone implementation using the ML hardware should be faster than my current setup.
- tkems 3y ago
  Just as an example, my phone has 512gb of onboard storage and 16gb of RAM. I think it is safe to say that in the not-to-distant future, we will see the option to run multiple models on device with only a slight storage hit.
criley2 3y ago
I wager there is literally zero chance that Apple will let you have any freedom in the model used, and there's zero chance that you'll be using anything except the Siri_Model unless you hack your iDevice. Just like there is 1 app store and 1 assistant, Apple will fight forcefully to make sure there is 1 AI model.
- eitland 3y ago
  EU is already demanding that Apple allows more app stores, isn't it?
  - criley2 3y ago
    Well, then if you can convince the EU to force Apple to allow uncensored models that will happily teach you how to make bombs and all manner of things like that, then you can have those models on your iDevice. Not sure the EU is going to jump on board that kind of free speech though.
    - dzhiurgis 3y ago
      Can you allow uncensored models but they need to be wrapped around Apple's on-device censorship firewall? Or is there a common API for llm constitution (or however it is called)?
      - MBCook 3y ago
        Apple already has the CoreML framework anyone can use to do on device processing. They even have tools to help convert things into the format the iPhone uses. As others have pointed out you can already find apps with public models on the Apple App Store. They’re not doing anything to censor any of this.
    - danogentili 3y ago
      You could make the same (unconvincing) argument for apps on third-party stores, teaching the same dangerous things or doing even worse things like (gasp) allowing pornography :P
colechristensen 3y ago
My next step at least is various models running at home that i can talk to remotely with my phone. Join a discord channel and talk to my bots. My bots respond or do things back in the channel or with whatever else they have access to.

by tkems 3y ago

itake 3y ago

I am cautiously optimistic, but the reality is these models are huge. LLama 7b is 13g. Even if there is enough cpu cycles, the phone's storage can't support multiple models (and there will need to be multiple models). Compare that to cloud, you can run the latest and greatest model and effectively unlimited storage.

mejutoco 3y ago
But the cloud would cost apple money, while if ran in your own device they can sell you the hardware for a healthy margin, and market privacy. I think a single useful model on device or many tiny ones are feasible. Even today, the photos app on my iphone knows what pictures are dog pictures, can remove backgrounds, etc.
- josho 3y ago
  It would simply become a part of the iCloud+ bundle and contribute to service revenue. Hell, that’s likely what will happen anyway. The models run on device with cloud assistance.
ynniv 3y ago
A reasonable quantization (Q4_K_M) is 4GB on disk and uses 6.5 GB of RAM. It will run at reasonable speed on an iPhone 15 Pro today. Using a smaller model or a more efficient quantization on a phone two years from now (aka whatever's in their lab today) will be easier than running the latest First Person Shooter. Welcome to the future. Edit: I see a few comments about hundreds of GB. That works for storage, but DRAM consumes battery just by existing, and that's been keeping RAM sizes from increasing the way storage has.
deleted 3y ago
[deleted]
wayfinder 3y ago
13 GB a lot? What year is it? 1995? If Apple or Google wanted to add 1 TB to their phones, it wouldn’t cost them anywhere near what they charge us to add 256 GB. A copy of GTA V takes like 100 GB.
furyofantares 3y ago
Inference/RAM is 100% the bottleneck, not storage, and it isn't close. These models all have to be fully loaded in memory to run. I think the max RAM on an iPhone is 8GB, max storage is 1TB, so we're talking 2 orders of magnitude more ability to store models vs actually running them. And of course you'd really rather not use the full RAM for inference.
- BryantD 3y ago
  You're 100% correct on the technical challenges here, but the fully loaded requirement is what Apple is trying to address as per the paper they just published. I'm somewhat skeptical that they've solved it since it's a hard problem. We'll see!
theshrike79 3y ago
I'm currently running dolphin 2.2.1 mistral on an M1 Mac Mini with 16GB of RAM. Zero optimisations, just running base LM Studio, I haven't even bothered with GPU acceleration. It thinks about 5-10 seconds before answering and writes the answer about the speed I can read it. The current top of the line iPads are more powerful than this and an optimised modern iPhone implementation using the ML hardware should be faster than my current setup.
tkems 3y ago
Just as an example, my phone has 512gb of onboard storage and 16gb of RAM. I think it is safe to say that in the not-to-distant future, we will see the option to run multiple models on device with only a slight storage hit.

criley2 3y ago

I wager there is literally zero chance that Apple will let you have any freedom in the model used, and there's zero chance that you'll be using anything except the Siri_Model unless you hack your iDevice. Just like there is 1 app store and 1 assistant, Apple will fight forcefully to make sure there is 1 AI model.

eitland 3y ago
EU is already demanding that Apple allows more app stores, isn't it?
- criley2 3y ago
  Well, then if you can convince the EU to force Apple to allow uncensored models that will happily teach you how to make bombs and all manner of things like that, then you can have those models on your iDevice. Not sure the EU is going to jump on board that kind of free speech though.
  - dzhiurgis 3y ago
    Can you allow uncensored models but they need to be wrapped around Apple's on-device censorship firewall? Or is there a common API for llm constitution (or however it is called)?
    - MBCook 3y ago
      Apple already has the CoreML framework anyone can use to do on device processing. They even have tools to help convert things into the format the iPhone uses. As others have pointed out you can already find apps with public models on the Apple App Store. They’re not doing anything to censor any of this.
  - danogentili 3y ago
    You could make the same (unconvincing) argument for apps on third-party stores, teaching the same dangerous things or doing even worse things like (gasp) allowing pornography :P

colechristensen 3y ago

My next step at least is various models running at home that i can talk to remotely with my phone. Join a discord channel and talk to my bots. My bots respond or do things back in the channel or with whatever else they have access to.