12 ms·

It doesn't need to stay true forever. The alternative is Google / Android devices and OpenAI wrapper apps, both of which usually offer a half baked UI, poor p

It doesn't need to stay true forever. The alternative is Google / Android devices and OpenAI wrapper apps, both of which usually offer a half baked UI, poor privacy practices, and a completely broken UX when the internet connection isn't perfect. Pair this with the completely subpar Android apps, Google dropping support for an app about once a month, and suddenly I'm okay with the lesser of two evils. I know they aren't running a charity, I even hypothesized that Apple just can't build good services so they pivoted to focusing on this fake "privacy" angle. In the end, iPhones are likely going to be better for edge AI than whatever is out there, so I'm looking forward to this.

jocaal 2y ago
> better for edge AI than whatever is out there, so I'm looking forward to this What exactly are you expecting? The current hype for AI is large language models. The word 'large' has a certain meaning in that context. Much larger that can fit on your phone. Everyone is going crazy about edge AI, what am I missing?
- jchanimal 2y ago
  It fits on your phone, and your phone can offload battery burning tasks to nearby edge servers. Seems like the path consumer-facing AI will take.
  - deleted 2y ago
    [deleted]
- jitl 2y ago
  Quantized LLMs can run on a phone, like Gemini Nano or OpenLLAMA 3B. If a small local model can handle simple stuff and delegate to a model in the data center for harder tasks and with better connectivity you could get an even better experience.
  - SmellTheGlove 2y ago
    > If a small local model can handle simple stuff and delegate to a model in the data center for harder tasks and with better connectivity you could get an even better experience. Distributed mixture of experts sounds like an idea. Is anyone doing that?
    - cheschire 2y ago
      Sounds like an attack vector waiting to happen if you deploy enough competing expert devices into a crowd. I’m imagining a lot of these LLM products on phones will be used for live translation. Imagine a large crowd event of folks utilizing live AI translation services being told completely false translations because an actor deployed a 51% attack.
      - jagger27 2y ago
        I’m not particularly scared of a 51% attack between the devices attached to my Apple ID. If my iPhone splits inference work with my idle MacBook, Apple TV, and iPad, what’s the problem there?
    - moneywoes 2y ago
      what about in situations with no bandwidth?
  - mr_toad 2y ago
    Using RAG a smaller local LLM combined with local data (e.g. your emails, iMessages etc) can be useful than a large external LLM that doesn’t have your data. No point asking GPT4 “what time does John’s party start?”, but a local LLM can do better.
    - jwells89 2y ago
      This is why I think Apple’s implementation of LLMs is going to be a big deal, even if it’s not technically as capable. Just making Siri better able to converse (e.g. ask clarifying questions) and giving it the context offered by user data will make it dramatically more useful than silo’d off remote LLMs.
- callalex 2y ago
  In the hardware world, last year’s large has a way of becoming next year’s small. For a particularly funny example of this, check out the various letter soup names that people keep applying to screen resolutions. https://en.m.wikipedia.org/wiki/Display_resolution_standards#UWQHD https://en.m.wikipedia.org/wiki/Display_resolution_standards...
- gopher_space 2y ago
  > Everyone is going crazy about edge AI, what am I missing? If you clone a model and then bake in a more expensive model's correct/appropriate responses to your queries, you now have the functionality of the expensive model in your clone. For your specific use case. The size of the resulting case-specific models are small enough to run on all kinds of hardware, so everyone's seeing how much work can be done on their laptop right now. One incentive for doing so is that your approaches to problems are constrained by the cost and security of the Q&A roundtrip.
kernal 2y ago
>subpar Android apps Care to cite these subpar Android apps? The app store is filled to the brim with subpar and garbage apps. >Google dropping support for an app about once a month I mean if you're going to lie why not go bigger >I'm okay with the lesser of two evils. So the more evil company is the one that pulled out of China because they refused to hand over their users data to the Chinese government on a fiber optic silver plate?
- martimarkov 2y ago
  Google operates in China albeit via their HK domain. They also had project DragonFly if you remember. The lesser of two evils is that one company doesn’t try to actively profile me (in order for their ads business to be better) with every piece of data it can find and forces me to share all possible data with them. Google is famously known to kill apps that are good and used by customers: https://killedbygoogle.com/ https://killedbygoogle.com/ As for the subpar apps: there is a massive difference between the network traffic when on the Home Screen between iOS and Android.
  - kernal 2y ago
    >Google operates in China albeit via their HK domain. The Chinese government has access to the iCloud account of every Chinese Apple user. >They also had project DragonFly if you remember. Which never materialized. >The lesser of two evils is that one company doesn’t try to actively profile me (in order for their ads business to be better) with every piece of data it can find and forces me to share all possible data with them. Apple does targeted and non targeted advertising as well. Additionally, your carrier has likely sold all of the data they have on you. Apple was also sued for selling user data to ad networks. Odd for a Privacy First company to engage in things like that. >Google is famously known to kill apps that are good and used by customers: https://killedbygoogle.com/ https://killedbygoogle.com/ Google has been around for 26 years I believe. According to that link 60 apps were killed in that timeframe. According to your statement that Google kills an app a month that would leave you 252 apps short. Furthermore, the numbers would indicate that Google has killed 2.3 apps per year or .192 apps per month. >As for the subpar apps: there is a massive difference between the network traffic when on the Home Screen between iOS and Android. Not sure how that has anything to do with app quality, but if network traffic is your concern there's probably a lot more an Android user can do than an iOS user to control or eliminate the traffic.
rfoo 2y ago
> The alternative is Google / Android devices No, the alternative is Android devices with everything except firmware built from source and signed by myself. And at the same time, being secure, too. You just can't have this on Apple devices. On Android side choices are limited too, I don't like Google and especially their disastrous hardware design, but their Pixel line is the most approachable one able to do all these. Heck, you can't even build your own app for your own iPhone without buying another hardware (a Mac, this is not a software issue, this is a legal issue, iOS SDK is licensed to you on the condition of using on Apple hardware only) and a yearly subscription. How is this acceptable at all?
- adamomada 2y ago
  The yearly subscription is for publishing your app on Apple’s store and definitely helps keep some garbage out. Running your own app on your own device is basically solved with free third party solutions now (see AltStore and since a newer method I can’t recall atm)
  - tsimionescu 2y ago
    Those are only available in the EU, and Apple has been huffing and puffing even here.
  - dns_snek 2y ago
    Notice that parent never talked about publishing apps, just building and running apps on their own device. "Publishing on AltStore" (or permanently running the app on your own device in any other way) still requires a $100/year subscription as far as I'm aware.
- simfree 2y ago
  WebGPU and many other features on iOS are unimplemented or implemented in half-assed or downright broken ways. These features work on all the modern desktop browsers and on Android tho!
  - Aloisius 2y ago
cbsmith 2y ago
Google has also been working on (and provides kits for) local machine learning on mobile devices... and they run on both iOS and Android. The Gemini App does send data in to Google for learning, but even that you can opt out of. Apple's definitely pulling a "Heinz" move with privacy, and it is true that they're doing a better job of it overall, but Google's not completely horrible either.
- Wytwwww 2y ago
  You won't be able to run any model even as remotely as capable as Gemini locally though.
  - cbsmith 2y ago
    It's kind of tautological that Gemini Nano is as capable as Gemini and it runs locally. https://store.google.com/intl/en/ideas/articles/pixel-feature-drop-december-2023/ https://store.google.com/intl/en/ideas/articles/pixel-featur...
    - Wytwwww 2y ago
      Yeah, I was thinking of their non-local model like Gemini advanced though. In any case iPhone probably don't have enough memory to run a 3.25B model? e.g. 15 pro only have 8 GB (and Gemini Nano seems to only work on the 12GB Pixel 8 Pro) and 14 has only 6GB, that hardly seems sufficient for even a small LLM if you still you want to run the full OS and other apps at the same time.

jocaal 2y ago

> better for edge AI than whatever is out there, so I'm looking forward to this What exactly are you expecting? The current hype for AI is large language models. The word 'large' has a certain meaning in that context. Much larger that can fit on your phone. Everyone is going crazy about edge AI, what am I missing?

jchanimal 2y ago
It fits on your phone, and your phone can offload battery burning tasks to nearby edge servers. Seems like the path consumer-facing AI will take.
- deleted 2y ago
  [deleted]
jitl 2y ago
Quantized LLMs can run on a phone, like Gemini Nano or OpenLLAMA 3B. If a small local model can handle simple stuff and delegate to a model in the data center for harder tasks and with better connectivity you could get an even better experience.
- SmellTheGlove 2y ago
  > If a small local model can handle simple stuff and delegate to a model in the data center for harder tasks and with better connectivity you could get an even better experience. Distributed mixture of experts sounds like an idea. Is anyone doing that?
  - cheschire 2y ago
    Sounds like an attack vector waiting to happen if you deploy enough competing expert devices into a crowd. I’m imagining a lot of these LLM products on phones will be used for live translation. Imagine a large crowd event of folks utilizing live AI translation services being told completely false translations because an actor deployed a 51% attack.
    - jagger27 2y ago
      I’m not particularly scared of a 51% attack between the devices attached to my Apple ID. If my iPhone splits inference work with my idle MacBook, Apple TV, and iPad, what’s the problem there?
  - moneywoes 2y ago
    what about in situations with no bandwidth?
- mr_toad 2y ago
  Using RAG a smaller local LLM combined with local data (e.g. your emails, iMessages etc) can be useful than a large external LLM that doesn’t have your data. No point asking GPT4 “what time does John’s party start?”, but a local LLM can do better.
  - jwells89 2y ago
    This is why I think Apple’s implementation of LLMs is going to be a big deal, even if it’s not technically as capable. Just making Siri better able to converse (e.g. ask clarifying questions) and giving it the context offered by user data will make it dramatically more useful than silo’d off remote LLMs.
callalex 2y ago
In the hardware world, last year’s large has a way of becoming next year’s small. For a particularly funny example of this, check out the various letter soup names that people keep applying to screen resolutions. https://en.m.wikipedia.org/wiki/Display_resolution_standards#UWQHD https://en.m.wikipedia.org/wiki/Display_resolution_standards...
gopher_space 2y ago
> Everyone is going crazy about edge AI, what am I missing? If you clone a model and then bake in a more expensive model's correct/appropriate responses to your queries, you now have the functionality of the expensive model in your clone. For your specific use case. The size of the resulting case-specific models are small enough to run on all kinds of hardware, so everyone's seeing how much work can be done on their laptop right now. One incentive for doing so is that your approaches to problems are constrained by the cost and security of the Q&A roundtrip.

kernal 2y ago

>subpar Android apps Care to cite these subpar Android apps? The app store is filled to the brim with subpar and garbage apps. >Google dropping support for an app about once a month I mean if you're going to lie why not go bigger >I'm okay with the lesser of two evils. So the more evil company is the one that pulled out of China because they refused to hand over their users data to the Chinese government on a fiber optic silver plate?

martimarkov 2y ago
Google operates in China albeit via their HK domain. They also had project DragonFly if you remember. The lesser of two evils is that one company doesn’t try to actively profile me (in order for their ads business to be better) with every piece of data it can find and forces me to share all possible data with them. Google is famously known to kill apps that are good and used by customers: https://killedbygoogle.com/ https://killedbygoogle.com/ As for the subpar apps: there is a massive difference between the network traffic when on the Home Screen between iOS and Android.
- kernal 2y ago
  >Google operates in China albeit via their HK domain. The Chinese government has access to the iCloud account of every Chinese Apple user. >They also had project DragonFly if you remember. Which never materialized. >The lesser of two evils is that one company doesn’t try to actively profile me (in order for their ads business to be better) with every piece of data it can find and forces me to share all possible data with them. Apple does targeted and non targeted advertising as well. Additionally, your carrier has likely sold all of the data they have on you. Apple was also sued for selling user data to ad networks. Odd for a Privacy First company to engage in things like that. >Google is famously known to kill apps that are good and used by customers: https://killedbygoogle.com/ https://killedbygoogle.com/ Google has been around for 26 years I believe. According to that link 60 apps were killed in that timeframe. According to your statement that Google kills an app a month that would leave you 252 apps short. Furthermore, the numbers would indicate that Google has killed 2.3 apps per year or .192 apps per month. >As for the subpar apps: there is a massive difference between the network traffic when on the Home Screen between iOS and Android. Not sure how that has anything to do with app quality, but if network traffic is your concern there's probably a lot more an Android user can do than an iOS user to control or eliminate the traffic.

rfoo 2y ago

> The alternative is Google / Android devices No, the alternative is Android devices with everything except firmware built from source and signed by myself. And at the same time, being secure, too. You just can't have this on Apple devices. On Android side choices are limited too, I don't like Google and especially their disastrous hardware design, but their Pixel line is the most approachable one able to do all these. Heck, you can't even build your own app for your own iPhone without buying another hardware (a Mac, this is not a software issue, this is a legal issue, iOS SDK is licensed to you on the condition of using on Apple hardware only) and a yearly subscription. How is this acceptable at all?

adamomada 2y ago
The yearly subscription is for publishing your app on Apple’s store and definitely helps keep some garbage out. Running your own app on your own device is basically solved with free third party solutions now (see AltStore and since a newer method I can’t recall atm)
- tsimionescu 2y ago
  Those are only available in the EU, and Apple has been huffing and puffing even here.
- dns_snek 2y ago
  Notice that parent never talked about publishing apps, just building and running apps on their own device. "Publishing on AltStore" (or permanently running the app on your own device in any other way) still requires a $100/year subscription as far as I'm aware.
simfree 2y ago
WebGPU and many other features on iOS are unimplemented or implemented in half-assed or downright broken ways. These features work on all the modern desktop browsers and on Android tho!
- Aloisius 2y ago

cbsmith 2y ago

Google has also been working on (and provides kits for) local machine learning on mobile devices... and they run on both iOS and Android. The Gemini App does send data in to Google for learning, but even that you can opt out of. Apple's definitely pulling a "Heinz" move with privacy, and it is true that they're doing a better job of it overall, but Google's not completely horrible either.

Wytwwww 2y ago
You won't be able to run any model even as remotely as capable as Gemini locally though.
- cbsmith 2y ago
  It's kind of tautological that Gemini Nano is as capable as Gemini and it runs locally. https://store.google.com/intl/en/ideas/articles/pixel-feature-drop-december-2023/ https://store.google.com/intl/en/ideas/articles/pixel-featur...
  - Wytwwww 2y ago
    Yeah, I was thinking of their non-local model like Gemini advanced though. In any case iPhone probably don't have enough memory to run a 3.25B model? e.g. 15 pro only have 8 GB (and Gemini Nano seems to only work on the 12GB Pixel 8 Pro) and 14 has only 6GB, that hardly seems sufficient for even a small LLM if you still you want to run the full OS and other apps at the same time.