4 ms·

I'm sure it's part of their ML with the newer iOS but this does make me happy to think of the AI running on the hardware. Yesterday when driving I got a message

I'm sure it's part of their ML with the newer iOS but this does make me happy to think of the AI running on the hardware. Yesterday when driving I got a message and usually it would just say "Jane says 'blah blah' and sent a photo." But last night Siri in CarPlay said "Jane says 'blah blah' and sent a photo of a dog in a blanket on a couch". And when I checked the photo they sent it was exactly that. If it's E2E encrypted, I don't think it'd be able to identify that during transit, so it had to have been locally done. Thought it was interesting, and a nice QoL update. Excited to see how this can grow if they introduce more things to be processed with AI on the hardware.

bayindirh 3y ago
I love how Apple obsessively does things on the edge, on the device itself rather than on the cloud. Sometimes my lowly iPhone X gets super hot during charging. Next morning I either get the facial recognition updated, a new memory generated or something else has happened. It's nice, confidence evoking even. The features you like are run on your device, with your data not leaving your device, on its free time. This is the AI I like. Personal, confined and private.
- chatmasta 3y ago
  They're in the best possible position to innovate on the literal edge, because they've got a fleet of billions of devices that mostly auto-update. As hardware advances, Apple will have the clearest path to deploying truly personal and locally hosted AI.
  - consumer451 3y ago
    > truly personal and locally hosted AI. If we can combine this with a neuro bill of privacy/rights [0] then the future might not be as bleak as it seems. [0] https://www.cirsd.org/en/horizons/horizons-winter-2021-issue-no-18/its-time-for-neuro--rights https://www.cirsd.org/en/horizons/horizons-winter-2021-issue... > Sean Carroll & Nita Farahany on Ethics, Law, and Neurotechnology https://www.preposterousuniverse.com/podcast/2023/03/13/229-nita-farahany-on-ethics-law-and-neurotechnology/ https://www.preposterousuniverse.com/podcast/2023/03/13/229-...
- crazygringo 3y ago
  And because it's their business model, you don't have to worry about it changing. They do stuff on-device because they want to keep selling expensive devices. It's fantastic for them that they sell the hardware, unlike Google/OpenAI that have to spend on hardware.
  - pertymcpert 3y ago
    More so these days you buy some expensive hardware but keeping subscribing to apps and buy TV shows and cloud storage from them due to convenience. And then you buy the next model after a few years because why would you change a working formula for some Samsung/Google android BS that won’t be as clean….
kergonath 3y ago
> If it's E2E encrypted, I don't think it'd be able to identify that during transit, so it had to have been locally done. Yes. Image description has been done on-device for quite a while now, and it is great. You can look for pictures using a description, and you can look for any text in any picture as well. It works very, very well.
- ryanwaggoner 3y ago
  Yeah, but I agree with the parent; I think something has changed recently with Siri. The other day I asked it some questions as a joke that I knew would just say "here's what I found on the web", but it answered them. It made me wonder what they're changing under the hood.
- sn0wf1re 3y ago
  Yeah, apparently well enough to run a web service off of iPhone image recognition! https://findthatmeme.com/blog/2023/01/08/image-stacks-and-iphone-racks-building-an-internet-scale-meme-search-engine-Qzrz7V6T.html https://findthatmeme.com/blog/2023/01/08/image-stacks-and-ip...
  - kergonath 3y ago
    That is seriously impressive!
- nerdbert 3y ago
  > It works very, very well. I had always assumed that search box in Photos.app would be useless, but once in desperation I typed something in and it found exactly what I was looking for (a photo of one of my kids' passports I knew I'd taken but could not for the life of me remember where). Now I use it more and more, it makes my tens of thousands of photos dramatically more useful to me. That time we at at the restaurant with the big shrimp platter? The airbnb we stayed at where I took a picture of the cool coffee table? All right there.
  - pertymcpert 3y ago
baz00 3y ago
Apple photos is wonderful for this. I can search for “cows” and I get photos of cows. All on device in the middle of a mountain with no internet 50 miles in any direction.
- MBCook 3y ago
  Plus I believe it indexes the text in images too. So if you remember something on a sign that you took a picture of it can find the picture without you having to know where/when it was.
poyu 3y ago
It is amazing for accessibility users too, if you turn on VoiceOver, the Camera app and the Photos app tells you the description of what’s currently on display. And yes, it has been a feature for a while.
- MBCook 3y ago
  The magnifier app has that and reads the results live as you point the phone at things. It also has other accessibility features like helping you find doors. You may need to turn something on in settings before all of this shows up though, I don’t remember.
MBCook 3y ago
It was! And it works surprisingly well. It won’t describe movies/animated gifs or images that are very small though. It’s new for iOS 17.

bayindirh 3y ago

I love how Apple obsessively does things on the edge, on the device itself rather than on the cloud. Sometimes my lowly iPhone X gets super hot during charging. Next morning I either get the facial recognition updated, a new memory generated or something else has happened. It's nice, confidence evoking even. The features you like are run on your device, with your data not leaving your device, on its free time. This is the AI I like. Personal, confined and private.

chatmasta 3y ago
They're in the best possible position to innovate on the literal edge, because they've got a fleet of billions of devices that mostly auto-update. As hardware advances, Apple will have the clearest path to deploying truly personal and locally hosted AI.
- consumer451 3y ago
  > truly personal and locally hosted AI. If we can combine this with a neuro bill of privacy/rights [0] then the future might not be as bleak as it seems. [0] https://www.cirsd.org/en/horizons/horizons-winter-2021-issue-no-18/its-time-for-neuro--rights https://www.cirsd.org/en/horizons/horizons-winter-2021-issue... > Sean Carroll & Nita Farahany on Ethics, Law, and Neurotechnology https://www.preposterousuniverse.com/podcast/2023/03/13/229-nita-farahany-on-ethics-law-and-neurotechnology/ https://www.preposterousuniverse.com/podcast/2023/03/13/229-...
crazygringo 3y ago
And because it's their business model, you don't have to worry about it changing. They do stuff on-device because they want to keep selling expensive devices. It's fantastic for them that they sell the hardware, unlike Google/OpenAI that have to spend on hardware.
- pertymcpert 3y ago
  More so these days you buy some expensive hardware but keeping subscribing to apps and buy TV shows and cloud storage from them due to convenience. And then you buy the next model after a few years because why would you change a working formula for some Samsung/Google android BS that won’t be as clean….

kergonath 3y ago

> If it's E2E encrypted, I don't think it'd be able to identify that during transit, so it had to have been locally done. Yes. Image description has been done on-device for quite a while now, and it is great. You can look for pictures using a description, and you can look for any text in any picture as well. It works very, very well.

ryanwaggoner 3y ago
Yeah, but I agree with the parent; I think something has changed recently with Siri. The other day I asked it some questions as a joke that I knew would just say "here's what I found on the web", but it answered them. It made me wonder what they're changing under the hood.
sn0wf1re 3y ago
Yeah, apparently well enough to run a web service off of iPhone image recognition! https://findthatmeme.com/blog/2023/01/08/image-stacks-and-iphone-racks-building-an-internet-scale-meme-search-engine-Qzrz7V6T.html https://findthatmeme.com/blog/2023/01/08/image-stacks-and-ip...
- kergonath 3y ago
  That is seriously impressive!
nerdbert 3y ago
> It works very, very well. I had always assumed that search box in Photos.app would be useless, but once in desperation I typed something in and it found exactly what I was looking for (a photo of one of my kids' passports I knew I'd taken but could not for the life of me remember where). Now I use it more and more, it makes my tens of thousands of photos dramatically more useful to me. That time we at at the restaurant with the big shrimp platter? The airbnb we stayed at where I took a picture of the cool coffee table? All right there.
- pertymcpert 3y ago

baz00 3y ago

Apple photos is wonderful for this. I can search for “cows” and I get photos of cows. All on device in the middle of a mountain with no internet 50 miles in any direction.

MBCook 3y ago
Plus I believe it indexes the text in images too. So if you remember something on a sign that you took a picture of it can find the picture without you having to know where/when it was.

poyu 3y ago

It is amazing for accessibility users too, if you turn on VoiceOver, the Camera app and the Photos app tells you the description of what’s currently on display. And yes, it has been a feature for a while.

MBCook 3y ago
The magnifier app has that and reads the results live as you point the phone at things. It also has other accessibility features like helping you find doors. You may need to turn something on in settings before all of this shows up though, I don’t remember.

MBCook 3y ago

It was! And it works surprisingly well. It won’t describe movies/animated gifs or images that are very small though. It’s new for iOS 17.