4 ms·

Demos show that you can search for "Blue bag" even if the words "blue bag" don't appear on the screen. If there was a blue bag in a photo, say in a PowerPoint s

Demos show that you can search for "Blue bag" even if the words "blue bag" don't appear on the screen. If there was a blue bag in a photo, say in a PowerPoint slide, it will find it.

cflewis 2y ago
In some ways, I find this even creepier than when I thought it was more than an SQLite database.
cjk2 2y ago
To be fair Apple Photos does this. But it does it to your photos not your entire universe.
- hu3 2y ago
  Same for google photos. I searched for "smiling in beach" and it works flawlessly. I recon Microsoft will want to offload this kind of search processing to user machines with the advent in "AI processor computers" we saw recently: https://blogs.microsoft.com/blog/2024/05/20/introducing-copilot-pcs/ https://blogs.microsoft.com/blog/2024/05/20/introducing-copi...
  - mmcnl 2y ago
    I believe Apple does it completely locally on the device, whereas Google searches in the cloud.
- ethagnawl 2y ago
  Dropbox has also been doing this for years. It's helpful sometimes and other times completely misses the mark. Personally, I do find it to be creepy and it has me edging closer to using a more privacy-focused cloud solution to backup my important documents, photos, etc.
- sleepybrett 2y ago
  spotlight is also indexing pretty much every file you create on your mac.
- acdha 2y ago
  There’s a really interesting angle about how the model of the service fits with the user’s concept and consent here. Apple Photos only indexes things you choose to save there, and you have to grant access to other things like Siri using it, and most people seem to be comfortable with that because it’s easy to understand the scope and opt-out. Recall is the opposite: on by default and it records everything, which leaves people uncertain about whether they even can reliably opt-out or what that will mean if they’re interacting with someone who hasn’t, and combines with other areas of Windows 11 where Microsoft has been doing things like putting ads in the start menu which really cut into users’ trust. The blowback has been pretty impressive for what might otherwise have been a relatively ignored feature.
renewiltord 2y ago
My favourite feature in Apple and Google Photos. I use this all the time and have gotten so used to it I get annoyed when it doesn't find something.
simonw 2y ago
I'd love to know how they are doing this! Are they running an on-device CLIP embedding model of some sort? Could they even be shipping a multi-modal LLM like Phi-3 Vision?
- Firmwarrior 2y ago
  OneDrive and the local Microsoft photos app have been doing it for a decade already.

Demos show that you can search for "Blue bag" even if the words "blue bag" don't appear on the screen. If there was a blue bag in a photo, say in a PowerPoint slide, it will find it.

cflewis 2y ago

In some ways, I find this even creepier than when I thought it was more than an SQLite database.

cjk2 2y ago

To be fair Apple Photos does this. But it does it to your photos not your entire universe.

hu3 2y ago
Same for google photos. I searched for "smiling in beach" and it works flawlessly. I recon Microsoft will want to offload this kind of search processing to user machines with the advent in "AI processor computers" we saw recently: https://blogs.microsoft.com/blog/2024/05/20/introducing-copilot-pcs/ https://blogs.microsoft.com/blog/2024/05/20/introducing-copi...
- mmcnl 2y ago
  I believe Apple does it completely locally on the device, whereas Google searches in the cloud.
ethagnawl 2y ago
Dropbox has also been doing this for years. It's helpful sometimes and other times completely misses the mark. Personally, I do find it to be creepy and it has me edging closer to using a more privacy-focused cloud solution to backup my important documents, photos, etc.
sleepybrett 2y ago
spotlight is also indexing pretty much every file you create on your mac.
acdha 2y ago
There’s a really interesting angle about how the model of the service fits with the user’s concept and consent here. Apple Photos only indexes things you choose to save there, and you have to grant access to other things like Siri using it, and most people seem to be comfortable with that because it’s easy to understand the scope and opt-out. Recall is the opposite: on by default and it records everything, which leaves people uncertain about whether they even can reliably opt-out or what that will mean if they’re interacting with someone who hasn’t, and combines with other areas of Windows 11 where Microsoft has been doing things like putting ads in the start menu which really cut into users’ trust. The blowback has been pretty impressive for what might otherwise have been a relatively ignored feature.

renewiltord 2y ago

My favourite feature in Apple and Google Photos. I use this all the time and have gotten so used to it I get annoyed when it doesn't find something.

simonw 2y ago

I'd love to know how they are doing this! Are they running an on-device CLIP embedding model of some sort? Could they even be shipping a multi-modal LLM like Phi-3 Vision?

Firmwarrior 2y ago
OneDrive and the local Microsoft photos app have been doing it for a decade already.