9 ms·

Facebook’s AI unlocks the ability to search photos by what’s in them

73 points by kosei 10y ago

fapjacks 10y ago
Facebook's search is not useful. It's (obviously) segmented based on your own network, and generally useless for practical purposes. Perhaps law enforcement will find some use in this, but most people won't. Or at least, any utility beyond "Oh, remember that time I took a picture with that weird dog in it?"
- bpodgursky 10y ago
  Your power of imagination is truly inspirational.
  - fapjacks 10y ago
    I take it you haven't actually tried to put Facebook search to practical use. I used to run a nonprofit for combat veterans (since killed by California's AMT on all businesses including nonprofits) and used to pick up new candidates through word of mouth. I would then have to find these people often by name only. Facebook's graph search used to be okay for this purpose, but they've completely neutered all the utility out of their search over the years.
  - chrischen 10y ago
    He's talking about its current public potential, not idealized potential. For example go on facebook and use the search and tell me one interesing use case.
- jackhack 10y ago
  My worry: It will someday let agents scan your images for items of interest in an automated fashion. Firearms. Flags. Knives. Religious symbols. It's not to great a leap to imagine this happening in a very restrictive culture where possession of the wrong religious text could be considered criminal. Or a deck of playing cards. A cassette tape of music. Alcohol or Cigarettes. etc. Never forget technology is just a tool which can be used by dark sides of humanity too.
- forgottenpass 10y ago
  The analysis of the images isn't meant for the users, it's so Facebook knows what's in them. Adding a user-facing interface is an easy-to-add toy for users and an attempt to recruit them into improving training data.
ijafri 10y ago
now i know why facebook has been doin this. http://i.imgur.com/zPapJHM.png http://i.imgur.com/zPapJHM.png
- zapt02 10y ago
  I believe that was for accessibility reasons, ie screen readers.
  - mdrzn 10y ago
    but it also allows you to do the opposite things, like search for "sky" and find that image.
    - zapt02 10y ago
      didn't know that, cool!
- hobarrera 10y ago
  "alt" is what's read out by screen readers, or shown when an image fails to load (you'd see this a lot in a high latency connection).
justinjlynn 10y ago
Thinking of just how difficult this is, I'm reminded of https://xkcd.com/1425/ https://xkcd.com/1425/.
- neilcarpenter 10y ago
  And see http://parkorbird.flickr.com/ http://parkorbird.flickr.com/
  - timbre 10y ago
    Also http://birdsnap.com http://birdsnap.com and http://merlin.allaboutbirds.org/photo-id/ http://merlin.allaboutbirds.org/photo-id/
- oceanswave 10y ago
  To help with the implementation: https://www.microsoft.com/cognitive-services/en-us/apis https://www.microsoft.com/cognitive-services/en-us/apis https://www.ibm.com/watson/developercloud/services-catalog.html https://www.ibm.com/watson/developercloud/services-catalog.h... https://cloud.google.com/products/machine-learning/ https://cloud.google.com/products/machine-learning/
googleapologist 10y ago
I don't see how this is unlocking anything. Google has been doing the same for the past 5 years.
philsnow 10y ago
Google photos does this. I just opened Google photos and searched for "food" and aside from a couple pictures of some jack o lanterns (which are arguable, but I don't know anybody who eats theirs on Nov 1st), the results were spot on. "Unlocks" might not be the right word here.
- baby 10y ago
  same in iOS, you can search your photos like that. Actually, if you looked at the source on facebook, you could see photos being tagged with things like "one person, two people, smiles"
  - kalleboo 10y ago
    If you turn on VoiceOver on iOS (accessibility feature for the blind) and then tap on photos on the Photos app, it will read out whatever it found in that photo
  - dx034 10y ago
    The iOS version doesn't work very well for me. It often uses the location of when the picture was stored. If you receive a picture from someone showing Big Ben while you're in New York, it'll show the picture when you search for new york. So searching for places doesn't really work, searching in contents also seems to be much worse than Google. Another point where Apple apparently lags behind in the AI game.
    - gvurrdon 10y ago
      I found that both the Google and iOS versions could perform much the same sort of searches (e.g. "cat", "house") with roughly similar outcomes. However, Apple chooses to censor some searches (nothing illegal or dodgy, of course) and I could obtain no results for these despite having images which Google could find.
- aptwebapps 10y ago
  I think the idea here is to find things by analyzing the pictures directly as opposed to just searching contextual data like captions and link text. No idea if Google does that or not. Edit: Didn't know Google Photos was a thing and assumed parent meant Google Images. Carry on.
yunolisten 10y ago
Seriously, a 51.6MB "gif" that doesn't really add anything to the article?
- iklos55 10y ago
  Really off-topic, but: Are there (tech) news-sites which do not have this 10-100MB bloat with three js framweorks and gifs, ads and whatnot? I'd gladly pay for a minimalist news source...
  - throwanem 10y ago
    There's AMP, if you don't mind trading one set of problems for another.
  - CaptSpify 10y ago
    It's kind of a PIA, but I enjoy using umatrix and ublock origin. Allows me to only turn on the stuff that I want.
    - naranana 10y ago
      https://github.com/gorhill/uBlock/wiki/Per-site-switches#no-large-media-elements https://github.com/gorhill/uBlock/wiki/Per-site-switches#no-...
  - forgetsusername 10y ago
    >Are there (tech) news-sites which do not have this 10-100MB bloat with three js framweorks and gifs, ads and whatnot? The HackerNews comment section, with the added benefit of it being more informative.
  - castratikron 10y ago
    You could do what I do and open them in Links/Lynx, a command-line browser. You only need the text, anyways, right?
    - 10y ago
planetjones 10y ago
Must be unlocking the same thing that Amazon Prime Photos is doing. I have to say that scanning photos automatically and allowing me to search by what is in the photo is the only solution to cope with the proliferation of photos, especially from camera phones. I still curate my better photos in Lightroom and publish them to Flickr albums. But that approach just isn't scalable for all the "normal" photos I am taking with the iPhone; so privacy concerns aside, I am happy for a machine to help me find the photos I am interested in.
Raphmedia 10y ago
If you want to see what Facebook tag your photos you can install this chrome extension: [Show Facebook Computer Vision Tags ] https://chrome.google.com/webstore/detail/show-facebook-computer-vi/elafbihhbfmfihdflghclaclcilcnmie https://chrome.google.com/webstore/detail/show-facebook-comp...
zaque1213 10y ago
Recently on my phone I tried to copy and paste a picture from Facebook to my messaging app. Instead of copying the picture, it copied a text message that read "picture of dog and three people." I'm sure this is nothing new, but I'd never seen it before and thought it was interesting.
- hatsunearu 10y ago
  This reminds me, (very unrelated) a few years ago (almost 10), when I dragged and dropped an image from Wikipedia with Google Chrome to my Windows XP desktop, and to my surprise it actually saved it with a file name that is the description of the picture, not some arbitrarily chosen name or the Wikipedia file name. Sometimes it was really long and descriptive, like "A man wearing red and a woman wearing blue are playing soccer on a field blah blah blah". I wondered where that came from, and I looked for that description on that image's page on Wikipedia and couldn't find the same description anywhere (keep in mind, I was a middle schooler or something and it could very well be the case that I overlooked something) I was a little spooked, and I thought it was some sort of hidden feature in Chrome that labeled images like that. Does anyone know what was happening? It still bugs me to this day :P
  - TeMPOraL 10y ago
    AFAIR you can suggest a name for the file in response headers of HTTP request, so I'd suggest cURLing the image and looking at the response headers.
  - Kronopath 10y ago
    Could it have been the alt text for the image? I believe the grandparent comment was also pasting the alt text, since Facebook auto-generates alt text for blind users using ML (this is mentioned in the article).
jxy 10y ago
Google photos does this through their server. Apple Photos does this through your phone, or your laptop. (Data from these two are not synced???)
newscracker 10y ago
In the recent times, for some other unrelated reasons, my browsing of FB pages and people's profiles was a bit slow from the network and download side (I always use a browser to visit Facebook, not the app). I noticed how accurate Facebook was in classifying photos when I saw the alt text captions for photos (before the image loaded). It was impressive, being able to say approximately how many people were there, whether a person was smiling, whether a photo was a selfie, whether a person was standing, whether a picture contained a landscape, trees and many other things. It was also creepy that Facebook knew so much, because though I use Facebook for specific purposes and try to limit what I put there, I didn't imagine that images could convey so much information/metadata that would be useful to the underlying platform.
jeffrey-sean 10y ago
We have a free app that uses AI to find unprofessional Facebook, Instagram, and Twitter photos used by over 50,000 college students: https://www.renpup.com/ https://www.renpup.com/

73 points by kosei 10y ago

Facebook's search is not useful. It's (obviously) segmented based on your own network, and generally useless for practical purposes. Perhaps law enforcement will find some use in this, but most people won't. Or at least, any utility beyond "Oh, remember that time I took a picture with that weird dog in it?"

bpodgursky 10y ago
Your power of imagination is truly inspirational.
- fapjacks 10y ago
  I take it you haven't actually tried to put Facebook search to practical use. I used to run a nonprofit for combat veterans (since killed by California's AMT on all businesses including nonprofits) and used to pick up new candidates through word of mouth. I would then have to find these people often by name only. Facebook's graph search used to be okay for this purpose, but they've completely neutered all the utility out of their search over the years.
- chrischen 10y ago
  He's talking about its current public potential, not idealized potential. For example go on facebook and use the search and tell me one interesing use case.
jackhack 10y ago
My worry: It will someday let agents scan your images for items of interest in an automated fashion. Firearms. Flags. Knives. Religious symbols. It's not to great a leap to imagine this happening in a very restrictive culture where possession of the wrong religious text could be considered criminal. Or a deck of playing cards. A cassette tape of music. Alcohol or Cigarettes. etc. Never forget technology is just a tool which can be used by dark sides of humanity too.
forgottenpass 10y ago
The analysis of the images isn't meant for the users, it's so Facebook knows what's in them. Adding a user-facing interface is an easy-to-add toy for users and an attempt to recruit them into improving training data.

ijafri 10y ago

now i know why facebook has been doin this. http://i.imgur.com/zPapJHM.png http://i.imgur.com/zPapJHM.png

zapt02 10y ago
I believe that was for accessibility reasons, ie screen readers.
- mdrzn 10y ago
  but it also allows you to do the opposite things, like search for "sky" and find that image.
  - zapt02 10y ago
    didn't know that, cool!
hobarrera 10y ago
"alt" is what's read out by screen readers, or shown when an image fails to load (you'd see this a lot in a high latency connection).

justinjlynn 10y ago

Thinking of just how difficult this is, I'm reminded of https://xkcd.com/1425/ https://xkcd.com/1425/.

neilcarpenter 10y ago
And see http://parkorbird.flickr.com/ http://parkorbird.flickr.com/
- timbre 10y ago
  Also http://birdsnap.com http://birdsnap.com and http://merlin.allaboutbirds.org/photo-id/ http://merlin.allaboutbirds.org/photo-id/
oceanswave 10y ago
To help with the implementation: https://www.microsoft.com/cognitive-services/en-us/apis https://www.microsoft.com/cognitive-services/en-us/apis https://www.ibm.com/watson/developercloud/services-catalog.html https://www.ibm.com/watson/developercloud/services-catalog.h... https://cloud.google.com/products/machine-learning/ https://cloud.google.com/products/machine-learning/

googleapologist 10y ago

I don't see how this is unlocking anything. Google has been doing the same for the past 5 years.

philsnow 10y ago

Google photos does this. I just opened Google photos and searched for "food" and aside from a couple pictures of some jack o lanterns (which are arguable, but I don't know anybody who eats theirs on Nov 1st), the results were spot on. "Unlocks" might not be the right word here.

baby 10y ago
same in iOS, you can search your photos like that. Actually, if you looked at the source on facebook, you could see photos being tagged with things like "one person, two people, smiles"
- kalleboo 10y ago
  If you turn on VoiceOver on iOS (accessibility feature for the blind) and then tap on photos on the Photos app, it will read out whatever it found in that photo
- dx034 10y ago
  The iOS version doesn't work very well for me. It often uses the location of when the picture was stored. If you receive a picture from someone showing Big Ben while you're in New York, it'll show the picture when you search for new york. So searching for places doesn't really work, searching in contents also seems to be much worse than Google. Another point where Apple apparently lags behind in the AI game.
  - gvurrdon 10y ago
    I found that both the Google and iOS versions could perform much the same sort of searches (e.g. "cat", "house") with roughly similar outcomes. However, Apple chooses to censor some searches (nothing illegal or dodgy, of course) and I could obtain no results for these despite having images which Google could find.
aptwebapps 10y ago
I think the idea here is to find things by analyzing the pictures directly as opposed to just searching contextual data like captions and link text. No idea if Google does that or not. Edit: Didn't know Google Photos was a thing and assumed parent meant Google Images. Carry on.

yunolisten 10y ago

Seriously, a 51.6MB "gif" that doesn't really add anything to the article?

iklos55 10y ago
Really off-topic, but: Are there (tech) news-sites which do not have this 10-100MB bloat with three js framweorks and gifs, ads and whatnot? I'd gladly pay for a minimalist news source...
- throwanem 10y ago
  There's AMP, if you don't mind trading one set of problems for another.
- CaptSpify 10y ago
  It's kind of a PIA, but I enjoy using umatrix and ublock origin. Allows me to only turn on the stuff that I want.
  - naranana 10y ago
    https://github.com/gorhill/uBlock/wiki/Per-site-switches#no-large-media-elements https://github.com/gorhill/uBlock/wiki/Per-site-switches#no-...
- forgetsusername 10y ago
  >Are there (tech) news-sites which do not have this 10-100MB bloat with three js framweorks and gifs, ads and whatnot? The HackerNews comment section, with the added benefit of it being more informative.
- castratikron 10y ago
  You could do what I do and open them in Links/Lynx, a command-line browser. You only need the text, anyways, right?
  - 10y ago

planetjones 10y ago

Must be unlocking the same thing that Amazon Prime Photos is doing. I have to say that scanning photos automatically and allowing me to search by what is in the photo is the only solution to cope with the proliferation of photos, especially from camera phones. I still curate my better photos in Lightroom and publish them to Flickr albums. But that approach just isn't scalable for all the "normal" photos I am taking with the iPhone; so privacy concerns aside, I am happy for a machine to help me find the photos I am interested in.

Raphmedia 10y ago

If you want to see what Facebook tag your photos you can install this chrome extension: [Show Facebook Computer Vision Tags ] https://chrome.google.com/webstore/detail/show-facebook-computer-vi/elafbihhbfmfihdflghclaclcilcnmie https://chrome.google.com/webstore/detail/show-facebook-comp...

zaque1213 10y ago

Recently on my phone I tried to copy and paste a picture from Facebook to my messaging app. Instead of copying the picture, it copied a text message that read "picture of dog and three people." I'm sure this is nothing new, but I'd never seen it before and thought it was interesting.

hatsunearu 10y ago
This reminds me, (very unrelated) a few years ago (almost 10), when I dragged and dropped an image from Wikipedia with Google Chrome to my Windows XP desktop, and to my surprise it actually saved it with a file name that is the description of the picture, not some arbitrarily chosen name or the Wikipedia file name. Sometimes it was really long and descriptive, like "A man wearing red and a woman wearing blue are playing soccer on a field blah blah blah". I wondered where that came from, and I looked for that description on that image's page on Wikipedia and couldn't find the same description anywhere (keep in mind, I was a middle schooler or something and it could very well be the case that I overlooked something) I was a little spooked, and I thought it was some sort of hidden feature in Chrome that labeled images like that. Does anyone know what was happening? It still bugs me to this day :P
- TeMPOraL 10y ago
  AFAIR you can suggest a name for the file in response headers of HTTP request, so I'd suggest cURLing the image and looking at the response headers.
- Kronopath 10y ago
  Could it have been the alt text for the image? I believe the grandparent comment was also pasting the alt text, since Facebook auto-generates alt text for blind users using ML (this is mentioned in the article).

jxy 10y ago

Google photos does this through their server. Apple Photos does this through your phone, or your laptop. (Data from these two are not synced???)

newscracker 10y ago

In the recent times, for some other unrelated reasons, my browsing of FB pages and people's profiles was a bit slow from the network and download side (I always use a browser to visit Facebook, not the app). I noticed how accurate Facebook was in classifying photos when I saw the alt text captions for photos (before the image loaded). It was impressive, being able to say approximately how many people were there, whether a person was smiling, whether a photo was a selfie, whether a person was standing, whether a picture contained a landscape, trees and many other things. It was also creepy that Facebook knew so much, because though I use Facebook for specific purposes and try to limit what I put there, I didn't imagine that images could convey so much information/metadata that would be useful to the underlying platform.

jeffrey-sean 10y ago

We have a free app that uses AI to find unprofessional Facebook, Instagram, and Twitter photos used by over 50,000 college students: https://www.renpup.com/ https://www.renpup.com/