8 ms·

Facebook’s image outage reveals how the company’s AI tags your photos

183 points by saravana85 7y ago

ablation 7y ago
You could also see this if you viewed source when it was fully working, of course. And occasionally before images loaded in if you were on a slow connection. I don't think it's any great secret, but still interesting to see.
- ahje 7y ago
  I was thinking the same thing. I've noticed this multiple times in the past, but I guess alt-tags are too archaic for Average Joe.
- Kronen 7y ago
  It happens sometimes in Instagram without any big outage
- BinaryIdiot 7y ago
  I mean, journalists could also see this if they employed people with disabilities that required them to use such accessibility features. Considering it took an outage to show them I'm guessing few to zero companies do.
SpaceInvader 7y ago
This is nothing new, I've noticed that several months ago and talked about that with one of my security friends and he confirmed that Facebook is tagging pictures at least since 2016.
- likpok 7y ago
  Here's the announcement, even: https://code.fb.com/ios/under-the-hood-building-accessibility-tools-for-the-visually-impaired-on-facebook/ https://code.fb.com/ios/under-the-hood-building-accessibilit...
silviumatei 7y ago
I know for sure that this technology improves life of visual impaired persons as special software will read respective text.
mburst 7y ago
Looks like alt tags for accessibility purposes. Seems like a win win for all users. Easier to find photos and the ability for visually impaired users to get context on what their friends are sharing
burk96 7y ago
Not only is this nothing new as other commenters have observed, these tags have been used by screen readers and other accessibility services for awhile. While we can talk all we want about Facebook using their all seeing eyes for evil (which I agree they overstep a lot of boundaries), these tags are very useful for people who may require them and should not be scrutinized as an invasion of privacy.
mevile 7y ago
This stuff is in the HTML, you can see it with web inspector, or I think just by hovering your mouse over the image because that looks like it's in the title attribute for the image. This isn't it like something new was exposed inadvertently. I'm surprised The Verge wrote an article about this.
- sdrothrock 7y ago
  I think it's less about "look at these [creepy] things" and more about "hey guys, you saw all this text when Facebook broke, this is what it is." At least, that's how a lot of my non-tech friends seem to be taking it.
  - mevile 7y ago
    It's creepy on Facebook, but with any cloud photo storage, like Amazon's or Microsoft's, being able to search for "ocean" or "plants" in your own photos is really fucking cool.
    - hombre_fatal 7y ago
      Note: You can do this in Apple's Photos.app and the analysis is done 100% locally on your computer.
- GrumpyNl 7y ago
  This is what i see, when i inspect a element in ff. .d8888b. 888 888 d88P Y88b 888 888 Y88b. 888 888 This is a browser feature intended for "Y888b. 888888 .d88b. 88888b. 888 developers. If someone told you to copy-paste "Y88b. 888 d88""88b 888 "88b 888 something here to enable a Facebook feature "888 888 888 888 888 888 Y8P or "hack" someone's account, it is a Y88b d88P Y88b. Y88..88P 888 d88P scam and will give them access to your "Y8888P" "Y888 "Y88P" 88888P" 888 Facebook account. 888 888 888 See https://www.facebook.com/selfxss https://www.facebook.com/selfxss for more information.
CriticalCathed 7y ago
Dumb article. It's the alt text.
- gdfasfklshg4 7y ago
  This is true. Maybe this was posted to remind us of how unknowledgable the average computer user is regarding the web?
_Wintermute 7y ago
Maybe I'm used to my slow internet connection, but I've been seeing these tags before the images load for a couple of years now - I thought it was common knowledge.
- sdan 7y ago
  Same here!
- 9HZZRfNlpR 7y ago
  Things like that often come as a shock to normal users when they actually see or realize the amount of knowledge these companies have. The big silicon valley companies are literally peeping toms that the society yet tolerates.
  - MiroF 7y ago
    This always surprises me. How can you live today not understanding/predicting the vast scale of corporate/social media analytics...? If major hedge funds are monitoring parking lots with satellite cameras you can bet damn well facebook is using image recognition in its targeting
- jug 7y ago
  Besides - both Apple and Google openly reveals that they auto-tag images by allowing you to search your albums on these sites with the tags as a feature.
  - logo2k 7y ago
    And Microsoft too
- monk_e_boy 7y ago
  No one I talked to was surprised. I've always thought it was pretty neat. As a human I tag people in my photos, I let the machine do the boring stuff (this photo contains trees)
- YeGoblynQueenne 7y ago
furyofantares 7y ago
Lots of comments about how this isn’t new, or how it’s related to accessibility. The article mentions all this. That’s not what the story is. It’s not a very deep story, but I don’t think it’s quite so trivial as is being suggested. The story is about this info being exposed to users just browsing the site, and the reaction that elicits. Someone seeing the tags inline in their feed by a glitch will tend to have a very different response than someone who is asked to view source and look for it. I believe this is relevant because a regular conversation around here centers around the idea that many folks in tech circles find it creepy or unhealthy for society to have so much info being collected and processed for profit by giant corporations, and how most people outside of tech circles seem not to care at all.
- s188 7y ago
  I agree with your point entirely but I wonder if more people outside of tech would care if only they understood what these companies were doing with their personal data.
  - vorticalbox 7y ago
    Yes I believe the number of users on facebook would drop incredibly quickly
    - deleted 7y ago
      [deleted]
- NoodleIncident 7y ago
  That is an incredibly charitable interpretation of the word "reveal" in the headline. Every piece of information can be "revealed" to someone by that definition, it's happening all the time, and it wouldn't be a newsworthy event.
  - deleted 7y ago
    [deleted]
  - 7y ago
codepie 7y ago
There's also an extension which overlay the alt tags over images: https://github.com/ageitgey/show-facebook-computer-vision-tags https://github.com/ageitgey/show-facebook-computer-vision-ta...
gaucheph 7y ago
it's not about what it means for this community, where we all know the shortcut to the dev console in chrome. this is about regular users of Facebook having their own "they live" moments on social media. where they can see plainly how Facebook regards them as consumers. "oh just show him pictures of DRINK and TEXT, that's all he's into anyways".
- lukey_q 7y ago
  "They Live" is a great comparison. Of course this isn't new to someone on HN who follows this technology and the companies using it for a living, but for my uncle who just posts pictures of his dog it's a bit shocking to see that the computer knows that it's (likely) a picture of a dog next to a tree, even if he was aware on some level that computers these days are capable of that kind of classification.
UglycupRawky 7y ago
Just alt-text. Granted it is a slight insight though, as I assume it's added automatically, but I expect they pull a lot more info from people's images than just what the alt-text shows.
gdfasfklshg4 7y ago
And in other news people using screenreaders have been viewing these texts the whole time...
basgrootarch 7y ago
Why do they put this data in the alt-tag in human readable format? For what purpose or use? SEO on a personalized platform with very limited access by Googlebot? Just to show they can? It makes no sense to me.
- K2L8M11N2 7y ago
  For accessibility?
  - basgrootarch 7y ago
    I'd be surprised if they even care about that, but who knows, it's a possibility
    - SmellyGeekBoy 7y ago
      It's a legal requirement in some parts of the world. Oh, and it means a bigger potential userbase for them. Plus it's just generally an all-round considerate thing to do.
    - jackvalentine 7y ago
      Why would you be surprised? They put in effort for their iOS apps to work properly (and the alt-text tags are used for Instagram in "blind mode" for example). If you're a company that wants personal data from everyone, you want personal data from everyone.
    - andreigheorghe 7y ago
      You'd be surprised if the largest content delivery platform in the world cares about being accessible to the broadest range of users? https://code.fb.com/ios/under-the-hood-building-accessibility-tools-for-the-visually-impaired-on-facebook/ https://code.fb.com/ios/under-the-hood-building-accessibilit...
- deleted 7y ago
  [deleted]
- 7y ago
deleted 7y ago
[deleted]
iodbh 7y ago
I think I've never seen worst GDPR compliance : big "I Consent" button with no negative counterpart, except a link to the cookie policy where the "opt out" section invites you to disable cookies in your browser and manually edit local storage. Is that a joke ?
kabwj 7y ago
It’s funny how what purely is an accessibility feature done to help “differently advantaged folks” (how these journalists would say) is being turned against them by the media. Reminds me how they are vilifying open APIs and explaining them as a way Facebook “gives away your data”.
- dvtrn 7y ago
  Reading some of the comments right here, it’s interesting watching some eat it right up and go right along with it. Facebook is doing a lot with user data I have serious problems with. An accessibility feature like his probably ranks dead last or close to it.
buboard 7y ago
"Reveals". God journalists are lazy these days
twodave 7y ago
IMHO by now if you're still on Facebook you've made a conscious decision to let a business with an awful track-record of respecting its users' privacy basically have carte blanche over whatever you give to them. I'm sure there are a few folks out there who don't realize this, but most people I talk to about this say they a) hate it but b) don't want to leave the platform because it lets them keep in touch with people. To them, it's worth it. Not to me.
pepijndevos 7y ago
What I think is more interesting about this outage is that Instagram and WhatsApp were affected. This means your images are stored on the same servers, and possibly subject to the same AI scanning. I wonder if WhatsApp images are E2E encrypted...
mobilio 7y ago
If you're using mobile Facebook like - 0.facebook.com then this is well known for you.
YeGoblynQueenne 7y ago
>> Everyone knows the bit in The Matrix when Neo achieves digital messiah status and suddenly sees “reality” for what it really is: lines of trailing green code. Given that the Matrix was a simulation, what it "really is" can't have been trailing green code. What it "really is" must have been electric charges stored in memory circuits. What Neo saw was his own mind's interpretation of the software and hardware underlying the Matrix. Though of course the point about the green text is that it was a spiffy effect, and not a supposedly accurate representation of what computer memory "looks like".
- tehjoker 7y ago
  I believe the green text is also taken from a recipe book but the designer won't reveal which one.
djsumdog 7y ago
So ... feed images into Facebook to break Captcha?

183 points by saravana85 7y ago

You could also see this if you viewed source when it was fully working, of course. And occasionally before images loaded in if you were on a slow connection. I don't think it's any great secret, but still interesting to see.

ahje 7y ago
I was thinking the same thing. I've noticed this multiple times in the past, but I guess alt-tags are too archaic for Average Joe.
Kronen 7y ago
It happens sometimes in Instagram without any big outage
BinaryIdiot 7y ago
I mean, journalists could also see this if they employed people with disabilities that required them to use such accessibility features. Considering it took an outage to show them I'm guessing few to zero companies do.

SpaceInvader 7y ago

This is nothing new, I've noticed that several months ago and talked about that with one of my security friends and he confirmed that Facebook is tagging pictures at least since 2016.

likpok 7y ago
Here's the announcement, even: https://code.fb.com/ios/under-the-hood-building-accessibility-tools-for-the-visually-impaired-on-facebook/ https://code.fb.com/ios/under-the-hood-building-accessibilit...

silviumatei 7y ago

I know for sure that this technology improves life of visual impaired persons as special software will read respective text.

mburst 7y ago

Looks like alt tags for accessibility purposes. Seems like a win win for all users. Easier to find photos and the ability for visually impaired users to get context on what their friends are sharing

burk96 7y ago

Not only is this nothing new as other commenters have observed, these tags have been used by screen readers and other accessibility services for awhile. While we can talk all we want about Facebook using their all seeing eyes for evil (which I agree they overstep a lot of boundaries), these tags are very useful for people who may require them and should not be scrutinized as an invasion of privacy.

mevile 7y ago

This stuff is in the HTML, you can see it with web inspector, or I think just by hovering your mouse over the image because that looks like it's in the title attribute for the image. This isn't it like something new was exposed inadvertently. I'm surprised The Verge wrote an article about this.

sdrothrock 7y ago
I think it's less about "look at these [creepy] things" and more about "hey guys, you saw all this text when Facebook broke, this is what it is." At least, that's how a lot of my non-tech friends seem to be taking it.
- mevile 7y ago
  It's creepy on Facebook, but with any cloud photo storage, like Amazon's or Microsoft's, being able to search for "ocean" or "plants" in your own photos is really fucking cool.
  - hombre_fatal 7y ago
    Note: You can do this in Apple's Photos.app and the analysis is done 100% locally on your computer.
GrumpyNl 7y ago
This is what i see, when i inspect a element in ff. .d8888b. 888 888 d88P Y88b 888 888 Y88b. 888 888 This is a browser feature intended for "Y888b. 888888 .d88b. 88888b. 888 developers. If someone told you to copy-paste "Y88b. 888 d88""88b 888 "88b 888 something here to enable a Facebook feature "888 888 888 888 888 888 Y8P or "hack" someone's account, it is a Y88b d88P Y88b. Y88..88P 888 d88P scam and will give them access to your "Y8888P" "Y888 "Y88P" 88888P" 888 Facebook account. 888 888 888 See https://www.facebook.com/selfxss https://www.facebook.com/selfxss for more information.

CriticalCathed 7y ago

Dumb article. It's the alt text.

gdfasfklshg4 7y ago
This is true. Maybe this was posted to remind us of how unknowledgable the average computer user is regarding the web?

_Wintermute 7y ago

Maybe I'm used to my slow internet connection, but I've been seeing these tags before the images load for a couple of years now - I thought it was common knowledge.

sdan 7y ago
Same here!
9HZZRfNlpR 7y ago
Things like that often come as a shock to normal users when they actually see or realize the amount of knowledge these companies have. The big silicon valley companies are literally peeping toms that the society yet tolerates.
- MiroF 7y ago
  This always surprises me. How can you live today not understanding/predicting the vast scale of corporate/social media analytics...? If major hedge funds are monitoring parking lots with satellite cameras you can bet damn well facebook is using image recognition in its targeting
jug 7y ago
Besides - both Apple and Google openly reveals that they auto-tag images by allowing you to search your albums on these sites with the tags as a feature.
- logo2k 7y ago
  And Microsoft too
monk_e_boy 7y ago
No one I talked to was surprised. I've always thought it was pretty neat. As a human I tag people in my photos, I let the machine do the boring stuff (this photo contains trees)
YeGoblynQueenne 7y ago

furyofantares 7y ago

Lots of comments about how this isn’t new, or how it’s related to accessibility. The article mentions all this. That’s not what the story is. It’s not a very deep story, but I don’t think it’s quite so trivial as is being suggested. The story is about this info being exposed to users just browsing the site, and the reaction that elicits. Someone seeing the tags inline in their feed by a glitch will tend to have a very different response than someone who is asked to view source and look for it. I believe this is relevant because a regular conversation around here centers around the idea that many folks in tech circles find it creepy or unhealthy for society to have so much info being collected and processed for profit by giant corporations, and how most people outside of tech circles seem not to care at all.

s188 7y ago
I agree with your point entirely but I wonder if more people outside of tech would care if only they understood what these companies were doing with their personal data.
- vorticalbox 7y ago
  Yes I believe the number of users on facebook would drop incredibly quickly
  - deleted 7y ago
    [deleted]
NoodleIncident 7y ago
That is an incredibly charitable interpretation of the word "reveal" in the headline. Every piece of information can be "revealed" to someone by that definition, it's happening all the time, and it wouldn't be a newsworthy event.
- deleted 7y ago
  [deleted]
- 7y ago

codepie 7y ago

There's also an extension which overlay the alt tags over images: https://github.com/ageitgey/show-facebook-computer-vision-tags https://github.com/ageitgey/show-facebook-computer-vision-ta...

gaucheph 7y ago

it's not about what it means for this community, where we all know the shortcut to the dev console in chrome. this is about regular users of Facebook having their own "they live" moments on social media. where they can see plainly how Facebook regards them as consumers. "oh just show him pictures of DRINK and TEXT, that's all he's into anyways".

lukey_q 7y ago
"They Live" is a great comparison. Of course this isn't new to someone on HN who follows this technology and the companies using it for a living, but for my uncle who just posts pictures of his dog it's a bit shocking to see that the computer knows that it's (likely) a picture of a dog next to a tree, even if he was aware on some level that computers these days are capable of that kind of classification.

UglycupRawky 7y ago

Just alt-text. Granted it is a slight insight though, as I assume it's added automatically, but I expect they pull a lot more info from people's images than just what the alt-text shows.

gdfasfklshg4 7y ago

And in other news people using screenreaders have been viewing these texts the whole time...

basgrootarch 7y ago

Why do they put this data in the alt-tag in human readable format? For what purpose or use? SEO on a personalized platform with very limited access by Googlebot? Just to show they can? It makes no sense to me.

K2L8M11N2 7y ago
For accessibility?
- basgrootarch 7y ago
  I'd be surprised if they even care about that, but who knows, it's a possibility
  - SmellyGeekBoy 7y ago
    It's a legal requirement in some parts of the world. Oh, and it means a bigger potential userbase for them. Plus it's just generally an all-round considerate thing to do.
  - jackvalentine 7y ago
    Why would you be surprised? They put in effort for their iOS apps to work properly (and the alt-text tags are used for Instagram in "blind mode" for example). If you're a company that wants personal data from everyone, you want personal data from everyone.
  - andreigheorghe 7y ago
    You'd be surprised if the largest content delivery platform in the world cares about being accessible to the broadest range of users? https://code.fb.com/ios/under-the-hood-building-accessibility-tools-for-the-visually-impaired-on-facebook/ https://code.fb.com/ios/under-the-hood-building-accessibilit...
deleted 7y ago
[deleted]
7y ago

deleted 7y ago

[deleted]

iodbh 7y ago

I think I've never seen worst GDPR compliance : big "I Consent" button with no negative counterpart, except a link to the cookie policy where the "opt out" section invites you to disable cookies in your browser and manually edit local storage. Is that a joke ?

kabwj 7y ago

It’s funny how what purely is an accessibility feature done to help “differently advantaged folks” (how these journalists would say) is being turned against them by the media. Reminds me how they are vilifying open APIs and explaining them as a way Facebook “gives away your data”.

dvtrn 7y ago
Reading some of the comments right here, it’s interesting watching some eat it right up and go right along with it. Facebook is doing a lot with user data I have serious problems with. An accessibility feature like his probably ranks dead last or close to it.

buboard 7y ago

"Reveals". God journalists are lazy these days

twodave 7y ago

IMHO by now if you're still on Facebook you've made a conscious decision to let a business with an awful track-record of respecting its users' privacy basically have carte blanche over whatever you give to them. I'm sure there are a few folks out there who don't realize this, but most people I talk to about this say they a) hate it but b) don't want to leave the platform because it lets them keep in touch with people. To them, it's worth it. Not to me.

pepijndevos 7y ago

What I think is more interesting about this outage is that Instagram and WhatsApp were affected. This means your images are stored on the same servers, and possibly subject to the same AI scanning. I wonder if WhatsApp images are E2E encrypted...

mobilio 7y ago

If you're using mobile Facebook like - 0.facebook.com then this is well known for you.

YeGoblynQueenne 7y ago

>> Everyone knows the bit in The Matrix when Neo achieves digital messiah status and suddenly sees “reality” for what it really is: lines of trailing green code. Given that the Matrix was a simulation, what it "really is" can't have been trailing green code. What it "really is" must have been electric charges stored in memory circuits. What Neo saw was his own mind's interpretation of the software and hardware underlying the Matrix. Though of course the point about the green text is that it was a spiffy effect, and not a supposedly accurate representation of what computer memory "looks like".

tehjoker 7y ago
I believe the green text is also taken from a recipe book but the designer won't reveal which one.

djsumdog 7y ago

So ... feed images into Facebook to break Captcha?