17 ms·
Facebook’s automatic alt-text for images
- chippy 11y agoI'd like to compare Facebook's image tagging with Google Cloud Vision API https://cloud.google.com/vision/ https://cloud.google.com/vision/ I think it would be interesting to see which one is more accurate or verbose.
- odinduty 11y agoAh, the Twitter app for Android (beta version) recently added a feature that allows you to add a description to pictures you upload for impaired people.
- deleted 11y ago[deleted]
- sidcool 11y agoWith all due respect to conspiracies, this is a cool feature.
- TazeTSchnitzel 11y agoI suppose it'll be like YouTube's automatic subtitles for audio. It'll do a bad, but passable, job: at least the blind and visually impaired have some idea of what the image contains.
- hacker_9 11y agoAccurate, automatic descriptions of snapshots of peoples lives?? This is surely sending shock waves down the data mining community. Additionally Facebook are saying this was built as an aid for blind people, but this is surely just a cover for being able to take targeted ads to the next level.
- hanspeter 11y agoIf that was the case wouldn't they choose to simply not announce anything? I'm not ruling out that they might use it for targeting eventually, but if this was solely done as a cover it would be the equivalent of a terrorist entering an airport shouting out "My suitcase is just really heavy, I do not have a bomb in it at all!"
- pdkl95 11y agoThe goal is to normalize people to yet another type of surveillance and data mining.
- masklinn 11y ago> If that was the case wouldn't they choose to simply not announce anything? People would find out within a few days and out would come the pitchforks. Here facebook preempts the conversation and sets the tone, now worriers will be met with "Do you not want blind people to enjoy the web? Why do you hate them so?"
- Etheryte 11y agoDo elaborate how people would find out if Facebook simply used this data internally.
- masklinn 11y agoMight take a big longer, but I expect people running and testing ads would discover correlations with image posts.
- zeven7 11y agoFacebook was already using this data internally. You just didn't know they could do it until now.
- zappo2938 11y agoCould be worse. Google has me training AI to do the same thing every time I choose all the pictures with train locomotives in a reCaptcha.
- mkorfmann 11y agoFor a captcha to function properly, they already need to have this data I guess.
- maxerickson 11y agoreCAPTCHA is famous for having 2 tasks. This lets them use 1 for training. https://en.wikipedia.org/wiki/ReCAPTCHA#Criticism https://en.wikipedia.org/wiki/ReCAPTCHA#Criticism
- eterm 11y agoNot only this, but given how much google tracks you, google has a good enough idea about whether you are a bot already, hence the "captcha" where you just press a button. There are times they could give you that button but already knowing you are human they use you to train their AI. I wouldn't be surprised if they also score you against how well you train AI, and give better trainers more captchas. Google has so much data, any time you think "I wish we could do X, but we'd never get enough data", it's an opportunity that Google can move on but the rest of the world can't. That's another argument for breaking up monopolies.
- estel 11y ago> it's an opportunity that Google can move on but the rest of the world can't. That's another argument for breaking up monopolies. Wouldn't this mean that it would be an opportunity that /nobody/ could move on?
- taneq 11y agoI always wondered how many captchas were passthroughs to other captcha systems, using me as a mechanical turk component.
- leoalves 11y agoGoogle, Facebook, twitter .... Are all doing this in their backend. If you are worried about it. Don't use their services. The problem I see here is that now they are giving this data to spammers.
- dingaling 11y ago> If you are worried about it. Don't use their services. I don't use their services but apparently I'm still tagged in Facebook photos. I never accepted ToS permitting that.
- Karunamon 11y agoIf there's a picture of you floating around in public it's not illegal or immoral for them to say who's in it.
- Majestic121 11y agoIt is immoral for them to collect enough infos about me to be able to tell who I am from a picture, and then use it, while I do not use their service.
- Karunamon 11y agoWhy's that? Assuming all the info is public, they've done what any person could reasonably do. Put another way, with absolutely no snark or malice intended: Information about you is not necessarily owned by you.
- cortesoft 11y agoWhile I kind of agree with you, I don't think it is fair to automatically assume that just because an automatic computer system is doing the same thing that a human could do, it can't fundamentally change what that thing means. Things change when something can be done automatically and at great scale. Take, for instance, license plate scanning. Of course, anyone can see your license plate when you are driving around town, it is public information. So clearly it is fine if machines scan license plates and keep records of every license plate that crosses a bridge. A team of people could sit on the bridge and do the same thing, so no big deal. So clearly it is fine if we install the license plate scanners on every street corner in the city, and scan every plate they see. And it should be totally fine to put that data on a publicly searchable real-time database, so that I can input any license plate number and see immediately where that car is. These different scenarios might seem like merely matters of degree on the same bit of info, but what it means for us changes immensely as it becomes easier and faster to do it.
- ma2rten 11y agoThis is surely sending shock waves down the data mining community. Not really. From a technical point of view there is nothing impressive about this, except that they spend the time and money to collect all the training data. Also that it says "this image may contain:" tells you something about how accurate this actually is.
- hacker_9 11y agoNot my point. What isn't available to everyone is the sheer amount of real life images that Facebook has access to, which people are uploading of their lives as we speak. You could get an unprecedented view of almost anyone's life from that, seeing as users have past the 1 billion mark, and it's all in the hands of a company who sells your data for profit.
- manigandham 11y agoYour basic demographics and "likes" are already enough to do all the targeting necessary. If this is about the general privacy issue then it's back to the same old thing: don't share things that you want to keep private. Of course people's actions prove that they get more value out of sharing this content then is worth the privacy of not doing so.
- hacker_9 11y agoAgain, not my point. Social network sites are built off image sharing, that before computers did not understand. Now that they do, we are entered an unprecedented age of surveillance. Private is only private to humans, the Facebook machine can look at any photos it wants, and build up a profile for anyone, their friends, make predictions of behavior and so on. It's not the tech that is bad, it is the fact it is in the hands of for-profit companies, who are willing to sell your data just to make their bottom line.
- mynameisvlad 11y agoThen what is your point? manigandham literally adressed your concern; basic demographics, likes, and other sources for content are far better for targeting than what a picture could potentially contain, and that can be done with closer to 100% accuracy since a computer doesn't have to "guess" what the content of an image is, it already knows this content from the text in a status, or the location metadata associated with a post, or the people you're tagged with. This is incredibly inaccurate when compared with all those sources.
- DonHopkins 11y agoWell if 98% of your images are pictures of cats, you don't really have to solve the hard general problem.
- placeybordeaux 11y agoYou do if you want high accuracy on stuff other than cats.
- iaw 11y agoThis has been possible for, at a minimum, three years. There's an effort gap between building an ad targeting version and building a blind enabling version. The only new thing here is that Facebook released it to the public.
- coroutines 11y agoI kind of want to take this at face-value and say how great it is that they're making visually-impaired viewers a priority.
- d33 11y agoObligatory: https://www.youtube.com/watch?v=_wXHR-lad-Q https://www.youtube.com/watch?v=_wXHR-lad-Q It's impressive that people don't really connect the dots and see that as a huge threat to their freedom.
- ospfer 11y agoLast July, I lead a project in support of a federal agency to analyze current business processes and identify weaknesses in the agency Section 508 office. My work focused primarily on externally accessible internet sites and one of the most common 508 violations that we encountered was the lack of ALT-text on images. This agency utilized a number of automated scanning tools and processes, but lacked any ability to efficiently remediate these errors. While we never talked more than from a conceptual standpoint, a coworker and I discussed something along the lines of what Facebook has accomplished here through the use of Google's Neural Networks. Very cool to see this advancement come to life.
- dr_zoidberg 11y agoThis kind of systems/algorithims also allows to asign a certain semantic component to images (with a grain of salt, of course), which might enable further developments that weren't considered posible yet. Sadly, it also brings another complete set of cases to the oh-so-anoying "but Facebook/Google/Twitter/Amazon does it!" clichés that we'll now have to deal with...
- bla2 11y agoWarning, that page has an auto-play video with sound.
- verusfossa 11y agoI'm waiting for the day this is just a library you pass an image to and it returns an array. No, not a SaaS. Then on my own pump.io, diaspora, redMatrix etc. it just works. My data, my images, my network. I'm not against the tech at all though. Neat
- ma2rten 11y agoThere are already pre-trained networks out there. TensorFlow comes with an example command line tool that you can pass any image and it will tell you what is in the image. The classes that it can detect are from ImageNet, so that might be limiting.
- dr_zoidberg 11y agoIf you were to train your own net, ImageNet is one of the biggest and most complete datasets and you'd surely use it for training. The alternative is to make your own training set, which will cost you money and/or time. For a proof of concept or initial prototype (until your business can pay for it), those classes should be enough.
- ma2rten 11y agoImageNet is not going to have something like "smile" in it's dataset like they showed in the video. It has all kinds of possible dog breads instead. Maybe someone should create a website that lets volunteers label images for this purpose.
- raus22 11y agoSee visual genome for the next level beyond imagenet. http://visualgenome.org/ http://visualgenome.org/
- dr_zoidberg 11y agoYeah, in "Person, individual, someone, somebody, mortal, soul" there's the deeper category "smiler" which contains the categories "smirker" and "simperer". Some of the images are (note: my idea was to link to the images directly, but it isn't loading, so I had to take a screenshot and upload to imgur): * http://i.imgur.com/Wex6pSR.png http://i.imgur.com/Wex6pSR.png Which I say is pretty much what you'd need to train a net to detect people smiling (amongst other things). Of course, there are some refinements you can make to improve accuracy and presentation of the results. My point was: you should begin with datasets that are readily available, and then improve on need (and if resources are available to justify the investment).
- cphoover 11y agoIs this tool open source? would be a great contribution to the accessibility community.
- tlrobinson 11y agoVery cool. Obvious next step: build this into the OS/browser/screen reader.
- Spearchucker 11y agoI do wonder when somebody uses this technology to troll.
- Crespyl 11y agoThe techniques already exist[0] for some sophisticated trolling, though it may be hard to achieve in practice without direct access to the classifier being used. [0] http://karpathy.github.io/2015/03/30/breaking-convnets/ http://karpathy.github.io/2015/03/30/breaking-convnets/
- SimeVidas 11y agoGlad to see Mark explaining what a screen readers is to millions of people :-D
- buro9 11y agoThis is really what I wanted to use the Google Image API to do. But it's way too expensive. All I wanted was keywords for alt-text, dimensions for placeholder, and the dominant colour for placeholder background. https://cloud.google.com/vision/ https://cloud.google.com/vision/ The price for that would be $7.50 per 1,000 images for the first million images. I have some 60,000 images on the site I run and don't happen to have $450 in loose change laying around (the whole site costs less than that to run each month). I guess I don't care about alt tags that much.
- speedyapoc 11y agoI always find Facebook's example feed to be funny since it's a completely unrealistic depiction of what their site actually is for most users. Just quickly looking at the top few posts in my feed, I see someone celebrating their two year friendship with someone I don't know, one person sharing a link to a new airplane, four people sharing videos, one person liking a sponsored video, and finally one person updating their profile picture. I wish I could see actual updates from people instead of being kept abreast as to what piece of third party content they've liked at some point in time, what third party content they're sharing, etc.
- Toenex 11y ago> I wish I could see actual updates from people instead of being kept abreast as to what piece of third party content they've liked at some point in time, what third party content they're sharing, etc. This nicely captures the problem. Facebook (and probably most social media systems - I'm looking at you LinkedIn) are primarily interested in keeping you up to date with Facebook via the medium that is your human relationships. So Facebook wants you to know what your friends did on Facebook today so that you might do that same Facebook thing. This increases Facebook interactions which in turn become more information to propagate to others on Facebook. In the limit there is no need for Facebook because all anyone is ever doing is Facebook.
- manigandham 11y agoHow is this unrealistic? This is exactly the kind of stuff my feed shows. The people you follow probably aren't posting any updates (that you like to see)... and sharing content is an update, that's what that person decided to post. Your feed is what you make it.
- NeutronBoy 11y agoEverytime I see someone complain that their feed is full of memes, junk, and chain posts, I think 'Maybe you should just remove that person from your Facebook?'.
- supernintendo 11y ago
- skrjon 11y agoThere is more information on what Facebook is doing on the research site. https://research.facebook.com/blog/how-blind-people-interact-with-visual-content-on-social-networking-sites/ https://research.facebook.com/blog/how-blind-people-interact... Including a link to the publication that was written on the technology here. https://research.facebook.com/publications/how-blind-people-interact-with-visual-content-on-social-networking-services/ https://research.facebook.com/publications/how-blind-people-... I think its exciting and an honest attempt to make peoples lives better.
- whatever_dude 11y ago"Cat. Cats. Cat. Baby. Dog. Baby. Cat. Baby with dog."
- visarga 11y agoBag. Duck face. Nails. Duck face.
- shogun21 11y agoThis might be asking for too much, but why not use more of the image meta-data than these computer vision techniques? If I were blind, I really wouldn't care that this is an image of "two people, smiling". Facebook has facial recognition, tagging, and locations. It would be much more valuable to me to say "Peter and Laura smiling at Channel Islands State Park."
- visarga 11y agoThey are just tagging, but there are ML solutions for describing in natural language. Demo: http://googleresearch.blogspot.ro/2014/11/a-picture-is-worth-thousand-coherent.html http://googleresearch.blogspot.ro/2014/11/a-picture-is-worth...
- nickysielicki 11y agothat's honestly very creepy