15 ms·
Neural Networks That Describe Images
- jcr 12y agoAt the Bay Area Vision Meeting in 2013 [1], Fei-Fei Li and Olga Russakovsky gave a related talk on, "Analysis of Large-Scale Visual Recognition" [2,3] [1] http://bavm2013.splashthat.com/ http://bavm2013.splashthat.com/ [2] video: http://www.youtube.com/watch?v=DK6KfUsVN8w http://www.youtube.com/watch?v=DK6KfUsVN8w [3] slides: http://bavm2013.splashthat.com/img/events/46439/assets/a10b.stanford.pdf http://bavm2013.splashthat.com/img/events/46439/assets/a10b....
- thomasahle 12y agoIt would be great to have this for image search!
- amckenna 12y agoYup! That's exactly why there has been so much work on this from the likes of Google. Video and images represent a "dark media" in terms of indexing and search. It is very difficult without robust tagging (which is time intensive or relies on the articles and descriptions that surround the images) to search through images. This technology would make it much easier perform robust querying of this data.
- visarga 12y agoIt will revolutionize porn search, for sure.
- dmritard96 12y agounfortunately came to the same conclusion. could be a serious buzz kill though when it is wrong.
- notastartup 12y agothis is unsettling and amazing. not too far when we'll have robots that will be aware of what's going on around it.
- deleted 12y ago[deleted]
- arjie 12y agoFascinating. Also interesting to see the failure modes. Any human would quickly realize that the "boy doing backflip on wakeboard" is actually playing on a trampoline. Or the "two young girls playing with legos toy". Great stuff!
- DavidSJ 12y agoI think I'm a human, but I did not realize that boy was on a trampoline instead of a wakeboard until you pointed it out. I think the water in the background confused me. I was also probably biased by reading the network's label before reaching my own judgment.
- jameshart 12y agoRight - impressive though this is, also the cat isn't black, the woman with the bananas might well not be a woman, the girl isn't swinging on a swing, the construction worker is probably not in the road, and the guitarist is tuning the guitar, not playing it. It's pretty much wrong -in detail- on every count. The weird thing is how at a glance, it seems pretty much correct. And how some people here are willing to look at that and think 'automated law enforcement is clearly imminently possible'.
- negamax 12y agoWow.. talk about high expectations. This is really well done. Identifying multiple concepts in an image and describe it to a good degree of accuracy. And this will get better further.
- deleted 12y ago[deleted]
- yuncun 12y agoIdk if this is a daft question, but in the Visual-Semantic Alignment section, are those objects in the colored boxes actually being directly recognized by the software? Or are they inputted in some other way?
- tjr 12y agoFrom the paper: Our core insight is that we can leverage these large image-sentence datasets by treating the sentences as weak labels, in which contiguous segments of words correspond to some particular, but unknown location in the image. Our approach is to infer these alignments and use them to learn a generative model of descriptions.
- karpathy 12y agoThis gets a little more detailed into the work, but compared to other papers that have sprung up in this area recently, our paper slightly frowns on the idea of distilling a complex image into a single short sentence description. In that sense we are a little more ambitious and we're trying to produce snippets of text that cover the full image with descriptions on level of image regions. I would call our results encouraging, but there is certainly more work to be done here. And I think one of the limitations right now to do a good job is the amount of training data available to us.
- notastartup 12y agoso basically you have a database of pre existing sentences to match whats roughly likely to be on the page, not actually seeing individual objects and generating a grammatically correct and accurate description based on the individual objects?
- xanderjanz 12y agoSharing Pre built models are so cool, and definitely important to the advance machine learning science. Especially when you consider how mixing weight layers allows you to do things like understand Portuguese text better through English text.
- tintor 12y agoIt is interesting how the neural network labeled the woman in the lower right photo with the rectangle that includes the body only, without the head.
- tiler 12y agoI really appreciate that the Stanford group is always willing to post mistakes/mislabels. Kudos.
- sigterm 12y agoThe face detection on her research group's front page has one glaring mislabel (http://vision.stanford.edu/ http://vision.stanford.edu/). I always find that quite amusing.
- kolbe 12y agoPresentations like these make me realize how close we are to developing law enforcement (/police state) technology that will be very effective. I figure when the kinks are smoothed out, that we could run this on a video feed, and have crimes prevented right as they're about to happen. It's almost scary. Imagine 20 years from now, some guy pulls a gun on you, and a video feeds identifies his action, and immediately shoots a tranquilizer straight into his jugular with perfect aim.
- bprater 12y agoEffective and also scary. That same video feed could be archiving (for eternity) every single citizen's movement, action, or even words: "Citizen #5135: In 2015, spit on sidewalks 28x this year, jaywalked 49x, etc..."
- visarga 12y agoEven if this is not working now, it can still be applied later on video shot today. So we are already en route to being judged by such an automated guardian.
- smokeyj 12y agohttp://en.wikipedia.org/wiki/Recorded_Future http://en.wikipedia.org/wiki/Recorded_Future Don't worry. Google and the CIA have your back. /duck and covers from incoming thought police
- e12e 12y ago> some guy pulls a gun on you, and a video feeds identifies his action, and immediately shoots a tranquilizer straight into his jugular with perfect aim. Eh, no. Safe preventive action against a gun (or other lethal weapon, really) needs to happen before the weapon is ready for use. What if the assumed offender was wearing protection? Perfect aim wouldn't help. Lethal force might be possible to deploy - but probably not in a safe manner.
- ibrahima 12y agoThat's basically the plot of Person of Interest, right? (I've watched a few episodes but never paid much attention).
- zvanness 12y agoThis is one of the coolest things i've seen in a while. I'd guess this is super similar to what the folks on the DeepMind team at Google are working on now, with the overall vision of being to classify images that have no metadata and add them to a dynamically learning knowledge graph: http://www.newscientist.com/article/dn24946-google-buys-ai-firm-deepmind-to-boost-image-search.html http://www.newscientist.com/article/dn24946-google-buys-ai-f... http://en.wikipedia.org/wiki/Knowledge_Graph http://en.wikipedia.org/wiki/Knowledge_Graph http://appft.uspto.gov/netacgi/nph-Parser?Sect1=PTO2&Sect2=HITOFF&p=1&u=/netahtml/PTO/search-bool.html&r=1&f=G&l=50&co1=AND&d=PG01&s1=20140019484.PGNR.&OS=DN/20140019484&RS=DN/20140019484 http://appft.uspto.gov/netacgi/nph-Parser?Sect1=PTO2&Sect2=H... http://appft.uspto.gov/netacgi/nph-Parser?Sect1=PTO2&Sect2=HITOFF&p=1&u=/netahtml/PTO/search-bool.html&r=1&f=G&l=50&co1=AND&d=PG01&s1=20140019431&OS=20140019431&RS=20140019431 http://appft.uspto.gov/netacgi/nph-Parser?Sect1=PTO2&Sect2=H...
- hugozap 12y agoThey forgot to put the link to the npm module ;)
- ed 12y agoThis isn't really a breakthrough in object identification, as much as it is a clever pairing of identification with (mostly existing) language systems, is that right? Wondering whether there's any merit to sibling comments speculating this is the future of e.g. surveillance
- xanderjanz 12y agowell, this is doing more than just object recognition+language labeling. Notice how it understands the the girl is 'in' the white dress. That is a lot more comlicated that identifying a white dress and a girl.
- 31reasons 12y agoSomething doesn't seem right. If we are this good in image recognition, how come we are still using captchas? Are we so good in image recognition that it can identify young girl, bunch of bananas, guitar etc just by training it from an image set of just few thousand image. Whats the catch ? On one hand google says their self-driving car can't understand all the situations on the road while this algorithm can identify so many details from an image like a strong AI would. Feels very strange.
- cLeEOGPw 12y agoThese images are probably cherry picked as the most successful recognitions. We need more data until we can say whether it can actually recognize young girls in images confidently, or just sometimes gets "lucky". That's why they say that not all traffic situations can be understood. The algorithms aren't perfect.
- 31reasons 12y agoThey are using neural networks so they are never going to confidently say in what situations it will stop working.
- vidarh 12y agoCaptcha's largely fight the lowest common denominator, my making those who don't care enough (or have the knowledge) to work around them when they can just go elsewhere, so that you can invest your human resources fighting the more sophisticated attackers that actually target you. They are reasonably successful because spammers have enough other targets that not many see it as worth the extra effort (and clock cycles) to break them, not because most of them are particularly hard to beat any more.
- indymike 12y agoBreaking captchas also have an incentive that has been attractive enough to create an three entire industries -- spam, malware and security products to deal with spam and malware.
- 12y ago
- nulldata 12y agoFascinating that we've come this far, also: http://xkcd.com/1425/ http://xkcd.com/1425/
- Zeebrommer 12y agoI'm a bit skeptic. They have several very good examples, and those are really impressive. Everything about how often these striking results occur is 'coming soon' though. And since it is neural network-based, being lucky sometimes doesn't say anything about the statistical performance. What percentage of the test dataset was labelled correctly?
- gaurav_v 12y agoModel performance is discussed in their full paper, here: http://cs.stanford.edu/people/karpathy/deepimagesent/devisagen.pdf http://cs.stanford.edu/people/karpathy/deepimagesent/devisag...
- userbinator 12y agoI'd like to see the worst-case behaviour too - e.g. hilariously wrong results. Seeing the failures (and their rate) makes it more believable and enables a more unbiased evaluation of the capabilities of the system. It's like those "up to XX% better" claims - "up to", not "at least" being the key phrase here.
- yihyeh 12y agoBefore going through on object recognition in images, I'm curious about how they managed to arrange perfect sentence for the caption. Are we already at the point where NN can arrange perfect sentence when we throw bunch of words into it?
- ogrisel 12y agoYes, there is a recent trend to use Recurrent Neural Networks to model the structure and semantics of sentences. This used in particular to do research for Machine Translation by people at Google and the University of Montreal in particular: http://scholar.google.fr/scholar?q=rnn+lstm+machine+translation http://scholar.google.fr/scholar?q=rnn+lstm+machine+translat...
- userbinator 12y agoThey're not perfect but similar to what NN-based machine translators output; it's not completely consistent. E.g. "girl in pink dress is jumping in air." vs "black and white dog jumps over bar."
- fmax30 12y agoI remember doing something like this without the neural networks that is, but my results were very very bad. If i find that project in old laptop i'll post it to github.
- tawan 12y agoI'm impressed but skeptic, too. Laypeople might not get an accurate impression on what is possible. Considering the image with the guy in a black shirt playing guitar. What if it was a picture of a naked man standing next to a black shirt that is hanging on a clothes hook, and next to it is a guitar is mounted on a rack: My guess it that the computer will still spit out: Guy in black shirt playing guitar, just because this sentence is very plausible in the underlying language model.
- shultays 12y agoI am pretty sure some of those examples are correct only by a chance. "little girl is eating piece of cake."? Only thing that marks her as a girl is hair clip, did the software really saw that? "woman is holding bunch of bananas." hell, I (hopefully a human) would recognize her as a male at first glance.
- aurelius 12y agoNeural networks are impressive only in that they are able to give any kind of meaningful results at all. In the end, they are only a poor mimicry of real machine intelligence, and not much better, conceptually, than plain old nonlinear regression. Nobody has been able to determine what the structure of a neural network should look like for any given problem (network type, number of nodes, layers, activation functions), how many iterations of the parameter optimization algorithm are needed to achieve "optimal" results, and how "learning" is actually stored in the network. Statistical learning methods are obviously still useful, but I think the field is still wide open for something to emerge that is closer to true machine intelligence.
- bcbrown 12y agoWhat do you define 'machine intelligence' as?
- aurelius 12y agoJarvis from Iron Man, or KITT from Knight Rider.
- xanderjanz 12y agoPlease, try implementing non-linear regression to understand images. Tell me how it goes. Also, 'nonbody know hows learning is stored'? You very clearly have never worked with neural nets before. Experience is stored in the form of weight values.
- jameshart 12y agoOkay, you've got a neural net that does a really good job on identifying types of animals in pictures. unfortunately, whenever you show it a picture of a horse, it says 'fish'. Everything else, it's great at - marmosets, capybaras, dolphins, kangaroos; but it's got a complete blindspot for horses. Where's the incorrect data stored? How can you fix it? It's in the weight values, somewhere, but you can't go and change the weight values to fix the horse/fish cascade without breaking everything else it knows. Yes, we know 'where' the data is stored. But it's diffuse, not discrete, so we can't separate it from other data.