12 ms·

How convolutional neural networks see the world

214 points by makmanalp 11y ago

deleted 11y ago
[deleted]
ThomPete 11y ago
A little of–topic. There is something interesting to be said about what kind of awareness a future AI (or whatever we should call it) will simulate the world. Imagine what kind of perspectives are possible when thousands or millions of input sources are your senses.
- sigmar 11y ago
  >Imagine what kind of perspectives are possible when thousands or millions of input sources are your senses. Doesn't this exactly describe human sensory input? Though our brain is efficient by throwing out most of the data early on in the signal chain (as research has revealed in vision and auditory input). Will future AI also need to be as efficient?
  - seiji 11y ago
    Doesn't this exactly describe human sensory input? Well, consider having a 360º array of 30 cameras all integrated into a perfect spherical sensory experience. It's something we can't really imagine experiencing natively, but it would be trivial for eBrains to coalesce visual systems that way from eBirth. Our bodies have lots of low bitrate sensors like billions of individual sensory nerves distributed throughout our bodes (and they are each individually addressable in the brain), but we don't think of "touch" as a sense to "computationalize" like vision or sound or language. One amusing thing about AI sensors: nobody ever talks about superhuman smell. Where are the quantum AI noses?
    - tamana 11y ago
      Biochemical sensors, like what that discredited blood test company theranos tried to do, is a sort of superhuman smell.
  - ThomPete 11y ago
    But at a completely different scale though. You have to imagine a whole planets sensors as your sources combined with the whole planets knowledge and so on. I don't think it would make much sense to compare with the limited POV we are experiencing the world from.
  - CuriouslyC 11y ago
    Deep neural nets do precisely this. Though the specifics change from implementation to implementation, in general there is a large drop in the number of hidden units at each layer of the architecture.
- clickok 11y ago
  A little pedantic: you already have thousands of different input sources, from the classic "five senses" to less conscious awareness of things like your limb placement and internal biochemistry (e.g., hunger). There are even people who've "added" artificial senses by doing things like placing magnets under their skin (to sense magnetic fields) or strapping a device to their leg that buzzes in the direction of North. Not to mention the fact that we can use things like visually inspecting a screen to access senses (via information feeds) that are not available biologically. However, you're right that a robot/AI developed with the intention of feeding it all sorts of heterogeneous data will probably be able to process everything more effectively. In my own research (AI with a focus on reinforcement learning and robotics) I am sometimes surprised by how effective agents can be at making sense of their input streams. For example, an experiment will not go the way you expect because the robot can trivially solve a maze via sensing the current in the wiring beneath the floor. Of course, there's a limit in terms of how effective raw information can be. Humans don't need to see ultraviolet wavelengths because in general the spectrum of ~350-700nm provides all the information we need, and the brain is good at finding the salient aspects of what we see. If you just connect a new sensor to a robot, it might improve its ability to understand the world, or do nothing at all, because it can't incorporate this new information into its representation effectively. Or it doesn't add anything new, or at least nothing that it couldn't have figured out from existing input streams. For example, adding a stock ticker feed to your robot would probably not help it solve a particular task, unless your robot happens to be 50 feet tall and the task in question is "rampaging down Wall Street".
  - ThomPete 11y ago
    Hmm sure but we are talking about a completely different scale here. I mean as an AI you are connected to the whole planets sensors, you have the whole worlds knowledge in your possession and will be able to cross reference with what you are getting as input. You can prototype, do scenario planning on the fly, you can calculate and so on. Furthermore you are potentially getting inputs from other humans too and have the ability to mostly likely control a number of things which again provide new input.
akavel 11y ago
SPOILER WARNING! Nice read at the end of the article: "[...] Naturally, this does not qualify as "seeing" in any human sense, and from a scientific perspective it certainly doesn't mean that we somehow solved computer vision at this point. Don't believe the hype; we are merely standing on the first step of a very tall ladder. Some say that the hierarchical-modular decomposition of visual space learned by a convnet is analogous to what the human visual cortex does. It may or may not be true, but there is no strong evidence to believe so. Of course, one would expect the visual cortex to learn something similar, to the extent that this constitutes a "natural" decomposition of our visual world (in much the same way that the Fourier decomposition would be a "natural" decomposition of a periodic audio signal). But the exact nature of the filters and hierarchy, and the process through which they are learned, has most likely little in common with our puny convnets. The visual cortex is not convolutional to begin with, and while it is structured in layers, the layers are themselves structured into cortical columns whose exact purpose is still not well understood --a feature not found in our artificial networks (although Geoff Hinton is working on it). Besides, there is so much more to visual perception than the classification of static pictures --human perception is fundamentally sequential and active, not static and passive, and is tightly intricated with motor control (e.g. eye saccades). Think about this next time your hear some VC or big-name CEO appear in the news to warn you against the existential threat posed by our recent advances in deep learning. Today we have better tools to map complex information spaces than we ever did before, which is awesome, but at the end of the day they are tools, not creatures, and none of what they do could reasonably qualify as "thinking". Drawing a smiley face on a rock doesn't make it "happy", even if your primate neocortex tells you so. That said, visualizing what convnets learn is quite fascinating --who would have guessed that simple gradient descent with a reasonable loss function over a sufficiently large dataset would be enough to learn this beautiful hierarchical-modular network of patterns that manages to explain a complex visual space surprisingly well. Deep learning may not be intelligence is any real sense, but it's still working considerably better than anybody could have anticipated just a few years ago. Now, if only we understood why... ;-)"
- seiji 11y ago
  Drawing a smiley face on a rock doesn't make it "happy", even if your primate neocortex tells you so. That's one of the best analogies we've got with regard to "deep learning" versus reality. People around here seem to think the AI apocalypse is 3-5 years away and are rushing to fund billion dollar "sentient rock" research.
cjenken 11y ago
What I find interesting is that the auras that I get as a precursor to my migraines don't look far off from those images. If you were to take a C shaped slice and overlay it onto video and then animated the color you would pretty much have it. It would be interesting to see if they were connected somehow...
- Johnythree 11y ago
  Yes, same here! I get the vivid flickering lights occasionally (in a C shaped slice as you describe), but luckily don't get the headaches.
- mej10 11y ago
  Sounds like you are describing "scintillating scotoma", if you wanted a name for that visual distortion. https://en.wikipedia.org/wiki/Scintillating_scotoma https://en.wikipedia.org/wiki/Scintillating_scotoma
- Synaesthesia 11y ago
  That's interesting, it also resembles the patterns I see on LSD or 2C-B
fizixer 11y ago
Question to deep learning practitioners: Is there any research on comparing the 99.9% confidence about something wrong, to the human weakness of 'optical illusion'? If you watch a few minutes of this Dan Ariely TED Talk [1], we humans are fooled into near certainty that the table on the left has longer vertical distance that the horizontal distance of that on the right. Even after the speaker shows this is not the case, we still "confidently see" the wrong thing. (not to mention, we suffer from non-optical illusions as well, like cognitive biases, and fallacious reasoning, as shown later in the talk). It seems to me you guys have discovered 'machine optical illusion'. It's just that the machine has illusions about something completely different. I believe exploring the space of architectures/weights/what-not might lead towards making a machine get into the same kind of optical illusions as the human visual system gets into. At the least, it looks like a very promising research direction to me. [1] https://youtu.be/9X68dm92HVI?t=2m22s https://youtu.be/9X68dm92HVI?t=2m22s
- rasz_pl 11y ago
  length illusion happens higher up the meat stack. You see the picture for what it is, but visual cortex gets overridden. Happens all the time, blue dress, motion blur, saccades. A ton of stuff seen by the eye is filtered out or overruled by upper layers.
- brianchu 11y ago
  That is not an example of how weak our brain is. That is an example of how incredibly intelligent our brain is. If you saw both tables in the real, physical, natural world, because of the effect of perspective, the table on the left is indeed physically longer than the width of the table on the right. Our brain is correctly inferring this from perspective/trigonometry. The visual system is designed to correctly interpret the natural world, not correctly interpret lengths of line segments on a slide. This illustrates how hard it is to create computer algorithms that robustly infer 3D from 2D (this inference requires taking cues from shading, perspective, etc.) Almost all optical illusions are examples of the strength, not weakness, of the human brain. EDIT: I've emphasized the word "natural". The problem with the below examples of "illusions" constructed in the physical world is this: what do you expect the brain to do otherwise? When you see a scene in the world, is it a weakness that our brain discards the (extremely unlikely) possibility the scene is just a massive hologram five inches from your eyes, or a meticulously painted and lit cardboard scene? I don't.
yeukhon 11y ago
Just really curious? What is up with ML on HN today? So many posts about ML in a single day. After reading this article I have to say even humans have hard time actually understand images and patterns like the one shown in the article, let alone a machine. I wonder what would a machine say about the famous "is this dress blue or grey" photo last year.
mdonahoe 11y ago
What would happen if you generated a lot of images like this, tagged them as "conv-net filter", and then added them to the set of images that the neural nets trained on? Would the network learn more discriminating filters for everything else?
- dhj 11y ago
  Very interesting question and worth an experiment or two. Personally, I don't think it would help because the patterns are complex and it is an internal representation. It would be like feeding it a hash values and labelling them "hash values". That wouldn't necessarily help you discriminate pre-hashed values. That said, I would definitely be interested in the results of a real test.
tariqali34 11y ago
Machines do not have to think or see in the same way as humans. To dismiss machine capablities simply because they do not resemble human intelligence seems hopelessly naive.
rasz_pl 11y ago
Its striking how similar CNNs are to good old JPEG compression. Makes me think you would get better results by chopping up input into blocks, and slapping few more layers on top. From the article description I get that VGG16 tries to guess whole picture content from existence of particular patterns totally ignoring their location and arrangement, hence nonsensical magpie prediction. Photoshop magpie with a beak sticking out its ass and it will pass as genuine because all prerequisite patterns are there.
arketyp 11y ago
If you took every image agreeing with your conception of a magpie and composed it into one image, would it look anything like a magpie? The Google inceptionism team did the same experiment but supplied the networks with natural image priors. Perhaps hand-picked, but those examples were actually quite convicining.
Synaesthesia 11y ago
we are merely standing on the first step of a very tall ladder. ... Deep learning may not be intelligence is any real sense, but it's still working considerably better than anybody could have anticipated just a few years ago. Now, if only we understood why... ;-) Yes I'd agree with this, there needs to be a deeper level analysis for how the cognitive functions within the brain, for example how vision in the brain works. Such understanding will clearly take a great amount of effort to achieve, and I don't know if research enough research is aimed in this deeper theoretical direction, as opposed to neural net type machine learning.

214 points by makmanalp 11y ago

deleted 11y ago

[deleted]

A little of–topic. There is something interesting to be said about what kind of awareness a future AI (or whatever we should call it) will simulate the world. Imagine what kind of perspectives are possible when thousands or millions of input sources are your senses.

sigmar 11y ago
>Imagine what kind of perspectives are possible when thousands or millions of input sources are your senses. Doesn't this exactly describe human sensory input? Though our brain is efficient by throwing out most of the data early on in the signal chain (as research has revealed in vision and auditory input). Will future AI also need to be as efficient?
- seiji 11y ago
  Doesn't this exactly describe human sensory input? Well, consider having a 360º array of 30 cameras all integrated into a perfect spherical sensory experience. It's something we can't really imagine experiencing natively, but it would be trivial for eBrains to coalesce visual systems that way from eBirth. Our bodies have lots of low bitrate sensors like billions of individual sensory nerves distributed throughout our bodes (and they are each individually addressable in the brain), but we don't think of "touch" as a sense to "computationalize" like vision or sound or language. One amusing thing about AI sensors: nobody ever talks about superhuman smell. Where are the quantum AI noses?
  - tamana 11y ago
    Biochemical sensors, like what that discredited blood test company theranos tried to do, is a sort of superhuman smell.
- ThomPete 11y ago
  But at a completely different scale though. You have to imagine a whole planets sensors as your sources combined with the whole planets knowledge and so on. I don't think it would make much sense to compare with the limited POV we are experiencing the world from.
- CuriouslyC 11y ago
  Deep neural nets do precisely this. Though the specifics change from implementation to implementation, in general there is a large drop in the number of hidden units at each layer of the architecture.
clickok 11y ago
A little pedantic: you already have thousands of different input sources, from the classic "five senses" to less conscious awareness of things like your limb placement and internal biochemistry (e.g., hunger). There are even people who've "added" artificial senses by doing things like placing magnets under their skin (to sense magnetic fields) or strapping a device to their leg that buzzes in the direction of North. Not to mention the fact that we can use things like visually inspecting a screen to access senses (via information feeds) that are not available biologically. However, you're right that a robot/AI developed with the intention of feeding it all sorts of heterogeneous data will probably be able to process everything more effectively. In my own research (AI with a focus on reinforcement learning and robotics) I am sometimes surprised by how effective agents can be at making sense of their input streams. For example, an experiment will not go the way you expect because the robot can trivially solve a maze via sensing the current in the wiring beneath the floor. Of course, there's a limit in terms of how effective raw information can be. Humans don't need to see ultraviolet wavelengths because in general the spectrum of ~350-700nm provides all the information we need, and the brain is good at finding the salient aspects of what we see. If you just connect a new sensor to a robot, it might improve its ability to understand the world, or do nothing at all, because it can't incorporate this new information into its representation effectively. Or it doesn't add anything new, or at least nothing that it couldn't have figured out from existing input streams. For example, adding a stock ticker feed to your robot would probably not help it solve a particular task, unless your robot happens to be 50 feet tall and the task in question is "rampaging down Wall Street".
- ThomPete 11y ago
  Hmm sure but we are talking about a completely different scale here. I mean as an AI you are connected to the whole planets sensors, you have the whole worlds knowledge in your possession and will be able to cross reference with what you are getting as input. You can prototype, do scenario planning on the fly, you can calculate and so on. Furthermore you are potentially getting inputs from other humans too and have the ability to mostly likely control a number of things which again provide new input.

akavel 11y ago

SPOILER WARNING! Nice read at the end of the article: "[...] Naturally, this does not qualify as "seeing" in any human sense, and from a scientific perspective it certainly doesn't mean that we somehow solved computer vision at this point. Don't believe the hype; we are merely standing on the first step of a very tall ladder. Some say that the hierarchical-modular decomposition of visual space learned by a convnet is analogous to what the human visual cortex does. It may or may not be true, but there is no strong evidence to believe so. Of course, one would expect the visual cortex to learn something similar, to the extent that this constitutes a "natural" decomposition of our visual world (in much the same way that the Fourier decomposition would be a "natural" decomposition of a periodic audio signal). But the exact nature of the filters and hierarchy, and the process through which they are learned, has most likely little in common with our puny convnets. The visual cortex is not convolutional to begin with, and while it is structured in layers, the layers are themselves structured into cortical columns whose exact purpose is still not well understood --a feature not found in our artificial networks (although Geoff Hinton is working on it). Besides, there is so much more to visual perception than the classification of static pictures --human perception is fundamentally sequential and active, not static and passive, and is tightly intricated with motor control (e.g. eye saccades). Think about this next time your hear some VC or big-name CEO appear in the news to warn you against the existential threat posed by our recent advances in deep learning. Today we have better tools to map complex information spaces than we ever did before, which is awesome, but at the end of the day they are tools, not creatures, and none of what they do could reasonably qualify as "thinking". Drawing a smiley face on a rock doesn't make it "happy", even if your primate neocortex tells you so. That said, visualizing what convnets learn is quite fascinating --who would have guessed that simple gradient descent with a reasonable loss function over a sufficiently large dataset would be enough to learn this beautiful hierarchical-modular network of patterns that manages to explain a complex visual space surprisingly well. Deep learning may not be intelligence is any real sense, but it's still working considerably better than anybody could have anticipated just a few years ago. Now, if only we understood why... ;-)"

seiji 11y ago
Drawing a smiley face on a rock doesn't make it "happy", even if your primate neocortex tells you so. That's one of the best analogies we've got with regard to "deep learning" versus reality. People around here seem to think the AI apocalypse is 3-5 years away and are rushing to fund billion dollar "sentient rock" research.

cjenken 11y ago

What I find interesting is that the auras that I get as a precursor to my migraines don't look far off from those images. If you were to take a C shaped slice and overlay it onto video and then animated the color you would pretty much have it. It would be interesting to see if they were connected somehow...

Johnythree 11y ago
Yes, same here! I get the vivid flickering lights occasionally (in a C shaped slice as you describe), but luckily don't get the headaches.
mej10 11y ago
Sounds like you are describing "scintillating scotoma", if you wanted a name for that visual distortion. https://en.wikipedia.org/wiki/Scintillating_scotoma https://en.wikipedia.org/wiki/Scintillating_scotoma
Synaesthesia 11y ago
That's interesting, it also resembles the patterns I see on LSD or 2C-B

fizixer 11y ago

Question to deep learning practitioners: Is there any research on comparing the 99.9% confidence about something wrong, to the human weakness of 'optical illusion'? If you watch a few minutes of this Dan Ariely TED Talk [1], we humans are fooled into near certainty that the table on the left has longer vertical distance that the horizontal distance of that on the right. Even after the speaker shows this is not the case, we still "confidently see" the wrong thing. (not to mention, we suffer from non-optical illusions as well, like cognitive biases, and fallacious reasoning, as shown later in the talk). It seems to me you guys have discovered 'machine optical illusion'. It's just that the machine has illusions about something completely different. I believe exploring the space of architectures/weights/what-not might lead towards making a machine get into the same kind of optical illusions as the human visual system gets into. At the least, it looks like a very promising research direction to me. [1] https://youtu.be/9X68dm92HVI?t=2m22s https://youtu.be/9X68dm92HVI?t=2m22s

rasz_pl 11y ago
length illusion happens higher up the meat stack. You see the picture for what it is, but visual cortex gets overridden. Happens all the time, blue dress, motion blur, saccades. A ton of stuff seen by the eye is filtered out or overruled by upper layers.
brianchu 11y ago
That is not an example of how weak our brain is. That is an example of how incredibly intelligent our brain is. If you saw both tables in the real, physical, natural world, because of the effect of perspective, the table on the left is indeed physically longer than the width of the table on the right. Our brain is correctly inferring this from perspective/trigonometry. The visual system is designed to correctly interpret the natural world, not correctly interpret lengths of line segments on a slide. This illustrates how hard it is to create computer algorithms that robustly infer 3D from 2D (this inference requires taking cues from shading, perspective, etc.) Almost all optical illusions are examples of the strength, not weakness, of the human brain. EDIT: I've emphasized the word "natural". The problem with the below examples of "illusions" constructed in the physical world is this: what do you expect the brain to do otherwise? When you see a scene in the world, is it a weakness that our brain discards the (extremely unlikely) possibility the scene is just a massive hologram five inches from your eyes, or a meticulously painted and lit cardboard scene? I don't.

yeukhon 11y ago

Just really curious? What is up with ML on HN today? So many posts about ML in a single day. After reading this article I have to say even humans have hard time actually understand images and patterns like the one shown in the article, let alone a machine. I wonder what would a machine say about the famous "is this dress blue or grey" photo last year.

mdonahoe 11y ago

What would happen if you generated a lot of images like this, tagged them as "conv-net filter", and then added them to the set of images that the neural nets trained on? Would the network learn more discriminating filters for everything else?

dhj 11y ago
Very interesting question and worth an experiment or two. Personally, I don't think it would help because the patterns are complex and it is an internal representation. It would be like feeding it a hash values and labelling them "hash values". That wouldn't necessarily help you discriminate pre-hashed values. That said, I would definitely be interested in the results of a real test.

tariqali34 11y ago

Machines do not have to think or see in the same way as humans. To dismiss machine capablities simply because they do not resemble human intelligence seems hopelessly naive.

rasz_pl 11y ago

Its striking how similar CNNs are to good old JPEG compression. Makes me think you would get better results by chopping up input into blocks, and slapping few more layers on top. From the article description I get that VGG16 tries to guess whole picture content from existence of particular patterns totally ignoring their location and arrangement, hence nonsensical magpie prediction. Photoshop magpie with a beak sticking out its ass and it will pass as genuine because all prerequisite patterns are there.

arketyp 11y ago

If you took every image agreeing with your conception of a magpie and composed it into one image, would it look anything like a magpie? The Google inceptionism team did the same experiment but supplied the networks with natural image priors. Perhaps hand-picked, but those examples were actually quite convicining.

Synaesthesia 11y ago

we are merely standing on the first step of a very tall ladder. ... Deep learning may not be intelligence is any real sense, but it's still working considerably better than anybody could have anticipated just a few years ago. Now, if only we understood why... ;-) Yes I'd agree with this, there needs to be a deeper level analysis for how the cognitive functions within the brain, for example how vision in the brain works. Such understanding will clearly take a great amount of effort to achieve, and I don't know if research enough research is aimed in this deeper theoretical direction, as opposed to neural net type machine learning.