11 ms·

AI Has a Hallucination Problem That's Proving Tough to Fix

169 points by lnguyen 9y ago

abhishek0318 9y ago
If we are to mass deploy self driving cars, we must solve this problem. No one would want to travel by cars that could be tricked so easily. This also leads to a moral and legal questions. If a self driving car injures someone, who will be responsible, the person owning the car or the manufacturer?
- eitland 9y ago
  I think Volvo mentioned at some point that when they released fully self driving cars the responsibility would be on them as long as the car was in autonomous mode. Since then I think they've postponed the plans for fully automatic driving though.
- modernerd 9y ago
  “No one would want to travel by cars that could be tricked so easily.” https://xkcd.com/1958/ https://xkcd.com/1958/
  - perl4ever 9y ago
    That cartoon seems naive to me, in that one might be concerned about self-driving cars being vulnerable to malicious activity that scales in a way that attacks against human-driven cars don't.
    - swombat 9y ago
      Spreading tyre piercing shrapnel out your window on a busy motorway scales pretty well I imagine. Luckily moth people have better things to do with their time!
    - yorwba 9y ago
      The attacks that scale are unlikely to involve fooling the object detection pipeline by manipulating its input (because you need to physically modify the environment to do that), but rather something like compromising the firmware update servers, stealing the signing key and pushing a remote killswitch. So the weak point in this case isn't the fragility of machine learning against adversarial inputs, but old-fashioned network security.
- NewEntryHN 9y ago
  There's no need to completely solve it. It just needs to happen statistically less often than human failures. And I believe this won't be hard.
  - argonaut 9y ago
    Well currently the attacks work 100% of the time.
pdkl95 9y ago
> “People tend to trust each other in machine learning,” says Biggio. “The security mindset is exactly the opposite, you have to be always suspicious that something bad may happen.” It's not just machine learning; far too much tech - software and hardware - has this problem. Everyday here on HN you see discussions about the benefits of a new idea without any consideration for how it could be exploited.
- sillysaurus3 9y ago
  Everyday here on HN you see discussions about the benefits of a new idea Just the opposite. Rarely do you see any discussion of the benefits of a new idea. It's mostly hate. I suspect we're probably talking past each other, though. Concrete examples would resolve it. https://news.ycombinator.com/item?id=16542183 https://news.ycombinator.com/item?id=16542183 Notice the top comment is negative. In general, almost every top comment on HN is negative. It's hard to write well positively, just like it's hard to be a comedian and only make clean jokes.
  - falcolas 9y ago
    As a negative commenter on that topic: it's not a new idea. It's not even a new tool. It's an iteration on an existing tool to make it more user friendly. It's an iteration which, by all appearances, didn't go far enough. Tools, like ideas, get feedback when posted to a forum like this. Feedback is rarely going to be all positive, especially in the cases where a tool is still rough.
    - sillysaurus3 9y ago
      I didn't mean to imply it was a bad thing. I meant to say it's hard to write positively. Without straying too far from the original premise, it's worth pointing out that the idea of using a graph database for traditional tasks is in fact a new idea. Most people would not consider doing it, and that makes it new. Notice an interesting philosophical thing that happens every time philosophical debates occur: We quickly get mixed up in a debate over the meaning of words. "What do you mean by 'new'?" It's worth excluding such questions from debate. Questions over the meaning of words rarely go anywhere interesting. http://www.paulgraham.com/philosophy.html http://www.paulgraham.com/philosophy.html Oh dear, look where we've ended up. But it was an intriguing place, I hope. The overall point is that if you advocate something non-traditional on HN, you're in for a world of hurt. This place is nothing if not adversarial. You'd better have your I's dotted and your T's crossed. But those constraints are what make the medium so interesting.
kator 9y ago
One challenge I see in AI is that humans tend to attribute human traits to non-human entities (Anthropomorphism). This tendency leads people to expect something from AI that they can do quickly and to be shocked when we find AI to be brittle and lacking fundamental features of human understanding. Having raised children and now playing with my grandchildren, it often amazes me at just how much we take for granted in that comes "out of the box" with a human brain. Humans can build associations with very few samples, and we come pre-wired with all sorts of tools from primary systems like hearing, visual, sensory systems to complex capabilities like speech and communication abilities. I've worked with computers for 37 years, and the progression has been amazing, but we're still a long ways away from the primary capabilities of even a house cat. All this said I often wonder if the reason for the failure of current AI systems to wow us is the gap between power density of a human brain vs. compute systems. I've heard it said that DeepMind burned an order of magnitude more than the 20W/hr a typical human brain uses. When we have compute systems with that power density, we may see more emergent behaviors from our silicon-based friends. Either way, I think AI has useful applications today and I hope we will find many useful applications of these technologies to make our lives better and make more time for us to learn, love and care for one another.
- ianai 9y ago
  I think it’s related to learning. From my own experiences, it seems like our brains use the same technique to learn from all senses at once. Ie we can probably spot our equivalent of a random pixel through having multiple sensory inputs of all types processed together. We can know we don’t have a spot on our head despite seeing one in a mirror because we can touch or otherwise feel it is not there. That takes an incredible amount of cross reference across all of our combined senses. Similarly, my mind can differentiate from the floaters in my eyes because it remembers their existence. O do that it had to build a concept of a floater, etc.
  - danbruc 9y ago
    That can not be the important aspect, at least not in the general case. Show me only a few good photos of an object I have never seen - and one may be sufficient if the object is different enough from all the things I know - and it will become almost impossible for you to trick me into confusing it with anything else and certainly not be changing a few pixels in validation images. I may make mistakes if the image quality is bad or the lighting or if you show me an object that is at least visually very similar and where I am unaware of the distinguishing features. No where here are other senses than vision involved.
stochastic_monk 9y ago
I think that DeepXplore [0] is on the right track. I think that perhaps some methodological improvements could likely be made, but applying an adversarial, security-like approach to deep neural networks is a way to build fault-tolerance into these methods. I'll keep hoping that things like capsules and "smarter" network design/training and data augmentation will eventually, de novo, help add "safe" generalizability. Perhaps optimizing accuracy/minimizing loss in a more broadly, uniformly random way rather than in optimizing accuracy, weighted by frequency of observations. [0]: https://arxiv.org/abs/1705.06640 https://arxiv.org/abs/1705.06640
resource0x 9y ago
For the sake of argument, suppose someone hacked your program by causing buffer overflow. Then at least you know where the problem is, you go fix it and ship a patch the next day. Now suppose someone hacked your AI by causing hallucination. Do you know where the problem is? How to fix it? How long will it take to fix it? Does the fix really fix it? Etc... Not sure how this all is supposed to work.
- hannasanarion 9y ago
  The "hacking" is entirely different in that case. It's not like the person gets sudo power over your computer, it's that your car goes through a stop sign without stopping. It's easy to find the problem because you know exactly what the car should have seen, and fix that thing.
- wepple 9y ago
  Two points: - you suggest that software is easy and logical, in that you can triage, locate, and patch the buffer overflow. But in any sufficiently large codebase, there are simply near-unlimited more bugs to find: hence all the memory mitigation’s that have been implemented. So it’s not a robust comparison from the get go. - there’s constantly increasing research into being able to diagnose & analyze what CNN’s “see”: possibly not to the level and accuracy that you’d expect from lldb, but how long did it take to have really amazing debuggers for binary Applications?
  - resource0x 9y ago
    In a normal program, given an exploit, it's easy to understand the idea of attack and specific target of attack. In CNN case, because no one really understands how things work (see https://www.technologyreview.com/s/604087/the-dark-secret-at-the-heart-of-ai/ https://www.technologyreview.com/s/604087/the-dark-secret-at...), how can they understand why things don't work? The whole contraption is a black box.
    - wepple 9y ago
      > The whole contraption is a black box. That’s not true: there are massive amounts we can glean. Perhaps not as easy as reading the disassembly, but not black box. That link is far outdated/superseded by modern research: https://distill.pub/2018/building-blocks/ https://distill.pub/2018/building-blocks/
plaidfuji 9y ago
I think the obvious answer to this is to use a generative adversarial approach, but the more I think about that the more difficult it sounds in practice. Say once you've trained in the initial weights of your classifier from your training set, you have a generator start adding distortions, occlusion, and noise to the training set images, then train the classifier to recognize those as true positives. Of course the difficult part is defining a generator architecture that can learn to generate a wide enough variety of distortions, occlusions and noise...
gdubs 9y ago
Interestingly, the evolution of the human mind has dealt with similar challenges. A lot of what AI is doing (pattern matching, creative problem solving, etc) could be considered “right brain” activities. Some think that many human mental problems arise out of an unchecked, overactive right brain. We still mistake shadows on the wall for something sinister, or read more into a person’s glance than is really there. Some posit that the right brain is always hallucinating, in a sense; that psychedelic drugs simply disable the left brain, and allow the right brain to take center stage. Until the corpus callosum developed (which allows the left and right brain to send data back and forth across the divide), it’s possible that right brain insights came to us as “voices” in our heads. See: “Incognito: the secret lives of the brain.” “The Dragons of Eden: speculations on the evolution of human intelligence.”
- ohtwenty 9y ago
  This sounds like a bunch of pseudoscience? I've taken a bunch of courses on neurosciences etc, and there's certainly some left/right splits but nothing as severe as that… The wiki article is actually pretty accurate: https://en.m.wikipedia.org/wiki/Lateralization_of_brain_function https://en.m.wikipedia.org/wiki/Lateralization_of_brain_func...
  - gdubs 9y ago
    Probably more like outdated conjecture than pseudoscience (Sagan’s book, “Dragons of Eden”, was written in the 1970s), but point taken — contemporary neuroscience talks more about a modular theory of the mind than a strict Right / Left split.
    - cicero 9y ago
      It sounds to me like a desperate attempt to explain away religious experience.
falcolas 9y ago
"AI hallucination" and its negative effects is not limited to self-driving cars... for example, see the latest incident with Alexa "hallucinating" a command to laugh. What makes that scary to me, is Alexa also had to "hallucinate" the trigger words as well. How many times does Alexa "hallucinate" those trigger words and send random conversations off to a third party?
starchild_3001 9y ago
This is a highly biased article, imo. i) The authors and commenters of the article have a strong incentive to get funding for their projects. ii) The mentioned problems are mostly academic. The real-world implications are not tested, unknown, and likely overblown. Yes, one can create contrived examples. But that isn't same as real life. The way spoofing examples are created is by feeding small perturbations of the same image to a time-invariant detector. It's unclear if such attacks have much practical value. E.g. a password screen won't allow you to try more than 3-5 values. Attacks must be detectable (due to repeated tries of small perturbations). Plus noise and other perturbations can be added to the detector.
deleted 9y ago
[deleted]
bluetwo 9y ago
Thanks to those keeping AI claims grounded in truth and not hype.
deleted 9y ago
[deleted]
bjornsing 9y ago
The human visual cortex employs a combination of bottom-up and top-down processing. So before we see a dog the brain sort of generates a picture of a dog top-down, and compares that with the visual stimuli. My feeling is the problem with the "AI" described here is that it's still bottom-up only.
_dps 9y ago
I really wish the press (and a subset of highly vocal Deep Learning practitioners) would stop conflating Deep Networks with the entirety of AI/ML. This hallucination problem is 1) highly pronounced in this one class of methods that has surged in popularity in the last 10 years 2) difficult to address because these methods are (so far) quite opaque to human understanding I work on multiple in-production vision systems and in cases where we absolutely need to know why something went wrong we use much more conventional, but more transparent, learning algorithms. The performance loss is often an acceptable tradeoff for being able to understand your edge cases.
sologoub 9y ago
Actual audio from the “evil dot com” example can be found here: https://nicholas.carlini.com/code/audio_adversarial_examples/ https://nicholas.carlini.com/code/audio_adversarial_examples... It’s only hard to hear on a phone speaker at low volume on the 50dB example. All other examples have what sounds like some sort of static or background noise. Granted, to an average consumer it may sound like just bad audio, but it’s not imperceptible and thus can be screened for. Most likely we’ll end up in the same perpetual update cycle as other computer security - someone finds an exploit, that exploit is either reported for a bounty or discovered in active use in the wild, a fix is implemented and exploit is added to the test suite. For ML that will mean adding the examples of the exploits into the learning sets/providing negative re-enforcement feedbacks. The question will be whether we can get enough of these caught before some machinery injures someone because it saw something that wasn’t there or accepted a malicious command, and it becomes a media frenzy.
tptacek 9y ago
I don't have anything insightful to say about this (I think it's a super interesting area of research) but am commenting anyways to point out that Nicholas Carlini, one of the researchers cited, is also responsible for engineering the best of the later Microcorruption levels. Small world!
jeffdavis 9y ago
"The vision systems of autonomous vehicles, voice assistants able to spend money, and machine learning systems filtering unsavory content online all need to be trustworthy." I feel like we're building up to some horrible situation here, because these systems are never going to be worthy of trust in the same way that a human is. As just one example: as more and more of our lives happen digitally, more evidence of real crimes will be digital evidence. So these AI systems can be easily fooled into thinking you were in the wrong place or requested something that you didn't. And there is absolutely no way to correct the record.
- skybrian 9y ago
  It seems likely this would be handed the same way security cameras do it today, by saving the original images and having humans look at them. You might want to look into how a "chain of custody" is handed by the courts.
  - jeffdavis 9y ago
    Might work OK for video/audio (though DeepFakes calls that into question). But what about the timestamp on it? Or what about GPS location (did you even posses the phone at the time? Was the GPS data manipulated later?)? There will be a huge class of things that basically can't be questioned, denying you the ability to confront your accuser.
    - skybrian 9y ago
      Not a lawyer, but it's my understanding that the side that's presenting any physical evidence needs to show why it probably wasn't tampered with and the other side can question that. When gathering evidence that you want to hold up in court, you need to think about how to prove that.
- Gibbon1 9y ago
  > are never going to be worthy of trust in the same way that a human is. I have this worry that the glue that holds society together is empathy. Which is we feel other peoples emotional state as if it were our own. Most of us at least. You can bet for the governments and corporations that deploy AI technologies in order to control and manipulate people that is exactly the opposite of what they're interested in.
xstephen95x 9y ago
"hallucinations" is an excessive anthropomorphization of a math model in my opinion
hannasanarion 9y ago
Why are we all freaking out about the ability to "put a sticker on a stop sign that makes it invisible to YOLO". I could just as easily put a sticker on a stop sign that makes it invisible to people.
- ttlei 9y ago
  Uh... what possible type of stickers that would make a sign invisible to human?
- wepple 9y ago
  The difference is that a human will know it’s been tampered with and lower their confidence: the attacks against NNs make it think with 98% certainty it’s a 60mph sign, not a stop sign. Furthermore, if a human is monitoring the system, a small well-placed sticker will not likely alert them to the fact that the car is in trouble
visarga 9y ago
When the problem of adversarial examples will be solved, AI will have leapt one more step ahead. It's clear right now that adversarial examples are the elephant in the room. It's a make or break situation for the field but I think it will come out stronger. We're grasping at the limits of our neural network technology, trying to discriminate between ghosts of perceptions. I think the solution will come from marrying a top-down approach to the bottom-up one we're using right now. We need more prior knowledge about the world. We need to be able to simulate situations and understand their effects. Maybe what we're lacking right now is a mental simulator of the world, an imagination module. Coupling perception with imagination would reduce the sample complexity as well. Interesting to note that the researchers that started the adversarial examples craze and invented the imagination module (GAN - generative adversarial network) are one and the same - Ian Goodfellow. He was right on the spot to identify the weak point of deep learning.
sushirain 9y ago
Is it possible to create an adversarial example without access to the weights of a model, and without being able to forward many images through it?
- Fripplebubby 9y ago
  One method of creating adversarial examples in a "black box" setting is to create and train a local model as a stand-in for the actual model using the inputs and outputs of the actual model. [1] So, the answer is "no" but a qualified "no" since in practice this seems to work. The second part, being able to forward many images, is also a qualified "no". 1 - https://arxiv.org/abs/1602.02697 https://arxiv.org/abs/1602.02697
ams6110 9y ago
It doesn't seem surprising to me that AI can be fooled by adversarial input. Human brains, as advanced as they are compared to contemporary AI, are also vulnerable to this. People figured out a long time ago that hunters, soldiers, and other military assets can be camouflaged quite effectively by painting them in certain patterns, for example. Also ask any Illusionist / magician.
agjacobson 9y ago
I couldn’t resist because everyone’s thinking of cat-guacamole. Also Labsix. https://mashable.com/2017/11/02/mit-researchers-fool-google-ai-program.amp https://mashable.com/2017/11/02/mit-researchers-fool-google-...
dwaltrip 9y ago
> Making subtle changes to images, text, or audio can fool these systems into perceiving things that aren’t there. This isn't a hallucination problem. It is a robustness issue. The outputs of modern AI lack conceptual depth and substance. These algorithms produce very shallow categorizations that are only useful in narrow, constrained contexts. Not surprisingly, it isn't hard to break or hack these fragile categorizations. Sure, one could potentially argue that this has some similarity to human hallucinations, but I think that is a needless distraction. We know with great certainty that our AI techniques don't have the robustness and generality of animal intelligence. We are much less certain about the causes of human mental illness and any resulting hallucinations, so that analogy doesn't really lead us in a productive direction.
jwatte 9y ago
I am a little bit bored with these GAN style attacks. They show that the networks haven't yet generalized well enough. They don't show that the technology, or even the approach, is broken. Personally, I think the main problem is some combination of a little bit of over claiming in research, and a metric ton of over hyping and generalizing in both media and business circles. Robust multi model ensembles with strong generalization ability will show up within the foreseeable future, and will be no more susceptible to optical illusions than human beings.

169 points by lnguyen 9y ago

If we are to mass deploy self driving cars, we must solve this problem. No one would want to travel by cars that could be tricked so easily. This also leads to a moral and legal questions. If a self driving car injures someone, who will be responsible, the person owning the car or the manufacturer?

eitland 9y ago
I think Volvo mentioned at some point that when they released fully self driving cars the responsibility would be on them as long as the car was in autonomous mode. Since then I think they've postponed the plans for fully automatic driving though.
modernerd 9y ago
“No one would want to travel by cars that could be tricked so easily.” https://xkcd.com/1958/ https://xkcd.com/1958/
- perl4ever 9y ago
  That cartoon seems naive to me, in that one might be concerned about self-driving cars being vulnerable to malicious activity that scales in a way that attacks against human-driven cars don't.
  - swombat 9y ago
    Spreading tyre piercing shrapnel out your window on a busy motorway scales pretty well I imagine. Luckily moth people have better things to do with their time!
  - yorwba 9y ago
    The attacks that scale are unlikely to involve fooling the object detection pipeline by manipulating its input (because you need to physically modify the environment to do that), but rather something like compromising the firmware update servers, stealing the signing key and pushing a remote killswitch. So the weak point in this case isn't the fragility of machine learning against adversarial inputs, but old-fashioned network security.
NewEntryHN 9y ago
There's no need to completely solve it. It just needs to happen statistically less often than human failures. And I believe this won't be hard.
- argonaut 9y ago
  Well currently the attacks work 100% of the time.

pdkl95 9y ago

> “People tend to trust each other in machine learning,” says Biggio. “The security mindset is exactly the opposite, you have to be always suspicious that something bad may happen.” It's not just machine learning; far too much tech - software and hardware - has this problem. Everyday here on HN you see discussions about the benefits of a new idea without any consideration for how it could be exploited.

sillysaurus3 9y ago
Everyday here on HN you see discussions about the benefits of a new idea Just the opposite. Rarely do you see any discussion of the benefits of a new idea. It's mostly hate. I suspect we're probably talking past each other, though. Concrete examples would resolve it. https://news.ycombinator.com/item?id=16542183 https://news.ycombinator.com/item?id=16542183 Notice the top comment is negative. In general, almost every top comment on HN is negative. It's hard to write well positively, just like it's hard to be a comedian and only make clean jokes.
- falcolas 9y ago
  As a negative commenter on that topic: it's not a new idea. It's not even a new tool. It's an iteration on an existing tool to make it more user friendly. It's an iteration which, by all appearances, didn't go far enough. Tools, like ideas, get feedback when posted to a forum like this. Feedback is rarely going to be all positive, especially in the cases where a tool is still rough.
  - sillysaurus3 9y ago
    I didn't mean to imply it was a bad thing. I meant to say it's hard to write positively. Without straying too far from the original premise, it's worth pointing out that the idea of using a graph database for traditional tasks is in fact a new idea. Most people would not consider doing it, and that makes it new. Notice an interesting philosophical thing that happens every time philosophical debates occur: We quickly get mixed up in a debate over the meaning of words. "What do you mean by 'new'?" It's worth excluding such questions from debate. Questions over the meaning of words rarely go anywhere interesting. http://www.paulgraham.com/philosophy.html http://www.paulgraham.com/philosophy.html Oh dear, look where we've ended up. But it was an intriguing place, I hope. The overall point is that if you advocate something non-traditional on HN, you're in for a world of hurt. This place is nothing if not adversarial. You'd better have your I's dotted and your T's crossed. But those constraints are what make the medium so interesting.

kator 9y ago

One challenge I see in AI is that humans tend to attribute human traits to non-human entities (Anthropomorphism). This tendency leads people to expect something from AI that they can do quickly and to be shocked when we find AI to be brittle and lacking fundamental features of human understanding. Having raised children and now playing with my grandchildren, it often amazes me at just how much we take for granted in that comes "out of the box" with a human brain. Humans can build associations with very few samples, and we come pre-wired with all sorts of tools from primary systems like hearing, visual, sensory systems to complex capabilities like speech and communication abilities. I've worked with computers for 37 years, and the progression has been amazing, but we're still a long ways away from the primary capabilities of even a house cat. All this said I often wonder if the reason for the failure of current AI systems to wow us is the gap between power density of a human brain vs. compute systems. I've heard it said that DeepMind burned an order of magnitude more than the 20W/hr a typical human brain uses. When we have compute systems with that power density, we may see more emergent behaviors from our silicon-based friends. Either way, I think AI has useful applications today and I hope we will find many useful applications of these technologies to make our lives better and make more time for us to learn, love and care for one another.

ianai 9y ago
I think it’s related to learning. From my own experiences, it seems like our brains use the same technique to learn from all senses at once. Ie we can probably spot our equivalent of a random pixel through having multiple sensory inputs of all types processed together. We can know we don’t have a spot on our head despite seeing one in a mirror because we can touch or otherwise feel it is not there. That takes an incredible amount of cross reference across all of our combined senses. Similarly, my mind can differentiate from the floaters in my eyes because it remembers their existence. O do that it had to build a concept of a floater, etc.
- danbruc 9y ago
  That can not be the important aspect, at least not in the general case. Show me only a few good photos of an object I have never seen - and one may be sufficient if the object is different enough from all the things I know - and it will become almost impossible for you to trick me into confusing it with anything else and certainly not be changing a few pixels in validation images. I may make mistakes if the image quality is bad or the lighting or if you show me an object that is at least visually very similar and where I am unaware of the distinguishing features. No where here are other senses than vision involved.

stochastic_monk 9y ago

I think that DeepXplore [0] is on the right track. I think that perhaps some methodological improvements could likely be made, but applying an adversarial, security-like approach to deep neural networks is a way to build fault-tolerance into these methods. I'll keep hoping that things like capsules and "smarter" network design/training and data augmentation will eventually, de novo, help add "safe" generalizability. Perhaps optimizing accuracy/minimizing loss in a more broadly, uniformly random way rather than in optimizing accuracy, weighted by frequency of observations. [0]: https://arxiv.org/abs/1705.06640 https://arxiv.org/abs/1705.06640

resource0x 9y ago

For the sake of argument, suppose someone hacked your program by causing buffer overflow. Then at least you know where the problem is, you go fix it and ship a patch the next day. Now suppose someone hacked your AI by causing hallucination. Do you know where the problem is? How to fix it? How long will it take to fix it? Does the fix really fix it? Etc... Not sure how this all is supposed to work.

hannasanarion 9y ago
The "hacking" is entirely different in that case. It's not like the person gets sudo power over your computer, it's that your car goes through a stop sign without stopping. It's easy to find the problem because you know exactly what the car should have seen, and fix that thing.
wepple 9y ago
Two points: - you suggest that software is easy and logical, in that you can triage, locate, and patch the buffer overflow. But in any sufficiently large codebase, there are simply near-unlimited more bugs to find: hence all the memory mitigation’s that have been implemented. So it’s not a robust comparison from the get go. - there’s constantly increasing research into being able to diagnose & analyze what CNN’s “see”: possibly not to the level and accuracy that you’d expect from lldb, but how long did it take to have really amazing debuggers for binary Applications?
- resource0x 9y ago
  In a normal program, given an exploit, it's easy to understand the idea of attack and specific target of attack. In CNN case, because no one really understands how things work (see https://www.technologyreview.com/s/604087/the-dark-secret-at-the-heart-of-ai/ https://www.technologyreview.com/s/604087/the-dark-secret-at...), how can they understand why things don't work? The whole contraption is a black box.
  - wepple 9y ago
    > The whole contraption is a black box. That’s not true: there are massive amounts we can glean. Perhaps not as easy as reading the disassembly, but not black box. That link is far outdated/superseded by modern research: https://distill.pub/2018/building-blocks/ https://distill.pub/2018/building-blocks/

plaidfuji 9y ago

I think the obvious answer to this is to use a generative adversarial approach, but the more I think about that the more difficult it sounds in practice. Say once you've trained in the initial weights of your classifier from your training set, you have a generator start adding distortions, occlusion, and noise to the training set images, then train the classifier to recognize those as true positives. Of course the difficult part is defining a generator architecture that can learn to generate a wide enough variety of distortions, occlusions and noise...

gdubs 9y ago

Interestingly, the evolution of the human mind has dealt with similar challenges. A lot of what AI is doing (pattern matching, creative problem solving, etc) could be considered “right brain” activities. Some think that many human mental problems arise out of an unchecked, overactive right brain. We still mistake shadows on the wall for something sinister, or read more into a person’s glance than is really there. Some posit that the right brain is always hallucinating, in a sense; that psychedelic drugs simply disable the left brain, and allow the right brain to take center stage. Until the corpus callosum developed (which allows the left and right brain to send data back and forth across the divide), it’s possible that right brain insights came to us as “voices” in our heads. See: “Incognito: the secret lives of the brain.” “The Dragons of Eden: speculations on the evolution of human intelligence.”

ohtwenty 9y ago
This sounds like a bunch of pseudoscience? I've taken a bunch of courses on neurosciences etc, and there's certainly some left/right splits but nothing as severe as that… The wiki article is actually pretty accurate: https://en.m.wikipedia.org/wiki/Lateralization_of_brain_function https://en.m.wikipedia.org/wiki/Lateralization_of_brain_func...
- gdubs 9y ago
  Probably more like outdated conjecture than pseudoscience (Sagan’s book, “Dragons of Eden”, was written in the 1970s), but point taken — contemporary neuroscience talks more about a modular theory of the mind than a strict Right / Left split.
  - cicero 9y ago
    It sounds to me like a desperate attempt to explain away religious experience.

falcolas 9y ago

"AI hallucination" and its negative effects is not limited to self-driving cars... for example, see the latest incident with Alexa "hallucinating" a command to laugh. What makes that scary to me, is Alexa also had to "hallucinate" the trigger words as well. How many times does Alexa "hallucinate" those trigger words and send random conversations off to a third party?

starchild_3001 9y ago

This is a highly biased article, imo. i) The authors and commenters of the article have a strong incentive to get funding for their projects. ii) The mentioned problems are mostly academic. The real-world implications are not tested, unknown, and likely overblown. Yes, one can create contrived examples. But that isn't same as real life. The way spoofing examples are created is by feeding small perturbations of the same image to a time-invariant detector. It's unclear if such attacks have much practical value. E.g. a password screen won't allow you to try more than 3-5 values. Attacks must be detectable (due to repeated tries of small perturbations). Plus noise and other perturbations can be added to the detector.

deleted 9y ago

[deleted]

bluetwo 9y ago

Thanks to those keeping AI claims grounded in truth and not hype.

deleted 9y ago

[deleted]

bjornsing 9y ago

The human visual cortex employs a combination of bottom-up and top-down processing. So before we see a dog the brain sort of generates a picture of a dog top-down, and compares that with the visual stimuli. My feeling is the problem with the "AI" described here is that it's still bottom-up only.

_dps 9y ago

I really wish the press (and a subset of highly vocal Deep Learning practitioners) would stop conflating Deep Networks with the entirety of AI/ML. This hallucination problem is 1) highly pronounced in this one class of methods that has surged in popularity in the last 10 years 2) difficult to address because these methods are (so far) quite opaque to human understanding I work on multiple in-production vision systems and in cases where we absolutely need to know why something went wrong we use much more conventional, but more transparent, learning algorithms. The performance loss is often an acceptable tradeoff for being able to understand your edge cases.

sologoub 9y ago

Actual audio from the “evil dot com” example can be found here: https://nicholas.carlini.com/code/audio_adversarial_examples/ https://nicholas.carlini.com/code/audio_adversarial_examples... It’s only hard to hear on a phone speaker at low volume on the 50dB example. All other examples have what sounds like some sort of static or background noise. Granted, to an average consumer it may sound like just bad audio, but it’s not imperceptible and thus can be screened for. Most likely we’ll end up in the same perpetual update cycle as other computer security - someone finds an exploit, that exploit is either reported for a bounty or discovered in active use in the wild, a fix is implemented and exploit is added to the test suite. For ML that will mean adding the examples of the exploits into the learning sets/providing negative re-enforcement feedbacks. The question will be whether we can get enough of these caught before some machinery injures someone because it saw something that wasn’t there or accepted a malicious command, and it becomes a media frenzy.

tptacek 9y ago

I don't have anything insightful to say about this (I think it's a super interesting area of research) but am commenting anyways to point out that Nicholas Carlini, one of the researchers cited, is also responsible for engineering the best of the later Microcorruption levels. Small world!

jeffdavis 9y ago

"The vision systems of autonomous vehicles, voice assistants able to spend money, and machine learning systems filtering unsavory content online all need to be trustworthy." I feel like we're building up to some horrible situation here, because these systems are never going to be worthy of trust in the same way that a human is. As just one example: as more and more of our lives happen digitally, more evidence of real crimes will be digital evidence. So these AI systems can be easily fooled into thinking you were in the wrong place or requested something that you didn't. And there is absolutely no way to correct the record.

skybrian 9y ago
It seems likely this would be handed the same way security cameras do it today, by saving the original images and having humans look at them. You might want to look into how a "chain of custody" is handed by the courts.
- jeffdavis 9y ago
  Might work OK for video/audio (though DeepFakes calls that into question). But what about the timestamp on it? Or what about GPS location (did you even posses the phone at the time? Was the GPS data manipulated later?)? There will be a huge class of things that basically can't be questioned, denying you the ability to confront your accuser.
  - skybrian 9y ago
    Not a lawyer, but it's my understanding that the side that's presenting any physical evidence needs to show why it probably wasn't tampered with and the other side can question that. When gathering evidence that you want to hold up in court, you need to think about how to prove that.
Gibbon1 9y ago
> are never going to be worthy of trust in the same way that a human is. I have this worry that the glue that holds society together is empathy. Which is we feel other peoples emotional state as if it were our own. Most of us at least. You can bet for the governments and corporations that deploy AI technologies in order to control and manipulate people that is exactly the opposite of what they're interested in.

xstephen95x 9y ago

"hallucinations" is an excessive anthropomorphization of a math model in my opinion

hannasanarion 9y ago

Why are we all freaking out about the ability to "put a sticker on a stop sign that makes it invisible to YOLO". I could just as easily put a sticker on a stop sign that makes it invisible to people.

ttlei 9y ago
Uh... what possible type of stickers that would make a sign invisible to human?
wepple 9y ago
The difference is that a human will know it’s been tampered with and lower their confidence: the attacks against NNs make it think with 98% certainty it’s a 60mph sign, not a stop sign. Furthermore, if a human is monitoring the system, a small well-placed sticker will not likely alert them to the fact that the car is in trouble

visarga 9y ago

When the problem of adversarial examples will be solved, AI will have leapt one more step ahead. It's clear right now that adversarial examples are the elephant in the room. It's a make or break situation for the field but I think it will come out stronger. We're grasping at the limits of our neural network technology, trying to discriminate between ghosts of perceptions. I think the solution will come from marrying a top-down approach to the bottom-up one we're using right now. We need more prior knowledge about the world. We need to be able to simulate situations and understand their effects. Maybe what we're lacking right now is a mental simulator of the world, an imagination module. Coupling perception with imagination would reduce the sample complexity as well. Interesting to note that the researchers that started the adversarial examples craze and invented the imagination module (GAN - generative adversarial network) are one and the same - Ian Goodfellow. He was right on the spot to identify the weak point of deep learning.

sushirain 9y ago

Is it possible to create an adversarial example without access to the weights of a model, and without being able to forward many images through it?

Fripplebubby 9y ago
One method of creating adversarial examples in a "black box" setting is to create and train a local model as a stand-in for the actual model using the inputs and outputs of the actual model. [1] So, the answer is "no" but a qualified "no" since in practice this seems to work. The second part, being able to forward many images, is also a qualified "no". 1 - https://arxiv.org/abs/1602.02697 https://arxiv.org/abs/1602.02697

ams6110 9y ago

It doesn't seem surprising to me that AI can be fooled by adversarial input. Human brains, as advanced as they are compared to contemporary AI, are also vulnerable to this. People figured out a long time ago that hunters, soldiers, and other military assets can be camouflaged quite effectively by painting them in certain patterns, for example. Also ask any Illusionist / magician.

agjacobson 9y ago

I couldn’t resist because everyone’s thinking of cat-guacamole. Also Labsix. https://mashable.com/2017/11/02/mit-researchers-fool-google-ai-program.amp https://mashable.com/2017/11/02/mit-researchers-fool-google-...

dwaltrip 9y ago

> Making subtle changes to images, text, or audio can fool these systems into perceiving things that aren’t there. This isn't a hallucination problem. It is a robustness issue. The outputs of modern AI lack conceptual depth and substance. These algorithms produce very shallow categorizations that are only useful in narrow, constrained contexts. Not surprisingly, it isn't hard to break or hack these fragile categorizations. Sure, one could potentially argue that this has some similarity to human hallucinations, but I think that is a needless distraction. We know with great certainty that our AI techniques don't have the robustness and generality of animal intelligence. We are much less certain about the causes of human mental illness and any resulting hallucinations, so that analogy doesn't really lead us in a productive direction.

jwatte 9y ago

I am a little bit bored with these GAN style attacks. They show that the networks haven't yet generalized well enough. They don't show that the technology, or even the approach, is broken. Personally, I think the main problem is some combination of a little bit of over claiming in research, and a metric ton of over hyping and generalizing in both media and business circles. Robust multi model ensembles with strong generalization ability will show up within the foreseeable future, and will be no more susceptible to optical illusions than human beings.