5 ms·

I'm finishing a survey paper that discusses research with adversarial examples plus about 9 or 10 other attacks on or weaknesses of neural networks (and machine

by mholt 10y ago

I'm finishing a survey paper that discusses research with adversarial examples plus about 9 or 10 other attacks on or weaknesses of neural networks (and machine learning models in general). Overall conclusion is: much like the early Internet, we're rapidly advancing towards machine learning tech that works but isn't secure. And 20 years later, we're still trying to make the Internet secure... If neural networks are here to stay, maybe we should slow down their public deployment for a moment and understand them better first. It would be ideal to find fundamental structural/algorithmic changes that can harden them rather than relying on heuristics or other "wrappers" to make input/output safe to use in autonomous environments. The more that is "extra", the less those security features will be implemented. (We see this rampantly on the web today with HTTPS.)

tormeh 10y ago
Worse is better. Sure, making the internet safe from the start seems technically superior, but I guess that it would have delayed adoption so severely we wouldn't have come out ahead.
- sathackr 10y ago
  Exactly. Which is worse overall? No Internet? Or a hopelessly insecure internet? Should we wait till something is perfect before we deploy? Should we hold back on self driving cars until the tech is completely foolproof? What about the number of accidents/deaths that could be prevented by even a flawed autonomous car? Sometimes insecure and imperfect now is better than delaying a technology years until it can be 'perfected'.
  - mcguire 10y ago
    This is a fundamental problem with statistical approaches like NNs: we currently have no good idea when our how they will fail; it's the explanation problem. Putting a great deal of reliance on a system in that situation seems like a great mistake. "It looks like it works under normal circumstances" is not very reliable.
    - scarmig 10y ago
      Devil's advocate: suppose humans succeed only 99% of the time, but we can understand and characterize the 1% of failures fairly well. Now suppose machine learning-based approaches succeed 99.9% of the time and only fail 0.1%, but that 0.1% is for totally unknowable reasons. Is the former better than the latter? It's fun to imagine someone constructing adversarial examples to cause self-driving cars to crash. But is that really a big deal? Humans can just as easily be tripped up by an "adversarial" example: I can pretty easily point a laser pointer from my window at random humans driving a car. The legal system has well-defined mechanisms for dealing with this.
      - mcguire 10y ago
        Collectively, we have a lot of experience with the foibles of our visual system (to the extent that we don't notice what it does poorly, which is a problem, too). Further, we've arranged civilization to play to its strengths, which is one of the reasons we're so interested in facial recognition. The recognition systems look like they are doing the same thing we are, but they're not. Adding an adversary taking advantage of the difference is just sauce on top of the problem.
- mholt 10y ago
  Hard to say. Would the world have been less thirsty for the Internet ~5 years later? I'm just suggesting (not strongly in the paper, but here in the comments) that we carefully balance deployment with hardening.
- joe_the_user 10y ago
  The deployment of the Internet and the deployment of deep learning systems are quite different issues. The Internet faced the problem of harnessing the power of computer networks required a critical mass and that none of the "ideal" conceptions could achieve this (especially since "ideals" usually didn't understand the full potential of the system). Here "worse is better" made sense. Neural Nets and related systems are deployed by large companies with lots of data and used to provide guidance and decision making (where network effect isn't a big factor either way). With "guidance" - product recommendations or optional translations, the dangers seem relatively small. With decision making systems, the dangers seems considerable and not totally thought through - you have everything from people falsely flagged by face-recognition software to neural networks that derive racist loan policies to the spoofing potential that management recommendation engines might offer. Could you spoof a stock decision system by making a series of stock trades that has previous preceded a big move? I don't know but I assume someone is thinking of ways to use neural spoofing to get something from someone as we speak.
- Asooka 10y ago
  See: project Xanadu.
- craigching 10y ago
  I may be restating some of what has been said elsewhere, but I hope that this simplifies things a bit. What I believe the grandparent said was we should slow-roll neural networks where they have severe consequences, as in, life and death. I agree, slow-rolling the internet would have potentially affected adoption, but the stakes are lower than, say, a self-driving car going off the rails. In those cases, we should probably thoroughly understand how a car can be affected by adversarial attempts at "breaking the algo" before adopting it very widely. In other words, I wouldn't slow roll the internet because of fears of adversarial behavior, but I would in the face of self-driving cars. I hope that further discussion could describe how internet security is as life-and-death as self-driving cars, or to the contrary :)
  - Dylan16807 10y ago
    We don't have to be particularly worried about self-driving cars being confused by malicious physical actors. Such an actor could always do something like cut the brake lines instead.
    - mholt 10y ago
      That's easy to detect. A car with cut brake lines can choose not to drive. A car doesn't know when its neural net is being fooled.
zitterbewegung 10y ago
I was at DEFCON and the most interesting thing I saw there was that not only can you confuse deep learning networks but your adversarial examples have a high chance of transferring to other ML techniques. See https://raw.githubusercontent.com/cchio/deep-pwning/master/dpwn/repo-assets/transferability-misclassificaiton-matrix.png https://raw.githubusercontent.com/cchio/deep-pwning/master/d...
- moultano 10y ago
  The best explanation of why these transfer that I've seen is that the problem inherently arises from dot-products in high dimensions. Real data is somewhat random w.r.t. any particular set of activation coefficients, so the dot product looks like a sum of random variables and is subject to the central limit theorem. This keeps the real dot products small. If you are able to inject a teeny bit of the activation coefficients into the input, then as the dimensionality grows it gets easier and easier to overwhelm the real data with an epsilon sized change to the input.
peteretep 10y ago
> maybe we should slow down Please submit your proposal after the one for slowing down population growth, but before your one for slowing down general arms proliferation. Thanks.

by mholt 10y ago

tormeh 10y ago

Worse is better. Sure, making the internet safe from the start seems technically superior, but I guess that it would have delayed adoption so severely we wouldn't have come out ahead.

sathackr 10y ago
Exactly. Which is worse overall? No Internet? Or a hopelessly insecure internet? Should we wait till something is perfect before we deploy? Should we hold back on self driving cars until the tech is completely foolproof? What about the number of accidents/deaths that could be prevented by even a flawed autonomous car? Sometimes insecure and imperfect now is better than delaying a technology years until it can be 'perfected'.
- mcguire 10y ago
  This is a fundamental problem with statistical approaches like NNs: we currently have no good idea when our how they will fail; it's the explanation problem. Putting a great deal of reliance on a system in that situation seems like a great mistake. "It looks like it works under normal circumstances" is not very reliable.
  - scarmig 10y ago
    Devil's advocate: suppose humans succeed only 99% of the time, but we can understand and characterize the 1% of failures fairly well. Now suppose machine learning-based approaches succeed 99.9% of the time and only fail 0.1%, but that 0.1% is for totally unknowable reasons. Is the former better than the latter? It's fun to imagine someone constructing adversarial examples to cause self-driving cars to crash. But is that really a big deal? Humans can just as easily be tripped up by an "adversarial" example: I can pretty easily point a laser pointer from my window at random humans driving a car. The legal system has well-defined mechanisms for dealing with this.
    - mcguire 10y ago
      Collectively, we have a lot of experience with the foibles of our visual system (to the extent that we don't notice what it does poorly, which is a problem, too). Further, we've arranged civilization to play to its strengths, which is one of the reasons we're so interested in facial recognition. The recognition systems look like they are doing the same thing we are, but they're not. Adding an adversary taking advantage of the difference is just sauce on top of the problem.
mholt 10y ago
Hard to say. Would the world have been less thirsty for the Internet ~5 years later? I'm just suggesting (not strongly in the paper, but here in the comments) that we carefully balance deployment with hardening.
joe_the_user 10y ago
The deployment of the Internet and the deployment of deep learning systems are quite different issues. The Internet faced the problem of harnessing the power of computer networks required a critical mass and that none of the "ideal" conceptions could achieve this (especially since "ideals" usually didn't understand the full potential of the system). Here "worse is better" made sense. Neural Nets and related systems are deployed by large companies with lots of data and used to provide guidance and decision making (where network effect isn't a big factor either way). With "guidance" - product recommendations or optional translations, the dangers seem relatively small. With decision making systems, the dangers seems considerable and not totally thought through - you have everything from people falsely flagged by face-recognition software to neural networks that derive racist loan policies to the spoofing potential that management recommendation engines might offer. Could you spoof a stock decision system by making a series of stock trades that has previous preceded a big move? I don't know but I assume someone is thinking of ways to use neural spoofing to get something from someone as we speak.
Asooka 10y ago
See: project Xanadu.
craigching 10y ago
I may be restating some of what has been said elsewhere, but I hope that this simplifies things a bit. What I believe the grandparent said was we should slow-roll neural networks where they have severe consequences, as in, life and death. I agree, slow-rolling the internet would have potentially affected adoption, but the stakes are lower than, say, a self-driving car going off the rails. In those cases, we should probably thoroughly understand how a car can be affected by adversarial attempts at "breaking the algo" before adopting it very widely. In other words, I wouldn't slow roll the internet because of fears of adversarial behavior, but I would in the face of self-driving cars. I hope that further discussion could describe how internet security is as life-and-death as self-driving cars, or to the contrary :)
- Dylan16807 10y ago
  We don't have to be particularly worried about self-driving cars being confused by malicious physical actors. Such an actor could always do something like cut the brake lines instead.
  - mholt 10y ago
    That's easy to detect. A car with cut brake lines can choose not to drive. A car doesn't know when its neural net is being fooled.

zitterbewegung 10y ago

I was at DEFCON and the most interesting thing I saw there was that not only can you confuse deep learning networks but your adversarial examples have a high chance of transferring to other ML techniques. See https://raw.githubusercontent.com/cchio/deep-pwning/master/dpwn/repo-assets/transferability-misclassificaiton-matrix.png https://raw.githubusercontent.com/cchio/deep-pwning/master/d...

moultano 10y ago
The best explanation of why these transfer that I've seen is that the problem inherently arises from dot-products in high dimensions. Real data is somewhat random w.r.t. any particular set of activation coefficients, so the dot product looks like a sum of random variables and is subject to the central limit theorem. This keeps the real dot products small. If you are able to inject a teeny bit of the activation coefficients into the input, then as the dimensionality grows it gets easier and easier to overwhelm the real data with an epsilon sized change to the input.

peteretep 10y ago

> maybe we should slow down Please submit your proposal after the one for slowing down population growth, but before your one for slowing down general arms proliferation. Thanks.