4 ms·

I never understood the whole notion of attacks on deep-learning systems and all the research effort spent on defending against such attacks. After all, we're ta

I never understood the whole notion of attacks on deep-learning systems and all the research effort spent on defending against such attacks. After all, we're talking about systems that are brittle in the face of non-adversarial non-attacks, e.g. http://news.mit.edu/2019/object-recognition-dataset-stumped-worlds-best-computer-vision-models-1210 http://news.mit.edu/2019/object-recognition-dataset-stumped-.... The language of attacks and defenses implies that we are approaching the kind of robustness that we expect from say a bank app, when in fact we are lightyears away from that.

0xab 7y ago
I'm the lead author of the paper you cited. Glad you enjoyed our work :) Sure, you can break systems. That doesn't mean that they aren't useful! In many cases a system will see the same boring input many times over. People are often willing to be a bit flexible and help out when it happens to misread something. The fact that you can intentionally break systems like that, and that you can break them in a particular direction, like making them always think there's no danger in an image, is really worrisome. Our work shows that your autonomous car won't always work well; that its vision system has some systematic error which we can characterize now. Adversarial attacks show that someone can intentionally make your car see a lane, whenever and wherever they feel like it, and drive you off the road. It's a whole different ballgame, and the language of attack and defense really fits well.
hannxu 7y ago
Maybe we want the machines think as human think
ChandlerBang 7y ago
Thank @0xab for answering this question. Actually I hold the same opinion as @0xab. Those "non-adversarial non-attack" happens sometimes and there also exist adversarial attacks where adversary intentionally craft some pictures to fool the system. In some cases, the system already achieves a pretty high performance and generalize very well on various datasets but they may fail on adversarial images. For example, attackers can put some carefully-crafted stickers on the STOP sign and the self-driving car will give wrong predictions. The existence of adversarial attacks poses great challenges in applying deep learning model in safety-critical scenarios. Both non-adversarial non-attack and adversarial attack reflect the vulnerability of the current deep learning models. Thus it is of great significance to study these problem and find countermeasures against those attacks.

0xab 7y ago

I'm the lead author of the paper you cited. Glad you enjoyed our work :) Sure, you can break systems. That doesn't mean that they aren't useful! In many cases a system will see the same boring input many times over. People are often willing to be a bit flexible and help out when it happens to misread something. The fact that you can intentionally break systems like that, and that you can break them in a particular direction, like making them always think there's no danger in an image, is really worrisome. Our work shows that your autonomous car won't always work well; that its vision system has some systematic error which we can characterize now. Adversarial attacks show that someone can intentionally make your car see a lane, whenever and wherever they feel like it, and drive you off the road. It's a whole different ballgame, and the language of attack and defense really fits well.

hannxu 7y ago

Maybe we want the machines think as human think

ChandlerBang 7y ago

Thank @0xab for answering this question. Actually I hold the same opinion as @0xab. Those "non-adversarial non-attack" happens sometimes and there also exist adversarial attacks where adversary intentionally craft some pictures to fool the system. In some cases, the system already achieves a pretty high performance and generalize very well on various datasets but they may fail on adversarial images. For example, attackers can put some carefully-crafted stickers on the STOP sign and the self-driving car will give wrong predictions. The existence of adversarial attacks poses great challenges in applying deep learning model in safety-critical scenarios. Both non-adversarial non-attack and adversarial attack reflect the vulnerability of the current deep learning models. Thus it is of great significance to study these problem and find countermeasures against those attacks.