3 ms·

i understand why we'd like to have near-perfect classification but do we need 100% accuracy in practice? for example, the paper identifies an instance where a

i understand why we'd like to have near-perfect classification but do we need 100% accuracy in practice? for example, the paper identifies an instance where a stop sign was misclassified as a speed limit under various conditions. however, wouldnt applying bayesian priors ("how often do i see speed limits at intersections?") presumably negate misclassifications? rather than chase the elusive ~100% accuracy from a single model, why not take a layered, ensemble approach?

throwawayjava 9y ago
I think the traffic sign example was a nice setting for explaining the research. But of course really good examples always risk interfering with communication as much as they assist communication. I think that might've happened here. Concretely, I have a hard time believing that adversarial traffic signs will be the Achilles heel of self-driving cars. Vandalizing signs is already illegal, and if doing so causes or was intended to cause a crash, law enforcement will throw the book at the culprit. And this already happens to human drivers. See e.g., http://globegazette.com/news/local/prank-turns-to-peril-vandalism-called-likely-cause-of-serious/article_ae3c15e9-f3c0-5b00-a21d-2804a4d08806.html http://globegazette.com/news/local/prank-turns-to-peril-vand... or http://articles.latimes.com/1997-06-11/news/mn-2303_1_one-stop-sign http://articles.latimes.com/1997-06-11/news/mn-2303_1_one-st... Besides which, you can also just rip the sign out of the ground or replace the stop sign with an actual speed limit sign. From a SDC perspective, this is kind of the equivalent of using super clever software vulnerabilities when you have physical access to an un-encrypted HDD. Impressive technical tour de force, but misses the forest for the trees if your goal is security. Traffic sign vandalism is a social problem with a social solution. The first person who tries something like this in production with malicious intent toward SDCs will be charged with some variation of attempted or actual manslaughter/murder and opponents of SDCs will switch to strategies other than domestic terrorism to advance their agenda. This research is still very interesting and useful, of course, but primarily for other reasons. I just don't see traffic sign vandalism intended to confuse DNNs as a serious -- or unique -- threat to self-driving cars.
- cscurmudgeon 9y ago
  I think there is a point here that is being overlooked. The vandalism, a couple of small stickers, would be less detectable compared with a whole missing sign. I can see such attack methods evolving to produce undetectetable (to humans without software) adversarial images.
  - throwawayjava 9y ago
    I'm not saying we shouldn't practice defense in depth where it's possible. It's just that I don't think preventing this style of attack is actually the most relevant/impactful application of this style of research. Again, I'm not saying this research is irrelevant. I'm saying it's way more relevant than this particular example implies. > would be less detectable compared with a whole missing sign. This kind of assumes that the attack wouldn't work in the first place, no? If cars are crashing people are going to investigate to figure out why, and even the most thick-skulled beat cop can figure out that stop signs with rando weird ass stickers on them are probably worth running up the chain...
    - cscurmudgeon 9y ago
      > This kind of assumes that the attack wouldn't work in the first place, no? If cars are crashing people are going to investigate to figure out why, and even the most thick-skulled beat cop can figure out that stop signs with rando weird ass stickers on them are probably worth running up the chain... 1. That attack may not target only self-driving cars. 2. Imagine someone driving around with some random picture on their car at the back. Except that the picture is an attack.
- pessimizer 9y ago
  > Traffic sign vandalism is a social problem with a social solution. The first person who tries something like this in production with malicious intent toward SDCs will be charged with some variation of attempted or actual manslaughter/murder and opponents of SDCs will switch to strategies other than domestic terrorism to advance their agenda. It's tunnelvision to assume that the only attackers would be individual radical luddites. There do exist terrorist organizations and state actors who intend to run risks to do damage. And unless you're planning to monitor and respond quickly to all potential threats to all stop signs at all times (and all other possible attacks), what's to stop a group of 40 terrorists from covering their faces, stealing motorcycles, and putting 10,000 stickers on 10,000 signs in 10,000 intersections in 20 different cities in 1 night?
mholt 9y ago
A bigger concern in my mind is where 100% accuracy is needed, like when doing authentication (cf. any login screen that lets you in by looking at your face). A lot of experiments are done on visual data sets, but I don't see why these techniques need to be strictly for visual inputs. Is a machine learning model making decisions based on the results of its classifier? Then it is at risk of being fooled, and whatever decisions it's making, better be trivial ones. I wrote a document a few months ago that summarizes this and other risks that ML models currently face: https://matt.life/papers/security_privacy_neural_networks.pdf https://matt.life/papers/security_privacy_neural_networks.pd...

throwawayjava 9y ago

I think the traffic sign example was a nice setting for explaining the research. But of course really good examples always risk interfering with communication as much as they assist communication. I think that might've happened here. Concretely, I have a hard time believing that adversarial traffic signs will be the Achilles heel of self-driving cars. Vandalizing signs is already illegal, and if doing so causes or was intended to cause a crash, law enforcement will throw the book at the culprit. And this already happens to human drivers. See e.g., http://globegazette.com/news/local/prank-turns-to-peril-vandalism-called-likely-cause-of-serious/article_ae3c15e9-f3c0-5b00-a21d-2804a4d08806.html http://globegazette.com/news/local/prank-turns-to-peril-vand... or http://articles.latimes.com/1997-06-11/news/mn-2303_1_one-stop-sign http://articles.latimes.com/1997-06-11/news/mn-2303_1_one-st... Besides which, you can also just rip the sign out of the ground or replace the stop sign with an actual speed limit sign. From a SDC perspective, this is kind of the equivalent of using super clever software vulnerabilities when you have physical access to an un-encrypted HDD. Impressive technical tour de force, but misses the forest for the trees if your goal is security. Traffic sign vandalism is a social problem with a social solution. The first person who tries something like this in production with malicious intent toward SDCs will be charged with some variation of attempted or actual manslaughter/murder and opponents of SDCs will switch to strategies other than domestic terrorism to advance their agenda. This research is still very interesting and useful, of course, but primarily for other reasons. I just don't see traffic sign vandalism intended to confuse DNNs as a serious -- or unique -- threat to self-driving cars.

cscurmudgeon 9y ago
I think there is a point here that is being overlooked. The vandalism, a couple of small stickers, would be less detectable compared with a whole missing sign. I can see such attack methods evolving to produce undetectetable (to humans without software) adversarial images.
- throwawayjava 9y ago
  I'm not saying we shouldn't practice defense in depth where it's possible. It's just that I don't think preventing this style of attack is actually the most relevant/impactful application of this style of research. Again, I'm not saying this research is irrelevant. I'm saying it's way more relevant than this particular example implies. > would be less detectable compared with a whole missing sign. This kind of assumes that the attack wouldn't work in the first place, no? If cars are crashing people are going to investigate to figure out why, and even the most thick-skulled beat cop can figure out that stop signs with rando weird ass stickers on them are probably worth running up the chain...
  - cscurmudgeon 9y ago
    > This kind of assumes that the attack wouldn't work in the first place, no? If cars are crashing people are going to investigate to figure out why, and even the most thick-skulled beat cop can figure out that stop signs with rando weird ass stickers on them are probably worth running up the chain... 1. That attack may not target only self-driving cars. 2. Imagine someone driving around with some random picture on their car at the back. Except that the picture is an attack.
pessimizer 9y ago
> Traffic sign vandalism is a social problem with a social solution. The first person who tries something like this in production with malicious intent toward SDCs will be charged with some variation of attempted or actual manslaughter/murder and opponents of SDCs will switch to strategies other than domestic terrorism to advance their agenda. It's tunnelvision to assume that the only attackers would be individual radical luddites. There do exist terrorist organizations and state actors who intend to run risks to do damage. And unless you're planning to monitor and respond quickly to all potential threats to all stop signs at all times (and all other possible attacks), what's to stop a group of 40 terrorists from covering their faces, stealing motorcycles, and putting 10,000 stickers on 10,000 signs in 10,000 intersections in 20 different cities in 1 night?

mholt 9y ago

A bigger concern in my mind is where 100% accuracy is needed, like when doing authentication (cf. any login screen that lets you in by looking at your face). A lot of experiments are done on visual data sets, but I don't see why these techniques need to be strictly for visual inputs. Is a machine learning model making decisions based on the results of its classifier? Then it is at risk of being fooled, and whatever decisions it's making, better be trivial ones. I wrote a document a few months ago that summarizes this and other risks that ML models currently face: https://matt.life/papers/security_privacy_neural_networks.pdf https://matt.life/papers/security_privacy_neural_networks.pd...