4 ms·
Fooling Production-Grade Classification Systems with Adversarial Traffic Signs [pdf]
- lolc 7y agoWow I've read about adversarial images before but didn't think attacks would be this practical. It's weird to think that the added smudges have any relevance on the result when there is so much contrast. It appears easy to us because we're constantly training, cross-checking, and adjusting to conditions without much conscious interference. At some point, not now, we will accept what the machine says. Machines will be better (faster and more accurate) at number recognition than us. But this paper shows how far behind they are still. In the context of cars I think the machines will just refuse to drive like we do. They (really their builders) will demand much better controlled conditions to operate safely.
- quickthrower2 7y agoWe can still be fooled by sirnple things. Whats a siRNple?
- mannykannot 7y agoDealing with typos is the sort of robustness that people have, but that these image recognition programs are not displaying.
- kuu 7y agoJust if someone is interested, here is the link to the "landing" page, with the metadata and abstract of the paper: https://arxiv.org/abs/1907.00374 https://arxiv.org/abs/1907.00374
- b_tterc_p 7y agohttps://m.xkcd.com/1958/ https://m.xkcd.com/1958/ Not quite the same problem but the result is pretty much the same. The fact that computer vision can be fooled by attacks like these are lower risk than we might imagine. The more important question is what is the likelihood that such conditions occur naturally and randomly. Edit: also, the summary comment about this being the first time a physical object was used to fool a commercial classification system sounds... dubious...
- dexen 7y agoThe XKCD you linked is reasonable on the surface of it, but if you mull over it a bit more, you'll see the three glaring omissions: 1) people care much, much less about "hurting" automated systems, and 2) people feel much less responsibility for accidents & failures if they are separated by a level of indirection of automation 3) people feel much less responsible for mass actions that affect anonymous people far away Together, those three factors will indeed cause more incidents of people attempting to fool the self-driving cars. Especially if done by displaying malicious images via mass devices, like the light-up info boards, electronic billboards etc.. I don't expect much more actual accidents, tho.
- deleted 7y ago[deleted]
- empath75 7y agoAlso, people aren't generally fooled by bad road lines.
- b_tterc_p 7y agoPerhaps not. And they certainly aren’t fooled by adversarial speed signs. But they are fooled by plenty of other things which are equally easy to implement and equally dangerous.
- b_tterc_p 7y agoWhile I don’t disagree with any of your points, there are still going to be people in cars, and any attempt to cause them to behave erratically could easily become attempted murder charges. I wouldn’t expect this to be any more common than people throwing rocks at cars on the highway.
- bb123 7y agoMakes me think that there is still a long way to go here. All of the changes they made to the signs leave them still easily readable by humans. One could see how reflections, weather and other environmental factors would distort a road sign more than this. Speed limit signs are possibly the hardest example though because it requires the system to recognise the exact number on it, rather than just its existence. I wonder how much harder it would be to get it to misclassify a stop sign?
- FeepingCreature 7y agoThe road sign has to be modified in the specific manner of the attack; not just any distortion will do.
- Merrill 7y agoThe training set for the classifiers must not contain real-world traffic signs which are weathered, rusted, peeled, bent, smudged, and spray paint tagged.
- logfromblammo 7y agoDon't forget perforated by shotguns. And stickered. And those ones you sometimes find in malls, apartment complexes, or office parks that are an architectural standard sign with the traffic control sign symbol painted on one side, so you can't identify what the sign is from other angles. I hate those.
- Retric 7y agoThis seems like a meaningless attack. If you’re modifying a sign, you can simply replace it with any other sign. What exactly is the utility?
- yters 7y agoYes, just think about a fleet of autonomous trucks carrying loads of valuable merchandise. A couple of these roadsigns that normal human cops would not detect, and you can redirect the entire fleet to your lair of doom.
- en-us 7y agoThat's assuming the vehicle doesn't have other means of locating itself, such as GPS.
- shacharm 7y agoAn attacker could very well use a GPS jammer. Treating the navigation problem as a graph problem, placing these jammers where's needed in order to make the truck to follow the signs alone. Your line of thinking is really good. A well-crafted attack is hard to execute and little things can very well ruin an attack sequence. A subtle the difference between academic research and the real world.
- mannykannot 7y agoThe threat aspect of this is, indeed, largely a red herring. The relevance here is the current lack of robustness in the face of ambiguity. The line between images that are properly identified and those that are not is very complicated, not well characterized, and (to some extent[1]) implementation-dependent. One might say that this is true of people too, but people have various higher-level skills for detecting and resolving ambiguity. [1] As this study found adversarial images that affect a number of different implementations, there is some implementation-independence, which may be helpful in solving the problem.
- theon144 7y agoThe near-invisibility of the attack to a layman's (human) eye. I would think this kind of attack would take a non-negligible time to discover, unlike simply replacing a sign which is obvious to any random passerby.
- theon144 7y agoNice! I really appreciate the real-world testing and not just "proving" the NN can be fooled by running it against image files. This always seemed to me like a huge omission in reports about adversarial attacks, sure it can be fooled with pixel-perfect changes of input data, but would it survive literally any "real-world"-like environment? I am quite surprised it does!