8 ms·
Optical adversarial attack can change the meaning of road signs
- mschuster91 5y agoAcademically certainly interesting, but everything I've seen on adversarial attacks against ML models relies on knowing the model or having a way to run the model against attacker-determined input. How would one go and attack a real-world vehicle? Essentially you'd need to gut a vehicle entirely, mock every sensor and actuator to fool the car's control unit to believe it is still driving a car... a lot of effort.
- XorNot 5y agoDynamos for vehicle performance are common, and GPS spoofing is not hard. In a world where AI driving systems are a couple of decades old and on the used car market now, obtaining the source hardware would be very straight forward.
- mschuster91 5y agoI'd seriously hope that any AI self-drive would not just use GPS and motor load sensors to learn its place in the world. You'd also need to spoof accelerometers, gyroscopes, multiple cameras and in case of everything except Tesla also LIDAR, and that convincing enough that no fail-safe is triggered.
- deleted 5y ago[deleted]
- elihu 5y ago> everything I've seen on adversarial attacks against ML models relies on knowing the model or having a way to run the model against attacker-determined input I thought that actually turned out not to be the case quite a bit of the time. An attack that's trained against one model will often work against different models as well, because the various models, even if trained on different data, will often end up looking at similar features. I'm not an ML expert though, so maybe someone with more knowledge can chime in.
- asdff 5y agoOr, you could just pull a page from intelligence agencies. All you would need to do is bribe or radicalize a disgruntled employee, then you'd know the model.
- XorNot 5y agoThe problem with most of these types of "attacks" is that they're about the same as any other messing with roadsigns or what not. There's no end of things you could do around the roads to cause crashes, but they're all illegal and ultimately low impact. This is a lot more interesting though from how it highlights the difference between current machine interpretation and human interpretation of road signs: which is to say, explainable AI is still incredibly important - the ability to slightly discolor the signs and have a blackbox AI read something entirely different means the systems and trainers we have are not cueing off a "safe" set of inputs to deliver their interpretations. From that context, this is very interesting and important work - and I would argue also points the way to at least some basic road safety standards for self-driving systems (they should be resistant to this sort of non-human perceptible visual distortion).
- jcims 5y agoI do wonder if there could be some kind of 'adversarial training' of large networks, like those of Tesla, in which drivers in an area collude to normalize dangerous behavior in an area (e.g. drive in oncoming lane, stop at an overpass on a major highway, etc). Even with humans in the loop in the training process, there's only so much context they can bring to override data that appears authentic.
- asdff 5y agoI'd me more worried about adversarial employees. Tesla has a reputation of working people to the bone for not much comp compared to the competition. I can't imagine they are connecting their stressed workers to psychologists. If one engineer just cracked one day, they could potentially push an update to all cars that does something terrible.
- jcims 5y agoSure. I'd guess someone could also sneak an easter egg in there to cause the battery packs to self-immolate.
- hirundo 5y ago
- aaron695 5y agoStill stupid. This following article wouldn't have happened if the vehicle was automated, it would have seen something was wrong, and also already have the correct signage from it's database - 3 Are Sentenced to 15 Years in Fatal Stop Sign Prank https://www.washingtonpost.com/archive/politics/1997/06/21/3-who-stole-traffic-signs-sentenced-to-15-years/14c0a68c-e8fd-42ba-881c-454f442a27f8/ https://www.washingtonpost.com/archive/politics/1997/06/21/3... What does need to stop is moving vehicles having uncovered street signs on them, if it's not in place cover it up.
- ec109685 5y agoAs an aside, those defendants were eventually exonerated: https://www.law.umich.edu/special/exoneration/Pages/casedetail.aspx?caseid=4362 https://www.law.umich.edu/special/exoneration/Pages/casedeta...
- aaron695 5y agoI like these totally contradictory witness statements. Why you can't trust witness's and the article doesn't seem to even notice - "At a hearing on the motion, several witnesses testified that they had driven by the intersection days before the accident and the stop sign was already down" "A man who was driving to a luncheon the day before the crash testified that as he pulled up to the intersection, he saw a semi-truck backing up across the road toward the sign. He said that when he came back after lunch, the stop sign was on the ground."
- smoldesu 5y agoThis title doesn't even begin to scratch the surface of what this is capable of. A much more viable (and damaging) attack would be encoding "invisible" data onto a widely-shared photo, and then manipulating it to trip the CSAM filters for the respective platforms. These attacks are designed to be invisible in real-world applications, but I can imagine a similar implementation designed to be invisible to humans (eg. encoding important data as two seemingly identical colors, or manipulating dither patterns). Stuff like this is why we can't trust AI. That doesn't mean we can't use it, but most of our models are black-boxes that can only be "debugged" as much as a pizza can be "decooked"
- Causality1 5y agoSomething I didn't find terribly clear: did the researchers manage to fool the software into seeing a shape other than an octagon, or is the software so bad it thinks an octagonal sign could be anything other than a stop sign?
- deleted 5y ago[deleted]
- morpheos137 5y agothe latter. The software doesn't think. That is why neural networks are dumb unless allied with expert systems. As a matter of fact that is more or less how human cognition works: the fusion of a pattern recognition system with multimodal sensory inputs filtered through an expert system looking for meaning in what is sensed. See for example https://slazebni.cs.illinois.edu/fall18/lec12_adversarial.pdf https://slazebni.cs.illinois.edu/fall18/lec12_adversarial.pd...
- jdavis703 5y ago> that is more or less how human cognition works Yes, but pranksters frequently vandalize stop signs with messages like “stop driving” or “stop eating animals” and human drivers are smart enough to figure out what to do. I’ve never seen someone throw their car into park or toss their burger out the window upon seeing a confusing stop sign.
- vntok 5y ago> I’ve never seen someone throw their car into park or toss their burger out the window upon seeing a confusing stop sign. If you ever travelled by plane, surely you've seen people at the airport trash their perfectly edible food (snacks, etc.) upon seeing the confusing signs just before the safety checks, even though in many cases they would have been allowed to pass it through the gates.
- SuchAnonMuchWow 5y agoThere are cases where pranksters painted a road on a wall, and drivers were fooled by it. So it definitely happens with humans too. We are just too much used to the existence of optical illusions for humans that we fail to consider them a possibility for attacks. Adversarial attacks on neural networks are simply that: optical illusions, but designed for neural networks instead of humans.
- newpavlov 5y agoRealistic impact of such issues is usually quite exaggerated: https://xkcd.com/1958/ https://xkcd.com/1958/ I think that any massively deployed true autopilot system will maintain a global semantic map of roads. Most sudden traffic sign changes, especially those which may influence car behavior, will be manually verified and committed to the global map. Any consistent discrepancies between the model and reality will be investigated as well, especially if it causes customer complaints. And it can be followed by releasing map patches telling software to prefer the model over detection for the problematic sign. Considering that at this stage such maps will be probably integrated with government services (i.e. the map will be compared against official road data), I think such attacks on the autopilot systems will be found quite fast.
- barryvan 5y agoHow would that work where streets signs _do_ change suddenly? A couple of examples I see every day are roadworks (new fixed signage for the duration of the works, or even a person flipping a sign between "stop" and "slow") and "smart" freeways/signage (screen-based speed limit signs that change, and have additional signs for speed-limits if all screens are off). Here in AU, we also have "school zones" with speed limit signs that turn on only during certain times and dates. I'm not sure that governments would be quick enough to keep this "central roadmap" correct up-to-the-minute (to be honest, I suspect that "up-to-the-month" would be a stretch!).
- newpavlov 5y agoI would assume that "smart" signs would transmit their state via Vehicle-to-Infrastructure (V2I) network, same for roadworks (i.e. workers would mark affected zone and this state will be transmitted via a mobile battery-powered beacon). Such transmitted states will be signed, with road authority key in the root of the certificate tree, so it will be hard to spoof them (at the very least, it will be easy to identify responsible for key leaking). Of course, it assumes a proper digitalization of infrastructure and appropriate regulation changes, which is not given... But in my opinion massive adoption of full self-driving cars will be difficult without them. >Here in AU, we also have "school zones" with speed limit signs that turn on only during certain times and dates. Such conditions are usually fixed and do not suddenly change. Even OSM handles them today more or less fine.
- skybrian 5y agoIn case you’re thinking you’ve seen this sort of research before, the difference is that they don’t touch the sign, just illuminate it. I’m not sure how much a difference that makes in practice?
- abrookewood 5y agoIt makes a massive difference. To the human eye, the altered sign is not really distinguishable from the normal one and the attackers don't need to approach the sign in order to alter it. It makes a remote attack scenario quite achievable (depending on signal strength required for the laser etc). Think people camped out in a hotel room over looking a freeway and causing Telsas all to crash.
- skybrian 5y agoModifying street signs isn’t actually hard, though, and whoever did it could leave right away. I’m not sure how noticeable to humans the modifications need to be? A minor disadvantage is that it leaves evidence. The hotel room thing seems like a lot more work than quickly done vandalism, and I’m wondering how close to the sign they would need to be.
- chaircher 5y agoTo use a very clunky metaphor I suppose it's like the difference between gun crime and knife crime.
- kgeist 5y agoWhat about adding signed QR codes to road signs? It's practically impossible to forge those, it doesnt't require complex vision systems, and you don't have to revamp your whole road infrastucture, just gradually replace existing signs.
- ragebol 5y agoThe reason for reading the existing traffic signs is that they already exist. If you would design roads for autonomous cars, you'd replace the signs with eg. radio beacons and the like.
- mnd999 5y agoLet’s do that - on highways where it’s worth the investment. And let’s stop pretending that self-driving is in any way viable on normal roads. Outside of that relatively controlled environment, there are too many inputs and the reasoning requires is too complex.
- delusional 5y agoI think you'd agree that putting QR codes on signs is cheaper and less intrusive than replacing them with radio beacons.
- ragebol 5y agoI do agree that is would be cheaper, but they're ugly and quite intrusive as I can't read them.
- BoxOfRain 5y agoYeah I don't know what it is about QR codes but I'm really not a fan of the aesthetic they have. There's something cheap, mechanistic, institutional about them that I can't quite put my finger on.
- asah 5y agoclever! and it reuses the existing (low) tech of painted signage.
- keiferski 5y agoThis reminded me of Operation Greif, a German covert mission during WW2. https://en.m.wikipedia.org/wiki/Operation_Greif https://en.m.wikipedia.org/wiki/Operation_Greif ...German soldiers, wearing captured British and U.S. Army uniforms and using captured Allied vehicles, were to cause confusion in the rear of the Allied line. Reconnaissance patrols of three or four men were to reconnoiter on both sides of the Meuse river and also pass on bogus orders to any U.S. units they met, reverse road signs, remove minefield warnings, and cordon off roads with warnings of nonexistent mines. As a result, U.S. troops began asking other soldiers questions that they felt only Americans would know the answers to in order to flush out the German infiltrators, which included naming certain states' capitals, sports and trivia questions related to the U.S., etc. This practice resulted in U.S. brigadier general Bruce Clarke being held at gunpoint for some time after he incorrectly said the Chicago Cubs were in the American League[7][8][9][10] and a captain spending a week in detention after he was caught wearing German boots. General Omar Bradley was repeatedly stopped in his staff car by checkpoint guards who seemed to enjoy asking him such questions. Is there a programmatic equivalent of asking what league the Chicago Cubs are in?
- nightcracker 5y agoWhen used to filter out computers from humans such a procedure is known as a Turing test.
- deleted 5y ago[deleted]
- sokoloff 5y agoSee also CAPTCHA (a somewhat forced acronym of “Completely Automated Public Turing test to tell Computers and Humans Apart”)
- an_ko 5y agoI'd you mean "question that sounds basic, but which many professional programmers wouldn't be able to answer at gunpoint": How do prefix and suffix increment operators (++x vs x++) differ in functionality?
- ajsnigrutin 5y agoI've never driven a tesla or any other car with "autopilot" features.. theoretically, if someone was to 'borrow' a 20km/h sign from somewhere, and take it to a 130km/h highway, and attach it to a random spot, would the car slow down, until the driver intervened? Or would it "guess" there's something wrong with such a drastic speed change, and beep/ignore/alert the driver?
- watermelon0 5y agoIf you have a car with speed sign recognition, and activated cruise control, it would definitely slow down, albeit slowly.
- tyfon 5y agoLuckily the tesla autopilot ignores speed limits on the high-way, it only follows it (when on auto pilot) on slower roads. There have however been issues with e-trons and id.X cars reading side street signs here in Norway and braking hard on the highway.
- dkdbejwi383 5y agoIs a stop sign really the same shape and colour as a speed limit sign in the USA? In the UK they are a different shape and colour, you don't need the text to know a STOP sign is what it is.
- dagw 5y agoIs a stop sign really the same shape and colour as a speed limit sign in the USA? No, but some of these sign reading algorithms probably don't take color and shape into account. A naive implementation might first checks the sign for something that can be OCRed, and if it finds that that is a 2 digit number that matches known speed limits then it's a speed limit.
- dkdbejwi383 5y agoSeems like a bad idea in hindsight! Any system like this should be considering multiple factors, including the shape and size, as well as evaluating the behaviour of other drivers to see if the perceived instruction seems reasonable. After all, that's what we do as humans.
- dagw 5y agoOne of the core problems with machine learning is that we often don't really know why the model does what it does. We train the neural network on 100000 images of road sign, 5000 of which are stop signs and we know it will identify stop signs, but we don't know what it is that makes the algorithm think something actually is a Stop sign. Is it the color, the shape, the word "Stop"? The training set had red octagonal signs and signs with numbers, but no red octagonal signs with numbers. Since the the model had never seen a sign with just two big digits that wasn't a speed limit sign it decides that the numbers was more significant than the shape or color and went with that. What this shows is how important it is to include intentionally adversarial data in your training set if you want your model to be robust in 'unexpected' situations.