6 ms·
This weakness is one that I think will plague self driving cars as sign recognition will be key and without some ability to insure that they cannot be dangerous
by ChuckMcM 9y ago
This weakness is one that I think will plague self driving cars as sign recognition will be key and without some ability to insure that they cannot be dangerously fooled, it will be hard to get them certified. The canonical example is to make a no left turn sign recognize as a no right turn sign and have the car go the wrong way on a one way street.
Clearly there is a marketing opportunity for t-shirts that make you recognize as other things. Who doesn't want to show up in an image search for toasters on Google Images ? :-)
But my current best guess on how this issue will be addressed will be with classifier diversity an voting systems. While that just moves the problem into a harder and harder to synthesize data set (something that not only is adversarial to one classifier but gives the same answer on several), I believe it will get us to the point where we can trust the level of work to defeat them is sufficiently hard to make it a non-threat.
- projectileboy 9y agoOne possible mitigation could be the use of geocoded roadway metadata. I rented a car in France that had this - it would let me know when I was exceeding the speed limit based on my position. Not hyper-accurate, and so probably not a replacement for all road signs, but certainly good enough for many situations.
- smogcutter 9y agoWas thinking of this too, but what happens when the map and "ground truth" disagree? The machine would need to be able to identify the rare circumstances when it should trust the map over its senses. If it's capable of that, this wouldn't be a problem in the first place.
- SomeStupidPoint 9y agoWhen driving, that probably doesn't really matter: If the map and road sign disagree, just do what everyone around you is doing (ie, go with the flow) until you recover agreement. It's not perfect, but stretches out attacks to having to compromise long stretches of road and multiple vehicle types. If the cars for the next mile somewhere can find pace, the whole group can; similarly, if any kind of car can find pace, they can all use that to set. Obviously safety checks and such, but human drivers only loosely follow signs and often get behavioral cues from other cars -- why can't automated cars do the same when they get confused?
- dTal 9y agoI am now picturing a mile-long phalanx of self-driving cars gaily careening off the end of a destroyed bridge because the map and road signs disagree and they're all "going with the flow". Or to put it another way, if all your friends drove off a bridge, would you?
- f137 9y agoWhat would you do if confused by the road signs? A car could 1) see what other drivers doing 2) ask them - and this is easies with v2v 3) ask some central authority owe the air
- Houshalter 9y agoI mean people can already do that to human drivers. You can easily remove a stop sign. Or put a sticker over an important sign. Any human driver will be fooled! Yet this doesn't seem to be a prevalent problem.
- ChuckMcM 9y agoAnd they have: http://www.nytimes.com/1997/06/21/us/3-are-sentenced-to-15-years-in-fatal-stop-sign-prank.html http://www.nytimes.com/1997/06/21/us/3-are-sentenced-to-15-y... It just makes it harder to catch who did it.
- Houshalter 9y agoIt's also harder to do. It requires technical knowledge and some planning, not some drunk teenagers.
- stonewhite 9y agoNot relevant to the discussion, but turns out these kids didn't steal those signs. https://www.law.umich.edu/special/exoneration/Pages/casedetail.aspx?caseid=4363 https://www.law.umich.edu/special/exoneration/Pages/casedeta...
- ChuckMcM 9y agoInteresting to read that follow up. The kids had stolen 21 signs, but not those. Thanks for the link.
- brudgers 9y agoBased on the history of artificial intelligence, I am less optimistic about the long term prospects of any silver bullet technology, and that's how neural networks and related systems are currently positioned. Ultimately, rule based systems are rule based systems, our inability to articulate the explicit rules should not be mistaken for their absence. Ten or one hundred or a thousand rule based classifiers voting is more of the same, not something different. The black box still has the same inputs and the same outputs no matter how many times the handle of the Pumping Lemma is raised and lowered.
- azernik 9y agoOr, simply with better machine learning. > The border between “truth” and “false” is almost linear. The first cool thing we can derive from it is that when you follow the gradient, once you find the area where the predicted class changes, you can be fairly confident that the attack is successful. On the other hand, it tells us that the structure the decision function is far simpler that most researchers thought it to be. Humans can be fooled by optical illusions as well; but those illusions are much more limited and much more noticeable than most of these. My (very non-specialist) interpretation of the italicize clause is that a) vulnerability to these attacks is a continuum, not a binary, and b) the current ease of these attacks reflects the crudity of our current ML techniques.
- ben_w 9y agoHabe you seen the Monroe-Einstein illusion? There is a continuous transition from one image to the other, depending on the angular size. This feels relevant to your point. https://static.independent.co.uk/s3fs-public/styles/article_small/public/thumbnails/image/2015/04/03/14/einstein.jpg https://static.independent.co.uk/s3fs-public/styles/article_...
- tambourine_man 9y agoI wouldn't call that an illusion, but more a consequence of a high-pass/low-pass filter at different distances with limited resolution. What I find much more disturbing is that even though I measured and known those lines are parallel or that those shades of gray are the same, I can't "unsee" the illusion.
- theptip 9y agoAgree, it seems to me that there's a sense in which this is a trivial issue; just throw more adversarial examples into the training set, and turn the crank until the classifier understands how to correctly handle those cases too. There's also a sense in which this is extremely non-trivial; any agent (human or machine) can be subjected to adversarial attacks. They just look different right now, and our current systems are vulnerable to very simple ones. It seems to me that improving an algorithm's resistance to adversarial attacks is much more feasible than improving a human's resistance to their own class of adversarial attacks.
- skybrian 9y agoAlternately, if you have multiple models that should normally give the same result, this can apparently be used to automatically find test cases where they differ: DeepXplore: automated whitebox testing of deep learning systems https://blog.acolyer.org/2017/11/01/deepxplore-automated-whitebox-testing-of-deep-learning-systems/ https://blog.acolyer.org/2017/11/01/deepxplore-automated-whi...
- candiodari 9y agoI have heard from the police in many places that humans have the same problem, and they have it in practice: There are places in almost every police zone where people driving misinterpret the situation almost perfectly consistently. Which means in some cases that the exact same accident keeps happening with a certain regularity, mostly just involving cars (and generally in that case nothing's done about it), but sometimes involving bikers or pedestrians and in those cases usually after a while the roads are changed so it stops happening. So here we are: 1) there are "adversarial attacks" on the human mind, in existing road situations 2) those attacks were built into our road network, by humans, and presumably accidentally 3) they systematically lead to accidents and cause deaths, mostly people who weren't even the human misinterpreting the situation. Also, a lot of economic damage is caused. 4) we do one of two things to fix it: a) we just accept this as a fact of life and don't care b) we change the road situation until we purely accidentally hit one that doesn't get misinterpreted. So how to deal with this for AI drivers ... I know ! How about the exact same way ? AI drivers, I've been stuck behind them in MTV traffic enough to know this, are far safer than good human drivers. They far exceed the ability average human drivers. And they wipe the floor with the very best humans where it comes to patience with fellow road users. Why do we hold them to a 100% standard ? Nobody and nothing, human or otherwise, matches up to a 100% perfect standard. A stick you use to beat a dog will not have a 100% success rate, once every 10 years or so that stick will break and maybe even injure the guy holding the stick by bouncing around, and yet somehow that is acceptable ... We need to talk about how good drivers need to be. 100% is simply not an acceptable answer.