10 ms·

Slight Street Sign Modifications Can Fool Machine Learning Algorithms

194 points by itcrowd 9y ago

Rjevski 9y ago
The right solution to this is to have an official database of signs and their GPS coordinates provided by the government or whoever is responsible for road safety, free of charge (you're not directly paying for having signs on the road, why would you have to pay for an electronic version of that?). Road signs were made for humans because we don't have the ability to connect to the internet and fetch the data in less than a second, but autonomous card do, so why not use it?
- castis 9y ago
  Introducing a single point of failure (1 database, across the internet), organized by a largely complex system (the DoT, possibly) to cope with an edge case doesn't feel like a very elegant solution.
  - pimmen 9y ago
    A more costly, less elegant solution whose problem is already shared by street signs in general (meaning we could plan for it) would be RFID tags or something that tells the computer what sign this is if it can't read the sign. You could also use this for training, so it learns to filter away poorly drawn swastikas from the sign. An attack on something like this would scale very poorly, as you would need physical access to all street signs. Issuing RFID to a street sign would be just another step along the manufacturing process of the sign, or as a step to the mounting of the sign. I can't take credit for this idea though, it's already been [explored in a paper](https://link.springer.com/chapter/10.1007/978-3-642-41647-7_15 https://link.springer.com/chapter/10.1007/978-3-642-41647-7_...).
    - tyingq 9y ago
      >An attack on something like this would scale very poorly, as you would need physical access to all street signs I wouldn't underestimate the eventual legions of unemployed truck drivers. They will probably lose eventually, but I expect a spirited effort.
      - pimmen 9y ago
        They would be most successful where population and sign density is high, which would be the cities. During such an attack, public transport could move around people while the authorities would take care of the attackers. The problem for them is that where they would cause the most harm, rural areas (since they are in most dire need of supplies a few towns away and don't have any good alternatives to cars), is where such an attack would be the hardest to implement. Canada, Australia, Iceland and Alaska has many roads where vital street signs can be tens of miles apart from each other, as well any actual people who might be effected by this. Also, demographic movements is working to their disadvantage; more and more people everyday are moving to large cities.
        tyingq 9y ago
        In the US they can focus on just the interstates. And, they are unemployed, already used to long boring trips, and have established communication networks between them. Oh, and disenfranchised friends at the various rural truck stops and motels that will also be razed by self driving tech.
      - kbos87 9y ago
        This actually doesn't seem that far fetched. Ex-drivers who are angry and have plenty of time on their hands out at night picking the RFID tags out of stop signs. Another scenario is more of a slow and steady war. I'd expect that autonomous vehicles will occasionally be alone and unoccupied, which makes them a tempting and defenseless target. You wouldn't even need to do any physical damage to disable them - just a little electrical tape over their sensors, or some chocks under their tires. Soon, the autonomous trucking cos have to employ some of the ex-truck drivers to go around re-enabling their vehicles. And of course, those ex-truck drivers are in cahoots with the disbalers. That could certainly drive down their savings on labor costs.
  - dogma1138 9y ago
    This isn't a single point of failure by any stretch of the imagination. Waze and other navigation systems have a database of signs, road speeds and traffic enforcement cameras. I don't know about the US but there are places where signs are in a database already. This can be crowd sourced just like anything else. Google Street View already has a pretty good DB to start from.
- hk__2 9y ago
  > Road signs were made for humans because we don't have the ability to connect to the internet and fetch the data in less than a second, but autonomous car[s] do, so why not use it? Because you shouldn’t need to connect to the Internet and resolve GPS coordinates to know there’s a sign 50 meters away. If autonomous cars can’t "see" signs, let make them "see-able" with e.g. small radio transmitters along the road.
  - Rjevski 9y ago
    The issue is that upgrading every sign with a transmitter is costly, would require power and ongoing maintenance (not to mention vandalism - a sign is just metal and usually pretty resistant, but any electronics will be destroyed by vandals in no time). An online database that can be cached (so you aren't in trouble if your network drops while you go through a tunnel or similar) is much cheaper to implement than upgrading physical signs.
    - pimmen 9y ago
      Since the sign is just metal, it could serve as the antenna. Bury the transmitter beneath the ground, there you could also hook it up to the power grid (unless you use the sign's surface to act as a solar panel). I would like to see vandals mess with that. However, it ain't cheap and simple to implement, I agree with that. In countries like Sweden, Finland, Iceland, Canada and Australia they probably have the means to outfit their cities since they have the wealth to do it, but the rural areas would be a real challenge since they are so sparsely populated even when the money is there. In contrast to when these countries pioneered the implementation of the Internet, there's nothing like the phone network to piggy back this time.
    - dogma1138 9y ago
      Passive or Semi-Active RFID isn't expensive. All signs can be a victim of vandalsim. I think computer vision, a sign DB an IR water marked QR code and some RFID combined can produce a pretty robust and tamper reselient system.
    - mannykannot 9y ago
- unityByFreedom 9y ago
  > The right solution to this is to have an official database of signs and their GPS coordinates Why does everyone think self driving cars can use GPS to identify sign locations? GPS is not accurate enough to do that, and the labeling would need to be done by hand, which is not feasible. Plus, signs change and move around, making the labeling task endless.
- panic 9y ago
  One solution is to add redundancy. A stop sign, for example, could broadcast a low-power radio signal, and be accompanied by a retro-reflective strip with a coded pattern on the side of the road leading up to it, and emit tones at particular ultrasonic frequencies… GPS could be a part of this, but it's easy to imagine what could go wrong. Somebody mistakenly installs a stop sign without updating the database. Another person makes a mistake configuring a cache parameter and the CDN starts serving up last year's map. There's a whole class of problems eliminated by keeping the information local. Imagine if you had to drive looking only at road information served over the internet -- would you trust it?
- guftagu 9y ago
  What do you think will happen when it (inevitably) gets hacked?
fennecfoxen 9y ago
Related, but different: Autonomous Trap 001 http://jamesbridle.com/works/autonomous-trap-001 http://jamesbridle.com/works/autonomous-trap-001 They used salt to construct a circle with a solid line on the inside ("do not cross") and a dashed line on the outside ("come on in").
- DanBC 9y ago
  Everyone makes it sound like this art project was tried with a real autonomous car, but it hasn't been. This is just a guy sprinkling salt around his regular car.
  - hellbanner 9y ago
    So that was just him driving in and not an autonomous car? https://vimeo.com/208642358 https://vimeo.com/208642358 The car drives into the salt circle then stops.
    - mikeash 9y ago
      Yes. From the interview above: Is this actually an autonomous car, or is it conceptual? I don't actually have a self-driving car, unfortunately....
      - lima 9y ago
        We're talking about it on HN, so I'd say he did it exactly right :)
jameshart 9y ago
Slight street sign modifications can completely fool humans too. Luckily: 1) most people aren't assholes so don't want to cause traffic accidents; and 2) we have laws and police organizations to track down and punish assholes who do
- dogma1138 9y ago
  Slight sign modification will not fool humans at least not on this manner. This looks like a couple of bumper stickers will cause problems even when they do not obstruct the actual sign. This is a pretty big issue that will have to be dealt with.
  - Neliquat 9y ago
    Yup, In my town most signs have been creatively modified in some way. Gonna be a scary time. That said, I feel like most signs can safely be ignored if you already know the route, as avs would. Maybe construction signs should just have a specific blinking led pattern to override..
  - jameshart 9y ago
    Except that in a world where self-driving vehicles are relying on visual recognition of signage, it would become obvious PRETTY QUICKLY if the modifications to a STOP sign made it look like a 65mph speedlimit, and it would be able to be treated exactly as severely as if someone had covered the stop sign with a 65mph speedlimit sign. We're not talking about modifications causing a sign to accidentally be mistaken for something else, we are talking about deliberate modifications to road signs that cause vehicles to misinterpret them. If you modified a sign so that most users still perceived it as a stop sign but colorblind people misread it as a speed limit you'd be doing exactly the same kind of thing. I mean, it's not like there's some free-speech right to graffiti on road signage in the first place, let alone to modify it so that some road users will misunderstand the sign's meaning. If you interfere with a roadsign in order to deliberately confuse road users, you are a criminal.
pacaro 9y ago
I would assume that you could somewhat trivially defend against this by validating the top n hypotheses from your classifier against reference images.
cs702 9y ago
IN OTHER NEWS: Images with Zero Modifications Can Completely Fool Human Sight! Examples: http://www.ritsumei.ac.jp/~akitaoka/index-e.html http://www.ritsumei.ac.jp/~akitaoka/index-e.html (In case it's not obvious, the implication is that it's possible to engineer sign street modifications that fool human beings too.)
- dogma1138 9y ago
  And there is a reason why we design signs the way we do and not use optical illusions.
  - djmips 9y ago
    This is about fooling vision systems with adversarial modifications. Everyone needs to get out of themselves and see that the human vision systems can be fooled with adversarial mods as well, just not the same bugs as computer vision...
dilemma 9y ago
Sounds like Artificial Intelligence is Stupid.
- visarga 9y ago
  It's not fair to compare with AI. In this case, a visual network is more like a reflex. They can fool the equivalent of a human reflex. The vision neural net feeds into a "world model" where such inconsistencies are resolved on a more abstract level. The same world model is being used to plan the path of the car. Even if the vision net makes an error, the internal model has ways to detect that there's a perception error if it doesn't make sense in the context.
- userbinator 9y ago
  ...stupid like a newborn baby: in very early development, but gradually learning. Personally, I'm not sure that AI will ever be as intelligent as a human, but the possibility is still somewhat uncomfortable.
dilemma 9y ago
Sounds like Artificial Intelligence is Stupid.
ragebol 9y ago
Yet humans are totally unphased by these modifications. What makes humans still excel at these edge cases?
- wolfgke 9y ago
  > What makes humans still excel at these edge cases? It is rather that we built the signs in a way such that they are very robust for the human visual system to detect.
- taneq 9y ago
  I was pondering on this topic earlier today and came to the conclusion, "because our performance is judged only by humans." Imagine if an alien visited Earth, and was completely dumbfounded by our sketches and cartoon drawings. "But it looks nothing like a chicken", they say, looking at a sketch of a chicken and a photo of a chicken. "It's just an edge case that happens to trigger your 'chicken' neuron."
- acomjean 9y ago
  Our brains are pretty good at image processing. Plus we have an inherent understanding the world, including the way objects rotate and change shape. We're looking for the stop sign (big red hexegon). Though I've seen a new stop sign added recently, and the number of cars that don't see it is remarkable. I'm not an expert on AI though ( a few classed at university and I specialized in something else). Its the general classifiers that give us trouble. We want to show it 100s of stop sign pictures and have it figure out when we show it new one. But we're not asking it, is this a stop sign, we're asking "what is this". We can write software that probably is good at finding hexagons and colors and thus stop signs. Take facial recognition, its remarkably good at this point, but its doing one thing ( though I've seen a computer id 3 people in a photo with 2 in it , because a third person's photo was in the background.)
  - userbinator 9y ago
    We're looking for the stop sign (big red hexegon). Though I've seen a new stop sign added recently, and the number of cars that don't see it is remarkable. These two sentences together show that our brains aren't just looking for the signs; we're also looking at many other aspects of the situation and even taking into account past experience (e.g. is this an intersection? Have I seen a stop sign here before? If I'm new to this area, I'm likely going to be far more alert to the signage.) If someone planted a (non-modified) stop sign on the side of a highway, where the road is completely straight and with no intersection, I bet some drivers won't even see it, those who do will be puzzled, and approximately none of them will even try to stop.
bitL 9y ago
Alright, so this is an image augmentation problem. Another training set with white noise variations, random unrelated pixellation/overlay texts can solve this. Simply your training wasn't general enough.
- visarga 9y ago
  Constructing adversarial examples is a sophisticated task, but it's being done.
  - bitL 9y ago
    Sure, and it should go hand in hand with appropriate image augmentation. Maybe make something like a GAN that would try to generate anti-obfuscation image augmentation for every new adversarial example...? Our eyes can be fooled easily anyway.
mannykannot 9y ago
Whenever this sort of issue comes up, there are a number of responses pointing out that humans can be fooled too, and I am guessing that the generally unstated implication is that therefore it is not a big deal. Two important differences that this point of view overlooks are that humans are usually able to tell when they are not fully understanding their visual input, and they are able to respond appropriately, which includes acting cautiously and taking actions that will help resolve the uncertainty. Artificial visual systems that have, at best, only a rudimentary understanding of what they are looking at, are in no position to act this way, and can assign high confidence values to what we would regard as ludicrous interpretations of the scene. These things will be fixed in time. There is nothing to be gained by pretending that they are not problems.
- visarga 9y ago
  > Artificial visual systems that have, at best, only a rudimentary understanding of what they are looking at, are in no position to act this way, and can assign high confidence values to what we would regard as ludicrous interpretations of the scene. It's not a good idea to generalize like that. There are many AI systems that use neural nets for perception and then use different models, such as conditional random fields (CRF) to see if entities make sense in context. Not to mention that these adversarial examples don't work well from all distances and angles. As a car changes position slightly, the adversarial example disappears. It's very hard to create an adversarial example that works from all angles on all networks. There is also a special training regimen that fortifies neural nets against adversarial examples. The problem of adversarial images is closely related to GANs and has been brought into attention by the same researcher, Ian Goodfellow. So it's not a neglected problem - some of the top minds are on it.
  - dogma1138 9y ago
    The adversarial examples don't need to work from all distances and angles. Many street signs especially in rural areas might have a pretty small visibility window and they will only be visible from a pretty fixed angle until you pass them. So while you might not be able to develop an adversarial example that works for all signs from all distances and angles it wouldn't be hard to develop one for a specific sign. The angle for that sign is known, the distance in which most cars could resolve the sign is also known, the average speed on the road can also be taken into account which gives you a rough idea of the time span the autonomous driver would have to identify and resolve the sign. Heck if you know the algorithm it wouldn't be that hard to build an app that basically allows you to take a picture of the sign with your phone from the road, select a car or subset of cars you want to jam and it would generate a likely pattern to apply to the sign to jam those cars. I should also add that in this case some of the examples that do seem to work look to be of the shape and size of common bumper stickers, this is pretty worrisome since these are not that uncommon on street signs, especially in rural areas where they are more or less at reachable heights. I also wonder how well do these systems deal with graffiti.
infruset 9y ago
Couldn't slightly randomizing the image before inputting it to the neural network invalidate such manipulations?
ape4 9y ago
Some context knowledge would help. For example, approaching an intersection... should I stop here? Oh, yah, there is a distorted red sign. Also, figuring out the type of sign for the outline. Then the icon inside seems like an approach that could work.
panic 9y ago
The road environment is designed for people to see and understand. Without a human-emulating "general AI", it's unlikely that every sign and surface marking out there will be legible to a machine. IMO, the whole premise of training these machines to see roads like a human is flawed. Their developers should be working with government agencies (like the FHWA in the US) to create new standards that are aware of the capabilities of the machine.
- FLUX-YOU 9y ago
  The machine evolves too quickly right now for standards to be worth it. Let the craze die down before doing something like that. Even if we intentionally supported machine drivers with special signs, beacons, or other things, those are still going to be subject to interference and defacing.
- mrtksn 9y ago
  So before transportation there were no roads, then Humans begun using animals and the roads were formed to accommodate animal based transport. Then Animal powered mechanical transport(carriages) was invented and people made roads to accommodate the new means of transport. Then the fully mechanical transport came and we just made roads for these machines. For some reason, this time Humans are trying to keep the roads the same. Well, actually not, there are project to create roads designed for self driving vehicles. I think those in the software industry have a hammer and now see everything as a nail. What a successful AI transportation system would be? Probably based on specifically designed infrastructure that is using the AI tech that never worked for the old school roads.
- hacker_9 9y ago
  I agree and I think the whole self driving push should be incremental anyway: 1. To begin with it is only used on motorways/highways as they are long straight roads, where the AI can take over the boring part and the driver can be left to take over if something complex happens. 2. Shared AI maps across all cars that can route everyone to/from work. As this is shared, it can balance the load and result in the least amount of traffic jams. Humans still do the driving. 3. The government sets up AI cameras that monitor city routes, and cars can tap into this information as they drive. The benefit here would be seeing the road from all angles, as well as massive computers bigger than cars doing all the number crunching. Over time more and more city routes could become approved to be fully driven by AI. That's my approach to an incremental design at least.
smoyer 9y ago
Because CCD based cameras see so much more IR than we do, you could probably make a set of IR reflective/absorptive stickers that could be placed on street signs that would be virtually invisible to humans but totally alter what the CCD sees. OT: My favorite street sign graffiti is the hula-hoop stickers that artists put on pedestrian walking signs.
hellbanner 9y ago
https://news.ycombinator.com/item?id=14883900 https://news.ycombinator.com/item?id=14883900 "We'll just cover all of the possible use cases" - Self Driving Car Engineer
whipoodle 9y ago
Sorry I'm late, boss. Someone put a sticker on a street sign so my car drove into a bakery.
thanatropism 9y ago
I want to know now how I can paint my face slightly to fool Facebook's (or some other widespread) facial recognition system.
- userbinator 9y ago
  Perhaps not "slightly" but there's this: https://cvdazzle.com/ https://cvdazzle.com/ To other humans, those are very much recognisable humans, but face detectors won't think they are.
nine_k 9y ago
What makes me wonder is how slight modifications make the algorithm miss gross, highly visible features, e.g mistake a blue sign for a red sign, or an upward-pointing triangle for a downward-pointing. I suspect it won't be very hard to make the algorithm pay more attention to it, by specially teaching it to tell between such differences, and maybe by running several separate networks taught to tell apart particular narrow features, not complete signs.
- s_kilk 9y ago
  It seems to indicate that these machines are nowhere near as smart as they appear to be based on earlier successes. I'm starting to get worried about the possibility of another AI Winter if it turns out that reality and hype are too far apart.
  - jgalt212 9y ago
    The AI Winter will come only when the investment money runs out. As the Fed is keen to raise rates as slow as humanly possible (similar to 2004, and unlike 1994), I think the AI practitioners still have a quite a bit of runway left.
- deleted 9y ago
  [deleted]
windlessstorm 9y ago
Future humans only street.
geophile 9y ago
This article says that the training set was small. And presumably the misclassified images were unlike images in the training set. If the training set includes images intended to mislead, then wouldn't the classifier then be more tolerant?
JackFr 9y ago
I would also hope that the car uses contextual geolocation info. That a speed limit sign is not posted on the corner of a 4-way intersection, and a stop sign is not typically put on the side of a limited access highway. In fact I would expect that most of the driving regulations should be encoded in the map. Anomalies would be treated with extreme caution (and reported back to the home office).
- tomelders 9y ago
  I would expect that some of the more basic rules win out. e.g. Don't crash into another vehicle. Do hit a pedestrian etc etc.
thedevil 9y ago
In my state, there's a (fast) road that turns down under a bridge. From a distance, it kind of looks like I'm going to crash into the bridge. Even though I know the road wouldn't drive directly into the bridge, I slow down a little and look carefully to make sure I'm actually not going to crash into the bridge. When my perception doesn't fit my internal model, I gather more data (look at different parts of the bridge and what other cars are doing), or transform the data (ie turn my head slightly and look at the bridge and road from different angles) Edit: Likewise, when someone's tone doesn't match their words, I gather more data (look at their body language). Have any researchers experimented with neural nets to do the same? I haven't noticed any posts here about that.
- moyix 9y ago
  There was a paper that claimed those sorts of rotations and movements can help alleviate adversarial examples: https://arxiv.org/abs/1707.03501 https://arxiv.org/abs/1707.03501 However, OpenAI quickly refuted it by creating adversarial examples that continue to fool the classifier even when rotated, scaled, etc: https://blog.openai.com/robust-adversarial-inputs/ https://blog.openai.com/robust-adversarial-inputs/ So it looks like there's no "easy" way out here. Multiple types of sensors may help, but it seems likely that it will still be possible to construct examples that fool network over all sensor inputs at once. Ian Goodfellow and Nicolas Papernot have a good blog on machine learning security issues. One relevant post on why this is such a hard problem: http://www.cleverhans.io/security/privacy/ml/2017/02/15/why-attacking-machine-learning-is-easier-than-defending-it.html http://www.cleverhans.io/security/privacy/ml/2017/02/15/why-...
  - pishpash 9y ago
    Is it a big surprise that you can construct adversarial examples for algorithms? Don't humans have the same class of problems with optical illusions? And those are not even adversarial, just confusing. If we constructed truly adversarial examples for human neurology, I bet they would be equally insane.
    - blauditore 9y ago
roryisok 9y ago
The first thing my mind went to was the image of muttley putting up a detour sign
pishpash 9y ago
If you had a diversity of differently trained algorithms, they would not admit the same sorts of adversarial examples. The risk isn't that adversarial examples exist -- an exponentially small number of them always exist with any representation. The risk is that you can search for them en masse like offline cracking of passwords. If you could do that with the human mind, I hesitate to think what you would find.
- brownbat 9y ago
  Cognitive biases, optical illusions probably.
kumarvvr 9y ago
Sometimes it seems that NN systems and Deep learning systems are a dead end. Are there any other promising technologies that can replace or at-least augment current Machine Learning systems?
drawkbox 9y ago
I have wondered this about mirrors and LIDAR, sounds like it is being worked on [1] [1] http://ieeexplore.ieee.org/document/5409636/ http://ieeexplore.ieee.org/document/5409636/
mishraka 9y ago
This is the price we all pay for lazily defining a complex cognitive task as a mere image <-> label problem.
pteredactyl 9y ago
I don't understand the use of 'adverserial attack.' Makes it sounds like the machines, or the creators of the machines are at war. When in reality a stop sign sticker adds texture to a mostly mundane public realm.
- pishpash 9y ago
  Adversarial is a technical term.
craigyk 9y ago
The first time I learned of adversarial research my initial thought was GD, in our automated AI-driven future, the people who master this are going to live like wizards.
- resu_nimda 9y ago
  You might really like https://cvdazzle.com/ https://cvdazzle.com/, they at least look the part!
yters 9y ago
Give a general machine learning algorithm a long enough string of 1s with a few missing, and it will have no clue what fills the missing digits. ML is inherently incapable of matching human cognition.

194 points by itcrowd 9y ago

The right solution to this is to have an official database of signs and their GPS coordinates provided by the government or whoever is responsible for road safety, free of charge (you're not directly paying for having signs on the road, why would you have to pay for an electronic version of that?). Road signs were made for humans because we don't have the ability to connect to the internet and fetch the data in less than a second, but autonomous card do, so why not use it?

castis 9y ago
Introducing a single point of failure (1 database, across the internet), organized by a largely complex system (the DoT, possibly) to cope with an edge case doesn't feel like a very elegant solution.
- pimmen 9y ago
  A more costly, less elegant solution whose problem is already shared by street signs in general (meaning we could plan for it) would be RFID tags or something that tells the computer what sign this is if it can't read the sign. You could also use this for training, so it learns to filter away poorly drawn swastikas from the sign. An attack on something like this would scale very poorly, as you would need physical access to all street signs. Issuing RFID to a street sign would be just another step along the manufacturing process of the sign, or as a step to the mounting of the sign. I can't take credit for this idea though, it's already been [explored in a paper](https://link.springer.com/chapter/10.1007/978-3-642-41647-7_15 https://link.springer.com/chapter/10.1007/978-3-642-41647-7_...).
  - tyingq 9y ago
    >An attack on something like this would scale very poorly, as you would need physical access to all street signs I wouldn't underestimate the eventual legions of unemployed truck drivers. They will probably lose eventually, but I expect a spirited effort.
    - pimmen 9y ago
      They would be most successful where population and sign density is high, which would be the cities. During such an attack, public transport could move around people while the authorities would take care of the attackers. The problem for them is that where they would cause the most harm, rural areas (since they are in most dire need of supplies a few towns away and don't have any good alternatives to cars), is where such an attack would be the hardest to implement. Canada, Australia, Iceland and Alaska has many roads where vital street signs can be tens of miles apart from each other, as well any actual people who might be effected by this. Also, demographic movements is working to their disadvantage; more and more people everyday are moving to large cities.
      - tyingq 9y ago
        In the US they can focus on just the interstates. And, they are unemployed, already used to long boring trips, and have established communication networks between them. Oh, and disenfranchised friends at the various rural truck stops and motels that will also be razed by self driving tech.
    - kbos87 9y ago
      This actually doesn't seem that far fetched. Ex-drivers who are angry and have plenty of time on their hands out at night picking the RFID tags out of stop signs. Another scenario is more of a slow and steady war. I'd expect that autonomous vehicles will occasionally be alone and unoccupied, which makes them a tempting and defenseless target. You wouldn't even need to do any physical damage to disable them - just a little electrical tape over their sensors, or some chocks under their tires. Soon, the autonomous trucking cos have to employ some of the ex-truck drivers to go around re-enabling their vehicles. And of course, those ex-truck drivers are in cahoots with the disbalers. That could certainly drive down their savings on labor costs.
- dogma1138 9y ago
  This isn't a single point of failure by any stretch of the imagination. Waze and other navigation systems have a database of signs, road speeds and traffic enforcement cameras. I don't know about the US but there are places where signs are in a database already. This can be crowd sourced just like anything else. Google Street View already has a pretty good DB to start from.
hk__2 9y ago
> Road signs were made for humans because we don't have the ability to connect to the internet and fetch the data in less than a second, but autonomous car[s] do, so why not use it? Because you shouldn’t need to connect to the Internet and resolve GPS coordinates to know there’s a sign 50 meters away. If autonomous cars can’t "see" signs, let make them "see-able" with e.g. small radio transmitters along the road.
- Rjevski 9y ago
  The issue is that upgrading every sign with a transmitter is costly, would require power and ongoing maintenance (not to mention vandalism - a sign is just metal and usually pretty resistant, but any electronics will be destroyed by vandals in no time). An online database that can be cached (so you aren't in trouble if your network drops while you go through a tunnel or similar) is much cheaper to implement than upgrading physical signs.
  - pimmen 9y ago
    Since the sign is just metal, it could serve as the antenna. Bury the transmitter beneath the ground, there you could also hook it up to the power grid (unless you use the sign's surface to act as a solar panel). I would like to see vandals mess with that. However, it ain't cheap and simple to implement, I agree with that. In countries like Sweden, Finland, Iceland, Canada and Australia they probably have the means to outfit their cities since they have the wealth to do it, but the rural areas would be a real challenge since they are so sparsely populated even when the money is there. In contrast to when these countries pioneered the implementation of the Internet, there's nothing like the phone network to piggy back this time.
  - dogma1138 9y ago
    Passive or Semi-Active RFID isn't expensive. All signs can be a victim of vandalsim. I think computer vision, a sign DB an IR water marked QR code and some RFID combined can produce a pretty robust and tamper reselient system.
  - mannykannot 9y ago
unityByFreedom 9y ago
> The right solution to this is to have an official database of signs and their GPS coordinates Why does everyone think self driving cars can use GPS to identify sign locations? GPS is not accurate enough to do that, and the labeling would need to be done by hand, which is not feasible. Plus, signs change and move around, making the labeling task endless.
panic 9y ago
One solution is to add redundancy. A stop sign, for example, could broadcast a low-power radio signal, and be accompanied by a retro-reflective strip with a coded pattern on the side of the road leading up to it, and emit tones at particular ultrasonic frequencies… GPS could be a part of this, but it's easy to imagine what could go wrong. Somebody mistakenly installs a stop sign without updating the database. Another person makes a mistake configuring a cache parameter and the CDN starts serving up last year's map. There's a whole class of problems eliminated by keeping the information local. Imagine if you had to drive looking only at road information served over the internet -- would you trust it?
guftagu 9y ago
What do you think will happen when it (inevitably) gets hacked?

fennecfoxen 9y ago

Related, but different: Autonomous Trap 001 http://jamesbridle.com/works/autonomous-trap-001 http://jamesbridle.com/works/autonomous-trap-001 They used salt to construct a circle with a solid line on the inside ("do not cross") and a dashed line on the outside ("come on in").

DanBC 9y ago
Everyone makes it sound like this art project was tried with a real autonomous car, but it hasn't been. This is just a guy sprinkling salt around his regular car.
- hellbanner 9y ago
  So that was just him driving in and not an autonomous car? https://vimeo.com/208642358 https://vimeo.com/208642358 The car drives into the salt circle then stops.
  - mikeash 9y ago
    Yes. From the interview above: Is this actually an autonomous car, or is it conceptual? I don't actually have a self-driving car, unfortunately....
    - lima 9y ago
      We're talking about it on HN, so I'd say he did it exactly right :)

jameshart 9y ago

Slight street sign modifications can completely fool humans too. Luckily: 1) most people aren't assholes so don't want to cause traffic accidents; and 2) we have laws and police organizations to track down and punish assholes who do

dogma1138 9y ago
Slight sign modification will not fool humans at least not on this manner. This looks like a couple of bumper stickers will cause problems even when they do not obstruct the actual sign. This is a pretty big issue that will have to be dealt with.
- Neliquat 9y ago
  Yup, In my town most signs have been creatively modified in some way. Gonna be a scary time. That said, I feel like most signs can safely be ignored if you already know the route, as avs would. Maybe construction signs should just have a specific blinking led pattern to override..
- jameshart 9y ago
  Except that in a world where self-driving vehicles are relying on visual recognition of signage, it would become obvious PRETTY QUICKLY if the modifications to a STOP sign made it look like a 65mph speedlimit, and it would be able to be treated exactly as severely as if someone had covered the stop sign with a 65mph speedlimit sign. We're not talking about modifications causing a sign to accidentally be mistaken for something else, we are talking about deliberate modifications to road signs that cause vehicles to misinterpret them. If you modified a sign so that most users still perceived it as a stop sign but colorblind people misread it as a speed limit you'd be doing exactly the same kind of thing. I mean, it's not like there's some free-speech right to graffiti on road signage in the first place, let alone to modify it so that some road users will misunderstand the sign's meaning. If you interfere with a roadsign in order to deliberately confuse road users, you are a criminal.

pacaro 9y ago

I would assume that you could somewhat trivially defend against this by validating the top n hypotheses from your classifier against reference images.

cs702 9y ago

IN OTHER NEWS: Images with Zero Modifications Can Completely Fool Human Sight! Examples: http://www.ritsumei.ac.jp/~akitaoka/index-e.html http://www.ritsumei.ac.jp/~akitaoka/index-e.html (In case it's not obvious, the implication is that it's possible to engineer sign street modifications that fool human beings too.)

dogma1138 9y ago
And there is a reason why we design signs the way we do and not use optical illusions.
- djmips 9y ago
  This is about fooling vision systems with adversarial modifications. Everyone needs to get out of themselves and see that the human vision systems can be fooled with adversarial mods as well, just not the same bugs as computer vision...

dilemma 9y ago

Sounds like Artificial Intelligence is Stupid.

visarga 9y ago
It's not fair to compare with AI. In this case, a visual network is more like a reflex. They can fool the equivalent of a human reflex. The vision neural net feeds into a "world model" where such inconsistencies are resolved on a more abstract level. The same world model is being used to plan the path of the car. Even if the vision net makes an error, the internal model has ways to detect that there's a perception error if it doesn't make sense in the context.
userbinator 9y ago
...stupid like a newborn baby: in very early development, but gradually learning. Personally, I'm not sure that AI will ever be as intelligent as a human, but the possibility is still somewhat uncomfortable.

dilemma 9y ago

Sounds like Artificial Intelligence is Stupid.

ragebol 9y ago

Yet humans are totally unphased by these modifications. What makes humans still excel at these edge cases?

wolfgke 9y ago
> What makes humans still excel at these edge cases? It is rather that we built the signs in a way such that they are very robust for the human visual system to detect.
taneq 9y ago
I was pondering on this topic earlier today and came to the conclusion, "because our performance is judged only by humans." Imagine if an alien visited Earth, and was completely dumbfounded by our sketches and cartoon drawings. "But it looks nothing like a chicken", they say, looking at a sketch of a chicken and a photo of a chicken. "It's just an edge case that happens to trigger your 'chicken' neuron."
acomjean 9y ago
Our brains are pretty good at image processing. Plus we have an inherent understanding the world, including the way objects rotate and change shape. We're looking for the stop sign (big red hexegon). Though I've seen a new stop sign added recently, and the number of cars that don't see it is remarkable. I'm not an expert on AI though ( a few classed at university and I specialized in something else). Its the general classifiers that give us trouble. We want to show it 100s of stop sign pictures and have it figure out when we show it new one. But we're not asking it, is this a stop sign, we're asking "what is this". We can write software that probably is good at finding hexagons and colors and thus stop signs. Take facial recognition, its remarkably good at this point, but its doing one thing ( though I've seen a computer id 3 people in a photo with 2 in it , because a third person's photo was in the background.)
- userbinator 9y ago
  We're looking for the stop sign (big red hexegon). Though I've seen a new stop sign added recently, and the number of cars that don't see it is remarkable. These two sentences together show that our brains aren't just looking for the signs; we're also looking at many other aspects of the situation and even taking into account past experience (e.g. is this an intersection? Have I seen a stop sign here before? If I'm new to this area, I'm likely going to be far more alert to the signage.) If someone planted a (non-modified) stop sign on the side of a highway, where the road is completely straight and with no intersection, I bet some drivers won't even see it, those who do will be puzzled, and approximately none of them will even try to stop.

bitL 9y ago

Alright, so this is an image augmentation problem. Another training set with white noise variations, random unrelated pixellation/overlay texts can solve this. Simply your training wasn't general enough.

visarga 9y ago
Constructing adversarial examples is a sophisticated task, but it's being done.
- bitL 9y ago
  Sure, and it should go hand in hand with appropriate image augmentation. Maybe make something like a GAN that would try to generate anti-obfuscation image augmentation for every new adversarial example...? Our eyes can be fooled easily anyway.

mannykannot 9y ago

Whenever this sort of issue comes up, there are a number of responses pointing out that humans can be fooled too, and I am guessing that the generally unstated implication is that therefore it is not a big deal. Two important differences that this point of view overlooks are that humans are usually able to tell when they are not fully understanding their visual input, and they are able to respond appropriately, which includes acting cautiously and taking actions that will help resolve the uncertainty. Artificial visual systems that have, at best, only a rudimentary understanding of what they are looking at, are in no position to act this way, and can assign high confidence values to what we would regard as ludicrous interpretations of the scene. These things will be fixed in time. There is nothing to be gained by pretending that they are not problems.

visarga 9y ago
> Artificial visual systems that have, at best, only a rudimentary understanding of what they are looking at, are in no position to act this way, and can assign high confidence values to what we would regard as ludicrous interpretations of the scene. It's not a good idea to generalize like that. There are many AI systems that use neural nets for perception and then use different models, such as conditional random fields (CRF) to see if entities make sense in context. Not to mention that these adversarial examples don't work well from all distances and angles. As a car changes position slightly, the adversarial example disappears. It's very hard to create an adversarial example that works from all angles on all networks. There is also a special training regimen that fortifies neural nets against adversarial examples. The problem of adversarial images is closely related to GANs and has been brought into attention by the same researcher, Ian Goodfellow. So it's not a neglected problem - some of the top minds are on it.
- dogma1138 9y ago
  The adversarial examples don't need to work from all distances and angles. Many street signs especially in rural areas might have a pretty small visibility window and they will only be visible from a pretty fixed angle until you pass them. So while you might not be able to develop an adversarial example that works for all signs from all distances and angles it wouldn't be hard to develop one for a specific sign. The angle for that sign is known, the distance in which most cars could resolve the sign is also known, the average speed on the road can also be taken into account which gives you a rough idea of the time span the autonomous driver would have to identify and resolve the sign. Heck if you know the algorithm it wouldn't be that hard to build an app that basically allows you to take a picture of the sign with your phone from the road, select a car or subset of cars you want to jam and it would generate a likely pattern to apply to the sign to jam those cars. I should also add that in this case some of the examples that do seem to work look to be of the shape and size of common bumper stickers, this is pretty worrisome since these are not that uncommon on street signs, especially in rural areas where they are more or less at reachable heights. I also wonder how well do these systems deal with graffiti.

infruset 9y ago

Couldn't slightly randomizing the image before inputting it to the neural network invalidate such manipulations?

ape4 9y ago

Some context knowledge would help. For example, approaching an intersection... should I stop here? Oh, yah, there is a distorted red sign. Also, figuring out the type of sign for the outline. Then the icon inside seems like an approach that could work.

panic 9y ago

The road environment is designed for people to see and understand. Without a human-emulating "general AI", it's unlikely that every sign and surface marking out there will be legible to a machine. IMO, the whole premise of training these machines to see roads like a human is flawed. Their developers should be working with government agencies (like the FHWA in the US) to create new standards that are aware of the capabilities of the machine.

FLUX-YOU 9y ago
The machine evolves too quickly right now for standards to be worth it. Let the craze die down before doing something like that. Even if we intentionally supported machine drivers with special signs, beacons, or other things, those are still going to be subject to interference and defacing.
mrtksn 9y ago
So before transportation there were no roads, then Humans begun using animals and the roads were formed to accommodate animal based transport. Then Animal powered mechanical transport(carriages) was invented and people made roads to accommodate the new means of transport. Then the fully mechanical transport came and we just made roads for these machines. For some reason, this time Humans are trying to keep the roads the same. Well, actually not, there are project to create roads designed for self driving vehicles. I think those in the software industry have a hammer and now see everything as a nail. What a successful AI transportation system would be? Probably based on specifically designed infrastructure that is using the AI tech that never worked for the old school roads.
hacker_9 9y ago
I agree and I think the whole self driving push should be incremental anyway: 1. To begin with it is only used on motorways/highways as they are long straight roads, where the AI can take over the boring part and the driver can be left to take over if something complex happens. 2. Shared AI maps across all cars that can route everyone to/from work. As this is shared, it can balance the load and result in the least amount of traffic jams. Humans still do the driving. 3. The government sets up AI cameras that monitor city routes, and cars can tap into this information as they drive. The benefit here would be seeing the road from all angles, as well as massive computers bigger than cars doing all the number crunching. Over time more and more city routes could become approved to be fully driven by AI. That's my approach to an incremental design at least.

smoyer 9y ago

Because CCD based cameras see so much more IR than we do, you could probably make a set of IR reflective/absorptive stickers that could be placed on street signs that would be virtually invisible to humans but totally alter what the CCD sees. OT: My favorite street sign graffiti is the hula-hoop stickers that artists put on pedestrian walking signs.

hellbanner 9y ago

https://news.ycombinator.com/item?id=14883900 https://news.ycombinator.com/item?id=14883900 "We'll just cover all of the possible use cases" - Self Driving Car Engineer

whipoodle 9y ago

Sorry I'm late, boss. Someone put a sticker on a street sign so my car drove into a bakery.

thanatropism 9y ago

I want to know now how I can paint my face slightly to fool Facebook's (or some other widespread) facial recognition system.

userbinator 9y ago
Perhaps not "slightly" but there's this: https://cvdazzle.com/ https://cvdazzle.com/ To other humans, those are very much recognisable humans, but face detectors won't think they are.

nine_k 9y ago

What makes me wonder is how slight modifications make the algorithm miss gross, highly visible features, e.g mistake a blue sign for a red sign, or an upward-pointing triangle for a downward-pointing. I suspect it won't be very hard to make the algorithm pay more attention to it, by specially teaching it to tell between such differences, and maybe by running several separate networks taught to tell apart particular narrow features, not complete signs.

s_kilk 9y ago
It seems to indicate that these machines are nowhere near as smart as they appear to be based on earlier successes. I'm starting to get worried about the possibility of another AI Winter if it turns out that reality and hype are too far apart.
- jgalt212 9y ago
  The AI Winter will come only when the investment money runs out. As the Fed is keen to raise rates as slow as humanly possible (similar to 2004, and unlike 1994), I think the AI practitioners still have a quite a bit of runway left.
deleted 9y ago
[deleted]

windlessstorm 9y ago

Future humans only street.

geophile 9y ago

This article says that the training set was small. And presumably the misclassified images were unlike images in the training set. If the training set includes images intended to mislead, then wouldn't the classifier then be more tolerant?

JackFr 9y ago

I would also hope that the car uses contextual geolocation info. That a speed limit sign is not posted on the corner of a 4-way intersection, and a stop sign is not typically put on the side of a limited access highway. In fact I would expect that most of the driving regulations should be encoded in the map. Anomalies would be treated with extreme caution (and reported back to the home office).

tomelders 9y ago
I would expect that some of the more basic rules win out. e.g. Don't crash into another vehicle. Do hit a pedestrian etc etc.

thedevil 9y ago

In my state, there's a (fast) road that turns down under a bridge. From a distance, it kind of looks like I'm going to crash into the bridge. Even though I know the road wouldn't drive directly into the bridge, I slow down a little and look carefully to make sure I'm actually not going to crash into the bridge. When my perception doesn't fit my internal model, I gather more data (look at different parts of the bridge and what other cars are doing), or transform the data (ie turn my head slightly and look at the bridge and road from different angles) Edit: Likewise, when someone's tone doesn't match their words, I gather more data (look at their body language). Have any researchers experimented with neural nets to do the same? I haven't noticed any posts here about that.

moyix 9y ago
There was a paper that claimed those sorts of rotations and movements can help alleviate adversarial examples: https://arxiv.org/abs/1707.03501 https://arxiv.org/abs/1707.03501 However, OpenAI quickly refuted it by creating adversarial examples that continue to fool the classifier even when rotated, scaled, etc: https://blog.openai.com/robust-adversarial-inputs/ https://blog.openai.com/robust-adversarial-inputs/ So it looks like there's no "easy" way out here. Multiple types of sensors may help, but it seems likely that it will still be possible to construct examples that fool network over all sensor inputs at once. Ian Goodfellow and Nicolas Papernot have a good blog on machine learning security issues. One relevant post on why this is such a hard problem: http://www.cleverhans.io/security/privacy/ml/2017/02/15/why-attacking-machine-learning-is-easier-than-defending-it.html http://www.cleverhans.io/security/privacy/ml/2017/02/15/why-...
- pishpash 9y ago
  Is it a big surprise that you can construct adversarial examples for algorithms? Don't humans have the same class of problems with optical illusions? And those are not even adversarial, just confusing. If we constructed truly adversarial examples for human neurology, I bet they would be equally insane.
  - blauditore 9y ago

roryisok 9y ago

The first thing my mind went to was the image of muttley putting up a detour sign

pishpash 9y ago

If you had a diversity of differently trained algorithms, they would not admit the same sorts of adversarial examples. The risk isn't that adversarial examples exist -- an exponentially small number of them always exist with any representation. The risk is that you can search for them en masse like offline cracking of passwords. If you could do that with the human mind, I hesitate to think what you would find.

brownbat 9y ago
Cognitive biases, optical illusions probably.

kumarvvr 9y ago

Sometimes it seems that NN systems and Deep learning systems are a dead end. Are there any other promising technologies that can replace or at-least augment current Machine Learning systems?

drawkbox 9y ago

I have wondered this about mirrors and LIDAR, sounds like it is being worked on [1] [1] http://ieeexplore.ieee.org/document/5409636/ http://ieeexplore.ieee.org/document/5409636/

mishraka 9y ago

This is the price we all pay for lazily defining a complex cognitive task as a mere image <-> label problem.

pteredactyl 9y ago

I don't understand the use of 'adverserial attack.' Makes it sounds like the machines, or the creators of the machines are at war. When in reality a stop sign sticker adds texture to a mostly mundane public realm.

pishpash 9y ago
Adversarial is a technical term.

craigyk 9y ago

The first time I learned of adversarial research my initial thought was GD, in our automated AI-driven future, the people who master this are going to live like wizards.

resu_nimda 9y ago
You might really like https://cvdazzle.com/ https://cvdazzle.com/, they at least look the part!

yters 9y ago

Give a general machine learning algorithm a long enough string of 1s with a few missing, and it will have no clue what fills the missing digits. ML is inherently incapable of matching human cognition.