6 ms·

> For example, consider the case of pasting a sticker on a speed limit sign that causes Teslas to swerve off the road. If your vision system can be caused to s

> For example, consider the case of pasting a sticker on a speed limit sign that causes Teslas to swerve off the road. If your vision system can be caused to swerve off a road by a sticker then maybe it shouldn't be used?

mgraczyk 5y ago
I bet I could cause a significant fraction of human vision systems to get in a crash with a well placed sticker. I'd replace "<- One Way" with "Detour ->".
- pixelgeek 5y ago
  But that is not a vision issue. That is providing people with incorrect information.
  - qw501428 5y ago
    Perhaps a mirror/reflective sticker that blinds drivers near a sharp curve?
  - b3morales 5y ago
    The specific trick doesn't really matter; the point is that it's possible to maliciously create a situation that makes human pilots act dangerously. We accept that the possibility can't be made nil, and we have post facto rules to deal with it. The same principle applies to traps for machines.
    - ClumsyPilot 5y ago
      Nope, it 0.1% of humans crash but 100% of teslas crash that's not 'the same'
      - MauranKilom 5y ago
        So would it be better/fine if Tesla randomized the CV ML models to some extent? I actually feel the answer is probably "yes", surprisingly...
- siboehm 5y ago
  This has always confused me as well. What would be the reason why some adversary would choose to craft an adversarial example and deploy it in the real world versus the much easier solution to just remove / obscure the sign?
  - tonyarkles 5y ago
    Depending on how big or small it needs to be, potentially for subtlety? Especially on current roads that are shared by humans and self-driving systems, a human observer will immediately notice that something is terribly wrong with a replaced sign. But... around here at least, signs have stickers or graffiti on them often enough. Like adding the name of a politician under a stop sign: "Stop [Harper]". An appropriately made adversarial example won't stick out visually the same way that a wholesale sign swap will.
  - laura_g 5y ago
    Because NeurIPS doesn't publish papers on stop sign removal yet :P
  - pueblito 5y ago
    Warfare comes to mind, as weapons gain increasingly powerful ai functions and become autonomous
- marcosdumay 5y ago
  I'm pretty sure this would fail to kill people on almost every place you could try it. And if it works somewhere, it's because there are other problems with the road that should be fixed. Human driving is full of redundancies, and there is a clear hierarchy of information. People will not rush into a road full of cars going on the other way, it doesn't matter what the signs say. If your automated driving system doesn't have those same features, it's not ready for use.
  - Ozzie_osman 5y ago
    You put too much faith in humans. Things like stop sign removal have caused deaths in the past. https://www.nytimes.com/1997/06/21/us/3-are-sentenced-to-15-years-in-fatal-stop-sign-prank.html https://www.nytimes.com/1997/06/21/us/3-are-sentenced-to-15-...
    - Vetch 5y ago
      Humans are not so bad as drivers. Your example is an event from over 2 decades ago and was deemed newsworthy. Humans drive in all kinds of conditions but death rate is about 1 per 100 million miles driven. A search reveals crashes to be on the order of hundreds of collisions per 100 million miles driven. Age, country, intoxication level, road design and laws, road and environmental conditions also play a major role such that accident rates for someone aged 30+ in a Northern European country are going to be a lot less than teenagers in a country where road laws are merely friendly suggestions (and considering the chaos of driving in those countries, the rates are actually surprisingly low).
      - catlikesshrimp 5y ago
        I will go further and say that almost everytime there is an accident the driver is somehow impaired. Lack of sleep, drugs, illness (old age included, mental disease), poor judgement (young age included, emotional distress) Humans are surprisingly good at driving under normal conditions.
        5y ago
- jstanley 5y ago
  I think in almost all cases that would not cause a crash. The drivers would see the oncoming traffic and stop rather than crash.
  - gmadsen 5y ago
    that assumes you can see the threat, if instead it led to an unprotected crossing at high speed, then you have a very different situation
- beerandt 5y ago
  There are multiple reasons for signs to have different shapes, sizes, and colors, and this is one of them. An orange diamond "detour" sign isn't easily confused for a smaller rectangle "one way" sign. Additionally, there should always be two large "do not enter" plus two large red "wrong way" signs that are visible to a driver from in the intersection before turning. Something as simple as tape or other coverings on an existing sign should never result in any confusion as to right-of-way for a driver paying attention.
  - rictic 5y ago
    Some people key off the shape enough that they wouldn't follow a wrongly-shaped detour sign, so you wouldn't fool everyone, but you'd absolutely fool a lot of people. I expect I'd be one of them.
- mannykannot 5y ago
  These attempts to imply a broad equivalence between current machine vision and human capabilities do not hold up under a modicum of scrutiny. Humans have well-developed models of how things should be, can detect when things seem wrong, and come up with ways to address the apparent anomaly (including taking steps to investigate and evaluate the situation.) Humans do not always use these capabilities well, but they have them, while similar capabilities are at best rudimentary and fragile in current AI. The premise of this article is that these capabilities will not come easy.
- jancsika 5y ago
  I see this kind of argument with blockchain bros as well, and it drives me nuts. If I write crappy paint program and all I can claim is, "It's no worse than the time/effort of drawing by hand," what exactly have I achieved in your opinion? And if the posts on HN wrt blockchain and ML constantly feature these "no-worse-than-what-we-are-replacing" arguments while posts about, say, paint programs don't, what does that say about the buzz around blockchain and ML? Edit: clarification
- glitchc 5y ago
  I bet you can't. Humans are anti-fragile and can compensate with other knowledge.
GistNoesis 5y ago
Turn a temporary road sign for 30 speed-limit into a 80 speed-limit with some black-tape (I have already seen it done when people were angry to be fined by speed-detector for a few excess km/h, (or just for the lulz) ). It probably won't fool humans, but it's an edge case that a self-driving car may ignore.

mgraczyk 5y ago

I bet I could cause a significant fraction of human vision systems to get in a crash with a well placed sticker. I'd replace "<- One Way" with "Detour ->".

pixelgeek 5y ago
But that is not a vision issue. That is providing people with incorrect information.
- qw501428 5y ago
  Perhaps a mirror/reflective sticker that blinds drivers near a sharp curve?
- b3morales 5y ago
  The specific trick doesn't really matter; the point is that it's possible to maliciously create a situation that makes human pilots act dangerously. We accept that the possibility can't be made nil, and we have post facto rules to deal with it. The same principle applies to traps for machines.
  - ClumsyPilot 5y ago
    Nope, it 0.1% of humans crash but 100% of teslas crash that's not 'the same'
    - MauranKilom 5y ago
      So would it be better/fine if Tesla randomized the CV ML models to some extent? I actually feel the answer is probably "yes", surprisingly...
siboehm 5y ago
This has always confused me as well. What would be the reason why some adversary would choose to craft an adversarial example and deploy it in the real world versus the much easier solution to just remove / obscure the sign?
- tonyarkles 5y ago
  Depending on how big or small it needs to be, potentially for subtlety? Especially on current roads that are shared by humans and self-driving systems, a human observer will immediately notice that something is terribly wrong with a replaced sign. But... around here at least, signs have stickers or graffiti on them often enough. Like adding the name of a politician under a stop sign: "Stop [Harper]". An appropriately made adversarial example won't stick out visually the same way that a wholesale sign swap will.
- laura_g 5y ago
  Because NeurIPS doesn't publish papers on stop sign removal yet :P
- pueblito 5y ago
  Warfare comes to mind, as weapons gain increasingly powerful ai functions and become autonomous
marcosdumay 5y ago
I'm pretty sure this would fail to kill people on almost every place you could try it. And if it works somewhere, it's because there are other problems with the road that should be fixed. Human driving is full of redundancies, and there is a clear hierarchy of information. People will not rush into a road full of cars going on the other way, it doesn't matter what the signs say. If your automated driving system doesn't have those same features, it's not ready for use.
- Ozzie_osman 5y ago
  You put too much faith in humans. Things like stop sign removal have caused deaths in the past. https://www.nytimes.com/1997/06/21/us/3-are-sentenced-to-15-years-in-fatal-stop-sign-prank.html https://www.nytimes.com/1997/06/21/us/3-are-sentenced-to-15-...
  - Vetch 5y ago
    Humans are not so bad as drivers. Your example is an event from over 2 decades ago and was deemed newsworthy. Humans drive in all kinds of conditions but death rate is about 1 per 100 million miles driven. A search reveals crashes to be on the order of hundreds of collisions per 100 million miles driven. Age, country, intoxication level, road design and laws, road and environmental conditions also play a major role such that accident rates for someone aged 30+ in a Northern European country are going to be a lot less than teenagers in a country where road laws are merely friendly suggestions (and considering the chaos of driving in those countries, the rates are actually surprisingly low).
    - catlikesshrimp 5y ago
      I will go further and say that almost everytime there is an accident the driver is somehow impaired. Lack of sleep, drugs, illness (old age included, mental disease), poor judgement (young age included, emotional distress) Humans are surprisingly good at driving under normal conditions.
      - 5y ago
jstanley 5y ago
I think in almost all cases that would not cause a crash. The drivers would see the oncoming traffic and stop rather than crash.
- gmadsen 5y ago
  that assumes you can see the threat, if instead it led to an unprotected crossing at high speed, then you have a very different situation
beerandt 5y ago
There are multiple reasons for signs to have different shapes, sizes, and colors, and this is one of them. An orange diamond "detour" sign isn't easily confused for a smaller rectangle "one way" sign. Additionally, there should always be two large "do not enter" plus two large red "wrong way" signs that are visible to a driver from in the intersection before turning. Something as simple as tape or other coverings on an existing sign should never result in any confusion as to right-of-way for a driver paying attention.
- rictic 5y ago
  Some people key off the shape enough that they wouldn't follow a wrongly-shaped detour sign, so you wouldn't fool everyone, but you'd absolutely fool a lot of people. I expect I'd be one of them.
mannykannot 5y ago
These attempts to imply a broad equivalence between current machine vision and human capabilities do not hold up under a modicum of scrutiny. Humans have well-developed models of how things should be, can detect when things seem wrong, and come up with ways to address the apparent anomaly (including taking steps to investigate and evaluate the situation.) Humans do not always use these capabilities well, but they have them, while similar capabilities are at best rudimentary and fragile in current AI. The premise of this article is that these capabilities will not come easy.
jancsika 5y ago
I see this kind of argument with blockchain bros as well, and it drives me nuts. If I write crappy paint program and all I can claim is, "It's no worse than the time/effort of drawing by hand," what exactly have I achieved in your opinion? And if the posts on HN wrt blockchain and ML constantly feature these "no-worse-than-what-we-are-replacing" arguments while posts about, say, paint programs don't, what does that say about the buzz around blockchain and ML? Edit: clarification
glitchc 5y ago
I bet you can't. Humans are anti-fragile and can compensate with other knowledge.

GistNoesis 5y ago

Turn a temporary road sign for 30 speed-limit into a 80 speed-limit with some black-tape (I have already seen it done when people were angry to be fined by speed-detector for a few excess km/h, (or just for the lulz) ). It probably won't fool humans, but it's an edge case that a self-driving car may ignore.