13 ms·

As somebody who works on computer vision, my general take on these things is that adversarial examples are like poison. It would be fairly easy to add poison t

As somebody who works on computer vision, my general take on these things is that adversarial examples are like poison. It would be fairly easy to add poison to a water supply or the air intake of a large building and kill a large number of people. This rarely happens though. It's ok that water sources, buildings, and people aren't completely immune to poison. The safety requirement isn't that poison can't hurt. Instead, we rely on weaker protections. We try to make known poisons hard to make, we try to track people who could make them, and we try to make it hard to deliver poison. I believe the same will be true of adversarial examples for vision (and language) models. We can try to make them hard to make, hard to posses anonymously, and hard to deliver. I think this will be much easier with computer vision than with poison, so I'm not worried about it. For example, consider the case of pasting a sticker on a speed limit sign that causes Teslas to swerve off the road. Governments should protect people from this in multiple ways, similarly to how they protect us from poison: 1. People who post these stickers should go to prison. 2. People who create and distribute these stickers knowing their purpose should go to prison. 3. Tesla should be civilly liable for cases where preventing such an incident was possible with known technology. 4. Roads should be modified over time to make it more difficult to do this attack. I think some combination of the above would be enough to make society as comfortable with adversarial example risk as we are with poison risk.

deleted 5y ago
[deleted]
voakbasda 5y ago
Your proposed laws do not cut out any exemption for research and experimentation, either with existing systems or potential new ones. This level of regulation would create an impossibly high barrier to entry and ensure that only the established players would remain in the marketplace. The last thing that I want to see is yet more regulatory capture, particularly in an industry that has yet to establish a reasonable baseline of success.
- mgraczyk 5y ago
  None of the things I listed would affect research. Researchers shouldn't be posting these on public highways, and researchers shouldn't be distributing them with the intent to cause harm.
  - mrfox321 5y ago
    It would once the govt gets involved. It's like saying that weapons research is just a free-for-all. The amount of regulation is correlated with the potential harm to society. Look at drug research. There is plenty of red tape that hinders it. Although, here, the "harm to society" is defined by the nation state. However, I agree with your proposals in the top-level comment.
  - d110af5ccf 5y ago
    This is naive. > researchers shouldn't be distributing them with the intent to cause harm It could be used to cause harm and publishing your work is distributing it (by definition). Similar laws have been (and are) used to target people for unjust reasons. Defacing street signs is already illegal. Making various abstract drawings illegal to wear on your clothing (or print on a sticker) is a terrible idea. If the algorithm used by a product can't handle a pedestrian wearing a sweatshirt with an adversarial example on it then that product simply isn't suitable for public use.
- Laremere 5y ago
  INAL, but actually putting adversarial image attacks on real roads is already illegal. If you modify a street sign, and as a result someone dies, that's a fairly easy case of Involuntary manslaughter. At a minimum, you can't modify street signs. Eg in Washington State: RCW 47.36.130 Meddling with signs prohibited. No person shall without lawful authority attempt to or in fact _alter_, deface, injure, knock down, or remove any official traffic control signal, _traffic device_ or railroad sign or signal, or any inscription, shield, or insignia thereon, or any other part thereof. (Underscore emphasis added). And if you're thinking about not putting it on a sign, but putting it elsewhere visible to cars: RCW 46.61.075.1 Display of unauthorized signs, signals, or markings. No person shall place, maintain or display upon or in view of any highway any unauthorized sign, signal, _marking or device_ which purports to be or is an imitation of or resembles an official traffic-control device or railroad sign or signal, or _which attempts to direct the movement of traffic_, or which hides from view or interferes with the effectiveness of an official traffic-control device or any railroad sign or signal. Where I'm unsure is producing these with the intent or knowledge that they will/could be used by someone to go do this. None of this makes using these for research and experimentation illegal.
pixelgeek 5y ago
> For example, consider the case of pasting a sticker on a speed limit sign that causes Teslas to swerve off the road. If your vision system can be caused to swerve off a road by a sticker then maybe it shouldn't be used?
- mgraczyk 5y ago
  I bet I could cause a significant fraction of human vision systems to get in a crash with a well placed sticker. I'd replace "<- One Way" with "Detour ->".
  - pixelgeek 5y ago
    But that is not a vision issue. That is providing people with incorrect information.
    - qw501428 5y ago
      Perhaps a mirror/reflective sticker that blinds drivers near a sharp curve?
    - b3morales 5y ago
      The specific trick doesn't really matter; the point is that it's possible to maliciously create a situation that makes human pilots act dangerously. We accept that the possibility can't be made nil, and we have post facto rules to deal with it. The same principle applies to traps for machines.
      - ClumsyPilot 5y ago
        Nope, it 0.1% of humans crash but 100% of teslas crash that's not 'the same'
        MauranKilom 5y ago
        So would it be better/fine if Tesla randomized the CV ML models to some extent? I actually feel the answer is probably "yes", surprisingly...
jimbob45 5y ago
>It would be fairly easy to add poison to a water supply or the air intake of a large building and kill a large number of people. I used to think this until someone walked me through the logistics of both and made me realize that you would need an agency-alerting level of poison for the water supply and some way to avoid people just shutting off the A/C and sticking their heads out of windows (also a huge amount of gas). Also the news can't exist to alert anyone immediately.
- xboxnolifes 5y ago
  > agency-alerting level of poison Doesn't this fall under: "We try to make known poisons hard to make, we try to track people who could make them, and we try to make it hard to deliver poison"? And the rest of it being you should use something odorless / tasteless.
ggoo 5y ago
I think a factor that should also be considered in your analogy is that poison is much more difficult to attain than stickers. I have to imagine that if poison was cheaply and widely available as stickers, we'd have a much larger problem than we currently see.
- wcarey 5y ago
  Most households contain at least several poisons (and precursors to hazardous gasses as well) amongst their cleaning supplies.
  - deleted 5y ago
    [deleted]
ReleaseCandidat 5y ago
> We try to make known poisons hard to make, we try to track people who could make them, and we try to make it hard to deliver poison. Actually no. We know that only some psychopaths would do that and so the risk is minimal. AI is currently simply not 'good enough' to be used in critical environments. The problem is that _any_ sticker or even dirt or snow or ... on any road sign can lead to misinterpretation, you can never proof that it's safe.
- mgraczyk 5y ago
  Sometime iceberg lettuce kills people (salmonella). We have safety regulations and inspections to mitigate that, but you can never prove that iceberg lettuce is safe.
  - d110af5ccf 5y ago
    This is specious. I can take actions to mitigate food poisoning such as rinsing my vegetables prior to consumption. If a company claims that their leafy greens are already rinsed and safe to consume without further prep work when in fact they are not then presumably they can be held liable. The risks surrounding vegetable packaging and distribution are well understood, readily quantifiable, and possible to mitigate. Computer vision algorithms on the other hand are poorly understood black boxes with seemingly arbitrary failure modes. We do not (yet) appear to understand how to quantify or mitigate the associated risks. The consequences of failure are quite severe in comparison to food poisoning. Only a fool would trust their life to them.
aneutron 5y ago
What you are proposing are what I think would be called a security theater. It gives the illusion of security, but they would absolutely not deter a determined threat actor. The only reason that the water supply isn't poisoned is it's unpractical for a single person to conduct the whole exploit chain: Construct the poison in enough quantities, gain access to facilities supplying the water, and actually throwing the compound in it. It's unpractical even for "underground" types. Especially the quantities required. Mathematics and computer science is a different story in my opinion. You cannot restrict science or thought. You can try, but good luck. The most you can do is delay it. If there is an attack that enables someone to flip a Tesla on the road (as suggested below), the security theater will hide the attack from common folk, but determined actors will reach it eventually, and at that point, they can deploy it as they wish. And in contrast to the water plant, the logistical endeavor to exploit it is absolutely easy in comparison: slap a sticker on your vehicle. Security by obscurity or by theater is rarely a good strategy in my opinion. We should absolutely be transparent about these kind of things, and allow researchers full access to develop attacks against these systems, and effectively communicate when they are found.
- mgraczyk 5y ago
  Right, and the protections for poison are also security theater for the same reason. In the real world that's ok. > The only reason that the water supply isn't poisoned is it's unpractical for a single person to conduct the whole exploit chain It's a quantitative question, just like with computer vision. If you don't like the poison example, consider viral DNA, which is also dangerous in the right hands and does not require massive supply chain control. Not everyone has access to a driving dataset like Teslas, and it would be difficult to trick a Tesla without such a dataset. We should allow researches to develop attacks, just like we should allow researchers to study poisons, DNA, and viruses.
  - flerovium 5y ago
    The lab leak theory isn't necessarily true, but this analogy isn't as conclusive as you make it sound...
- 5y ago
seany 5y ago
1 and 2 are almost always going to be impossible in the US due to the first amendment (this is a feature not a bug) 3 doesn't seem crazy, but it would practically end up with caps, which might not be what you're looking for 4 This both: seems possible, and will basically never happen due to cost in every little jurisdiction
- eximius 5y ago
  Defacing property is not free speech...?
  - seany 5y ago
    Just stand in view holding the image on a poster board.
- hacoo 5y ago
  I doubt 1 would be protected by the first amendment. It’s arguably equivalent to spraying graffiti on a stop sign so it’s unrecognizable. It would be an extremely difficult to enforce though.
  - boomboomsubban 5y ago
    Graffiti would just cause people to drive unsafely, in that hypothetical the sticker directly causes crashes. It'd be something like attempted murder.
- mgraczyk 5y ago
  #1 is certainly not a first amendment violation. In fact, the supreme court still holds that certain restrictions on billboards are allowed even for the purpose of preserving beauty. Safety is a much more compelling interest than beauty, so I don't expect states and cities will lose their ability to regulate road signage. See Metromedia, Inc. v. San Diego for example. #2 is expensive and difficult, but that's what we do for explosives, poisons, drugs, etc.
  - 5y ago
version_five 5y ago
I lot of this has been touched on already, but I think your rules could be reframed a bit to try simplify lawmaking and avoid security theatre as was mentioned. First, I assume it's already illegal to be "adversarial" to drivers. A bright light or changing signs etc already do that now. For example look at all the laser pointer stuff with planes. Second, I don't think self driving cars are just using the softmax output of an object detector as a direct input to car control decisions. In the absence of a stop sign, the expected behavior would be common sense and caution, the same as if someone removed the sign. If the SDC logic is not robust in this way, it's not safe for many other reasons. With this in mind, I think the situation is probably already reasonable well covered in existing regulations.
ketzo 5y ago
Relevant XKCD: https://xkcd.com/1958/ https://xkcd.com/1958/ > I worry about self-driving car safety features. > What's to stop someone from painting fake lines on the road, or dropping a cutout of a pedestrian onto a highway, to make cars swerve and crash? > Except... those things would also work on human drivers. What's stopping people now? > Yeah, causing car crashes isn't hard. > I guess it's just that most people aren't murderers? > Oh, right, I always forget. > An underappreciated component of our road safety system.
- citilife 5y ago
  That's how I feel about most dangerous situations in general and I think the national news highlights one-off events in a way we historically were not used to. For instance, taking out the United States internet would probably only required 3-4 strategic bombings. I bring this up because Tennessee had one of those bombed Christmas last year -- https://www.theverge.com/2020/12/28/22202822/att-outage-nashville-christmas-bombing https://www.theverge.com/2020/12/28/22202822/att-outage-nash... > This brought down wireless and wired networks across parts of Tennessee, Kentucky, and Alabama Most people aren't all that concerned about doing damage. Keep people happy and generally you don't have crime.
mnd999 5y ago
That worked super well with DVD CSS right? Let’s face it, people are going to print adversarial images on t-shirts.
- MikeHolman 5y ago
  If the adversarial image is intended to cause car accidents or bodily harm in some way, then the people printing the t-shirts and the people wearing them are already breaking the law. And if they actually do hurt someone, I imagine they would be criminally liable.
petermcneeley 5y ago
I am pretty sure that deliberately tricking an automated system into causing bodily harm is already coved by existing law. Think of all the automated systems that have existed before ML.
indymike 5y ago
> People who post these stickers should go to prison Doing things with intent to harm others is illegal, even if you use a sticker to do it. > Tesla should be civilly liable for cases where preventing such an incident was possible with known technology. This is currently likely the case, but is not proven until a lawsuit happens.
glitchc 5y ago
Adversarial examples don't confuse people, only algorithms. Perhaps you need to face the fact that if the CV algorithm fails against these examples when humans don't, then the CV algorithm is too brittle and should not be used in the real world. I don't trust my life to your "It kinda looks like a road, oh wait it's a pylon, I've been tricked, BAM!" dumpster fire of an algorithm. We used to have to craft robustness into algorithms based on the false positive rate. Nobody looks at a CFAR style approach anymore, and it shows. The state of the art approach of pinning everything on ML is a dead-end for CV.
- mabbo 5y ago
  > if the CV algorithm fails against these examples when humans don't, then the CV algorithm is too brittle and should not be used in the real world. This is the tricky bit. Night-time driving, bad weather, icy roads, bumper-to-bumper traffic: these are all situations in which some algorithms can outdo humans in terms of safety. Faster reactions, better vision (beyond what human eyes can see), and unlimited 'mental stamina' can make a big difference in safe driving. But then there will be the occasional situation in which the CV screws up, and there's an accident. Some of those are ones where many/most humans could have handled the situation better and avoided the accident. So how do we decide when the automated car is 'good enough'? Do we have to reach a point where in no situation could any human have done better? Must it be absolutely better than all humans, all the time? Because we may never reach that point. And all the while, we could be avoiding a lot more accidents (and deaths) from situations the AI could have handled.
  - glitchc 5y ago
    > Night-time driving, bad weather, icy roads, bumper-to-bumper traffic: these are all situations in which some algorithms can outdo humans in terms of safety. Faster reactions, better vision (beyond what human eyes can see), and unlimited 'mental stamina' can make a big difference in safe driving. To be clear we are talking about CV which relies on passive optical sensing in the visual spectrum through cameras, not radar or lidar or IR or multi-spectral sensors. Within this context, your statement is incorrect. A typical camera’s dynamic range is orders of magnitude lower than the human visual dynamic range. Ergo a camera sees a lot less at night compared to a human and what it does see is a lot more noisy. Note that this is the input to the detection, tracking and classification stages, the ouput of which feeds into the control loop(s). It doesn’t matter how good the control system is, it cannot avoid what the vision system cannot see.
rStar 5y ago
> 1. People who post these stickers should go to prison. 2. People who create and distribute these stickers knowing their purpose should go to prison. 3. Tesla should be civilly liable for cases where preventing such an incident was possible with known technology. 4. Roads should be modified over time to make it more difficult to do this attack. Translation: everyone else in the universe is responsible for solving my problem, and also I am not responsible for solving my problem, but i do want to profit from the current state of everything being broken all the time, and, i tell my family to keep their hands on the wheel
darepublic 5y ago
So your solution is to create a totalitarian state. So your flaky software can be secure. No thanks
dogleash 5y ago
There would be support to outlaw adversarial attacks towards self-driving cars. As other posters have suggested it probably already is illegal, or is a narrow expansion of scope for existing laws. >We can try to make them hard to make, hard to posses anonymously, and hard to deliver. To stretch your own analogy, I have a wide selection of poisons at home. Except we call them cleaning products, insecticide and automobile fluids. You can get public support against adversarial attacks on self-driving. Except the main use case for computer vision is passive surveillance. Good luck on that front. Oh, and just for funzies, I'll point out the irony that some of the people building CV surveillance systems would post on HN that regardless of regulation it'll exist no matter what the government wants. The argument was that it'd be so hard for the government to control CV surveillance, that law wouldn't prevent business from creating and using it anyway. When it comes to adversarial attacks, it seems more likely to involve actions of private individuals rather than businesses, and businesses minimize legal risk in a way individual citizens don't.
yjftsjthsd-h 5y ago
> I believe the same will be true of adversarial examples for vision (and language) models. We can try to make them hard to make, hard to posses anonymously, and hard to deliver. I think this will be much easier with computer vision than with poison, so I'm not worried about it. Erm. We can maybe do something about delivery, but stopping people from making (and thus, possessing) them is virtually impossible, since all you need is an undergrad-level understanding of ML (if that) and some freely-available software.
p2p_astroturf 5y ago
Your analysis does not break out of the well known box that is the classical ways of analyzing the security of a computer system (it actually creeps into DRM/TPM territory which is known insecure despite governments with guns). Thus the security of "AI" algorithms remains as insecure as it already was, and should not be used for anything that needs to be secure. If anything, the people who make critical infrastructure insecure should go to prison (after education is reformed to actually teach these basic problems). Your example is like how typical american citizens get their panties in a bunch and throw you in jail for 5000 years if you fake your identity, but this is only because they have build such insecure systems that comeletely break down once this happened. And this is yet another thing not fixed by policing. Sorry not sorry if I sound rude. You are basically asking me to go to jail so you can use some convenient AI consumer tech in lieu of proper solutions for stuff like authentication, court systems, and car driving (and all the other thing the wackos want to replace with AI).
- mgraczyk 5y ago
  No, I'm asking you to go to jail if you intentionally try to cause somebody to die.
  - brokenmachine 5y ago
    The people shilling crap AI should bear some responsibility in my opinion. If your garbage AI powered car can be tricked easier than an 8 year old, who's fault is it? We wouldn't let an 8 year old drive, but your garbage AI is fine?
  - p2p_astroturf 5y ago
    No, you're asking me to not communicate to other security resarchers about the problems with security systems (for example, publishing a PoC): > 2. People who create and distribute these stickers knowing their purpose should go to prison. ... and you're asking the entire world to change to make your vehicle work: > 4. Roads should be modified over time to make it more difficult to do this attack. 10 years ago when AI cars were getting memed into existence I would get dogpiled on for naysaying. Now you same people want laws to stop people from breaking what was already easily breakable and which you argued was unbreakable. Also, your poison analogy is invalid. Poison is not uncommon because of law, it's uncommon because it's uncommon. In the future, crazy people will be poisoning random stuff in the grocer because they don't like the demographic that shops there.
thaumasiotes 5y ago
> Instead, we rely on weaker protections. We try to make known poisons hard to make, we try to track people who could make them, and we try to make it hard to deliver poison. In most cases, our first and last defense against an attack of this type is to rely on the fact that nobody is interested in doing it.
mannerheim 5y ago
> 2. People who create and distribute these stickers knowing their purpose should go to prison. This would surely not pass constitutional muster.
- mgraczyk 5y ago
  You might be surprised to learn that the US government regulates things that are much closer to what you consider "speech". For example, > Federal law prohibits the possession with intent to sell or distribute obscenity, to send, ship, or receive obscenity, to import obscenity, and to transport obscenity across state borders for purposes of distribution. https://www.justice.gov/criminal-ceos/citizens-guide-us-federal-law-obscenity https://www.justice.gov/criminal-ceos/citizens-guide-us-fede... A specific recent case: https://www.mtsu.edu/first-amendment/article/167/united-states-v-williams https://www.mtsu.edu/first-amendment/article/167/united-stat... > “offers to engage in illegal transactions are categorically excluded from First Amendment protection.” If it's illegal to posses something, the government can ban offering to sell and distribute it.
  - mannerheim 5y ago
    Is federal obscenity law enforced? I doubt it can be enforced either, with the decision in Reno v. ACLU that struck down similar provisions of the Communications Decency Act as violating the First Amendment. > If it's illegal to posses something, the government can ban offering to sell and distribute it. This is begging the question, because first you would have to show that banning the mere possession of adversarial images doesn't violate the First Amendment. Otherwise you can make it illegal to possess books in a way to prevent the sale and distribution of literature.
    - d110af5ccf 5y ago
      > Is federal obscenity law enforced? It seems to vary. Typically not, and when attempts are made court verdicts seem to be mixed. But unfortunately sometimes people do get convicted and the convictions upheld. For example, United States v. Whorley. https://caselaw.findlaw.com/us-4th-circuit/1431669.html https://caselaw.findlaw.com/us-4th-circuit/1431669.html
deleted 5y ago
[deleted]

deleted 5y ago

[deleted]

voakbasda 5y ago

Your proposed laws do not cut out any exemption for research and experimentation, either with existing systems or potential new ones. This level of regulation would create an impossibly high barrier to entry and ensure that only the established players would remain in the marketplace. The last thing that I want to see is yet more regulatory capture, particularly in an industry that has yet to establish a reasonable baseline of success.

mgraczyk 5y ago
None of the things I listed would affect research. Researchers shouldn't be posting these on public highways, and researchers shouldn't be distributing them with the intent to cause harm.
- mrfox321 5y ago
  It would once the govt gets involved. It's like saying that weapons research is just a free-for-all. The amount of regulation is correlated with the potential harm to society. Look at drug research. There is plenty of red tape that hinders it. Although, here, the "harm to society" is defined by the nation state. However, I agree with your proposals in the top-level comment.
- d110af5ccf 5y ago
  This is naive. > researchers shouldn't be distributing them with the intent to cause harm It could be used to cause harm and publishing your work is distributing it (by definition). Similar laws have been (and are) used to target people for unjust reasons. Defacing street signs is already illegal. Making various abstract drawings illegal to wear on your clothing (or print on a sticker) is a terrible idea. If the algorithm used by a product can't handle a pedestrian wearing a sweatshirt with an adversarial example on it then that product simply isn't suitable for public use.
Laremere 5y ago
INAL, but actually putting adversarial image attacks on real roads is already illegal. If you modify a street sign, and as a result someone dies, that's a fairly easy case of Involuntary manslaughter. At a minimum, you can't modify street signs. Eg in Washington State: RCW 47.36.130 Meddling with signs prohibited. No person shall without lawful authority attempt to or in fact _alter_, deface, injure, knock down, or remove any official traffic control signal, _traffic device_ or railroad sign or signal, or any inscription, shield, or insignia thereon, or any other part thereof. (Underscore emphasis added). And if you're thinking about not putting it on a sign, but putting it elsewhere visible to cars: RCW 46.61.075.1 Display of unauthorized signs, signals, or markings. No person shall place, maintain or display upon or in view of any highway any unauthorized sign, signal, _marking or device_ which purports to be or is an imitation of or resembles an official traffic-control device or railroad sign or signal, or _which attempts to direct the movement of traffic_, or which hides from view or interferes with the effectiveness of an official traffic-control device or any railroad sign or signal. Where I'm unsure is producing these with the intent or knowledge that they will/could be used by someone to go do this. None of this makes using these for research and experimentation illegal.

pixelgeek 5y ago

> For example, consider the case of pasting a sticker on a speed limit sign that causes Teslas to swerve off the road. If your vision system can be caused to swerve off a road by a sticker then maybe it shouldn't be used?

mgraczyk 5y ago
I bet I could cause a significant fraction of human vision systems to get in a crash with a well placed sticker. I'd replace "<- One Way" with "Detour ->".
- pixelgeek 5y ago
  But that is not a vision issue. That is providing people with incorrect information.
  - qw501428 5y ago
    Perhaps a mirror/reflective sticker that blinds drivers near a sharp curve?
  - b3morales 5y ago
    The specific trick doesn't really matter; the point is that it's possible to maliciously create a situation that makes human pilots act dangerously. We accept that the possibility can't be made nil, and we have post facto rules to deal with it. The same principle applies to traps for machines.
    - ClumsyPilot 5y ago
      Nope, it 0.1% of humans crash but 100% of teslas crash that's not 'the same'
      - MauranKilom 5y ago
        So would it be better/fine if Tesla randomized the CV ML models to some extent? I actually feel the answer is probably "yes", surprisingly...

jimbob45 5y ago

>It would be fairly easy to add poison to a water supply or the air intake of a large building and kill a large number of people. I used to think this until someone walked me through the logistics of both and made me realize that you would need an agency-alerting level of poison for the water supply and some way to avoid people just shutting off the A/C and sticking their heads out of windows (also a huge amount of gas). Also the news can't exist to alert anyone immediately.

xboxnolifes 5y ago
> agency-alerting level of poison Doesn't this fall under: "We try to make known poisons hard to make, we try to track people who could make them, and we try to make it hard to deliver poison"? And the rest of it being you should use something odorless / tasteless.

ggoo 5y ago

I think a factor that should also be considered in your analogy is that poison is much more difficult to attain than stickers. I have to imagine that if poison was cheaply and widely available as stickers, we'd have a much larger problem than we currently see.

wcarey 5y ago
Most households contain at least several poisons (and precursors to hazardous gasses as well) amongst their cleaning supplies.
- deleted 5y ago
  [deleted]

ReleaseCandidat 5y ago

> We try to make known poisons hard to make, we try to track people who could make them, and we try to make it hard to deliver poison. Actually no. We know that only some psychopaths would do that and so the risk is minimal. AI is currently simply not 'good enough' to be used in critical environments. The problem is that _any_ sticker or even dirt or snow or ... on any road sign can lead to misinterpretation, you can never proof that it's safe.

mgraczyk 5y ago
Sometime iceberg lettuce kills people (salmonella). We have safety regulations and inspections to mitigate that, but you can never prove that iceberg lettuce is safe.
- d110af5ccf 5y ago
  This is specious. I can take actions to mitigate food poisoning such as rinsing my vegetables prior to consumption. If a company claims that their leafy greens are already rinsed and safe to consume without further prep work when in fact they are not then presumably they can be held liable. The risks surrounding vegetable packaging and distribution are well understood, readily quantifiable, and possible to mitigate. Computer vision algorithms on the other hand are poorly understood black boxes with seemingly arbitrary failure modes. We do not (yet) appear to understand how to quantify or mitigate the associated risks. The consequences of failure are quite severe in comparison to food poisoning. Only a fool would trust their life to them.

aneutron 5y ago

What you are proposing are what I think would be called a security theater. It gives the illusion of security, but they would absolutely not deter a determined threat actor. The only reason that the water supply isn't poisoned is it's unpractical for a single person to conduct the whole exploit chain: Construct the poison in enough quantities, gain access to facilities supplying the water, and actually throwing the compound in it. It's unpractical even for "underground" types. Especially the quantities required. Mathematics and computer science is a different story in my opinion. You cannot restrict science or thought. You can try, but good luck. The most you can do is delay it. If there is an attack that enables someone to flip a Tesla on the road (as suggested below), the security theater will hide the attack from common folk, but determined actors will reach it eventually, and at that point, they can deploy it as they wish. And in contrast to the water plant, the logistical endeavor to exploit it is absolutely easy in comparison: slap a sticker on your vehicle. Security by obscurity or by theater is rarely a good strategy in my opinion. We should absolutely be transparent about these kind of things, and allow researchers full access to develop attacks against these systems, and effectively communicate when they are found.

mgraczyk 5y ago
Right, and the protections for poison are also security theater for the same reason. In the real world that's ok. > The only reason that the water supply isn't poisoned is it's unpractical for a single person to conduct the whole exploit chain It's a quantitative question, just like with computer vision. If you don't like the poison example, consider viral DNA, which is also dangerous in the right hands and does not require massive supply chain control. Not everyone has access to a driving dataset like Teslas, and it would be difficult to trick a Tesla without such a dataset. We should allow researches to develop attacks, just like we should allow researchers to study poisons, DNA, and viruses.
- flerovium 5y ago
  The lab leak theory isn't necessarily true, but this analogy isn't as conclusive as you make it sound...
5y ago

seany 5y ago

1 and 2 are almost always going to be impossible in the US due to the first amendment (this is a feature not a bug) 3 doesn't seem crazy, but it would practically end up with caps, which might not be what you're looking for 4 This both: seems possible, and will basically never happen due to cost in every little jurisdiction

eximius 5y ago
Defacing property is not free speech...?
- seany 5y ago
  Just stand in view holding the image on a poster board.
hacoo 5y ago
I doubt 1 would be protected by the first amendment. It’s arguably equivalent to spraying graffiti on a stop sign so it’s unrecognizable. It would be an extremely difficult to enforce though.
- boomboomsubban 5y ago
  Graffiti would just cause people to drive unsafely, in that hypothetical the sticker directly causes crashes. It'd be something like attempted murder.
mgraczyk 5y ago
#1 is certainly not a first amendment violation. In fact, the supreme court still holds that certain restrictions on billboards are allowed even for the purpose of preserving beauty. Safety is a much more compelling interest than beauty, so I don't expect states and cities will lose their ability to regulate road signage. See Metromedia, Inc. v. San Diego for example. #2 is expensive and difficult, but that's what we do for explosives, poisons, drugs, etc.
- 5y ago

version_five 5y ago

I lot of this has been touched on already, but I think your rules could be reframed a bit to try simplify lawmaking and avoid security theatre as was mentioned. First, I assume it's already illegal to be "adversarial" to drivers. A bright light or changing signs etc already do that now. For example look at all the laser pointer stuff with planes. Second, I don't think self driving cars are just using the softmax output of an object detector as a direct input to car control decisions. In the absence of a stop sign, the expected behavior would be common sense and caution, the same as if someone removed the sign. If the SDC logic is not robust in this way, it's not safe for many other reasons. With this in mind, I think the situation is probably already reasonable well covered in existing regulations.

ketzo 5y ago

Relevant XKCD: https://xkcd.com/1958/ https://xkcd.com/1958/ > I worry about self-driving car safety features. > What's to stop someone from painting fake lines on the road, or dropping a cutout of a pedestrian onto a highway, to make cars swerve and crash? > Except... those things would also work on human drivers. What's stopping people now? > Yeah, causing car crashes isn't hard. > I guess it's just that most people aren't murderers? > Oh, right, I always forget. > An underappreciated component of our road safety system.

citilife 5y ago
That's how I feel about most dangerous situations in general and I think the national news highlights one-off events in a way we historically were not used to. For instance, taking out the United States internet would probably only required 3-4 strategic bombings. I bring this up because Tennessee had one of those bombed Christmas last year -- https://www.theverge.com/2020/12/28/22202822/att-outage-nashville-christmas-bombing https://www.theverge.com/2020/12/28/22202822/att-outage-nash... > This brought down wireless and wired networks across parts of Tennessee, Kentucky, and Alabama Most people aren't all that concerned about doing damage. Keep people happy and generally you don't have crime.

mnd999 5y ago

That worked super well with DVD CSS right? Let’s face it, people are going to print adversarial images on t-shirts.

MikeHolman 5y ago
If the adversarial image is intended to cause car accidents or bodily harm in some way, then the people printing the t-shirts and the people wearing them are already breaking the law. And if they actually do hurt someone, I imagine they would be criminally liable.

petermcneeley 5y ago

I am pretty sure that deliberately tricking an automated system into causing bodily harm is already coved by existing law. Think of all the automated systems that have existed before ML.

indymike 5y ago

> People who post these stickers should go to prison Doing things with intent to harm others is illegal, even if you use a sticker to do it. > Tesla should be civilly liable for cases where preventing such an incident was possible with known technology. This is currently likely the case, but is not proven until a lawsuit happens.

glitchc 5y ago

Adversarial examples don't confuse people, only algorithms. Perhaps you need to face the fact that if the CV algorithm fails against these examples when humans don't, then the CV algorithm is too brittle and should not be used in the real world. I don't trust my life to your "It kinda looks like a road, oh wait it's a pylon, I've been tricked, BAM!" dumpster fire of an algorithm. We used to have to craft robustness into algorithms based on the false positive rate. Nobody looks at a CFAR style approach anymore, and it shows. The state of the art approach of pinning everything on ML is a dead-end for CV.

mabbo 5y ago
> if the CV algorithm fails against these examples when humans don't, then the CV algorithm is too brittle and should not be used in the real world. This is the tricky bit. Night-time driving, bad weather, icy roads, bumper-to-bumper traffic: these are all situations in which some algorithms can outdo humans in terms of safety. Faster reactions, better vision (beyond what human eyes can see), and unlimited 'mental stamina' can make a big difference in safe driving. But then there will be the occasional situation in which the CV screws up, and there's an accident. Some of those are ones where many/most humans could have handled the situation better and avoided the accident. So how do we decide when the automated car is 'good enough'? Do we have to reach a point where in no situation could any human have done better? Must it be absolutely better than all humans, all the time? Because we may never reach that point. And all the while, we could be avoiding a lot more accidents (and deaths) from situations the AI could have handled.
- glitchc 5y ago
  > Night-time driving, bad weather, icy roads, bumper-to-bumper traffic: these are all situations in which some algorithms can outdo humans in terms of safety. Faster reactions, better vision (beyond what human eyes can see), and unlimited 'mental stamina' can make a big difference in safe driving. To be clear we are talking about CV which relies on passive optical sensing in the visual spectrum through cameras, not radar or lidar or IR or multi-spectral sensors. Within this context, your statement is incorrect. A typical camera’s dynamic range is orders of magnitude lower than the human visual dynamic range. Ergo a camera sees a lot less at night compared to a human and what it does see is a lot more noisy. Note that this is the input to the detection, tracking and classification stages, the ouput of which feeds into the control loop(s). It doesn’t matter how good the control system is, it cannot avoid what the vision system cannot see.

rStar 5y ago

> 1. People who post these stickers should go to prison. 2. People who create and distribute these stickers knowing their purpose should go to prison. 3. Tesla should be civilly liable for cases where preventing such an incident was possible with known technology. 4. Roads should be modified over time to make it more difficult to do this attack. Translation: everyone else in the universe is responsible for solving my problem, and also I am not responsible for solving my problem, but i do want to profit from the current state of everything being broken all the time, and, i tell my family to keep their hands on the wheel

darepublic 5y ago

So your solution is to create a totalitarian state. So your flaky software can be secure. No thanks

dogleash 5y ago

There would be support to outlaw adversarial attacks towards self-driving cars. As other posters have suggested it probably already is illegal, or is a narrow expansion of scope for existing laws. >We can try to make them hard to make, hard to posses anonymously, and hard to deliver. To stretch your own analogy, I have a wide selection of poisons at home. Except we call them cleaning products, insecticide and automobile fluids. You can get public support against adversarial attacks on self-driving. Except the main use case for computer vision is passive surveillance. Good luck on that front. Oh, and just for funzies, I'll point out the irony that some of the people building CV surveillance systems would post on HN that regardless of regulation it'll exist no matter what the government wants. The argument was that it'd be so hard for the government to control CV surveillance, that law wouldn't prevent business from creating and using it anyway. When it comes to adversarial attacks, it seems more likely to involve actions of private individuals rather than businesses, and businesses minimize legal risk in a way individual citizens don't.

yjftsjthsd-h 5y ago

> I believe the same will be true of adversarial examples for vision (and language) models. We can try to make them hard to make, hard to posses anonymously, and hard to deliver. I think this will be much easier with computer vision than with poison, so I'm not worried about it. Erm. We can maybe do something about delivery, but stopping people from making (and thus, possessing) them is virtually impossible, since all you need is an undergrad-level understanding of ML (if that) and some freely-available software.

p2p_astroturf 5y ago

Your analysis does not break out of the well known box that is the classical ways of analyzing the security of a computer system (it actually creeps into DRM/TPM territory which is known insecure despite governments with guns). Thus the security of "AI" algorithms remains as insecure as it already was, and should not be used for anything that needs to be secure. If anything, the people who make critical infrastructure insecure should go to prison (after education is reformed to actually teach these basic problems). Your example is like how typical american citizens get their panties in a bunch and throw you in jail for 5000 years if you fake your identity, but this is only because they have build such insecure systems that comeletely break down once this happened. And this is yet another thing not fixed by policing. Sorry not sorry if I sound rude. You are basically asking me to go to jail so you can use some convenient AI consumer tech in lieu of proper solutions for stuff like authentication, court systems, and car driving (and all the other thing the wackos want to replace with AI).

mgraczyk 5y ago
No, I'm asking you to go to jail if you intentionally try to cause somebody to die.
- brokenmachine 5y ago
  The people shilling crap AI should bear some responsibility in my opinion. If your garbage AI powered car can be tricked easier than an 8 year old, who's fault is it? We wouldn't let an 8 year old drive, but your garbage AI is fine?
- p2p_astroturf 5y ago
  No, you're asking me to not communicate to other security resarchers about the problems with security systems (for example, publishing a PoC): > 2. People who create and distribute these stickers knowing their purpose should go to prison. ... and you're asking the entire world to change to make your vehicle work: > 4. Roads should be modified over time to make it more difficult to do this attack. 10 years ago when AI cars were getting memed into existence I would get dogpiled on for naysaying. Now you same people want laws to stop people from breaking what was already easily breakable and which you argued was unbreakable. Also, your poison analogy is invalid. Poison is not uncommon because of law, it's uncommon because it's uncommon. In the future, crazy people will be poisoning random stuff in the grocer because they don't like the demographic that shops there.

thaumasiotes 5y ago

> Instead, we rely on weaker protections. We try to make known poisons hard to make, we try to track people who could make them, and we try to make it hard to deliver poison. In most cases, our first and last defense against an attack of this type is to rely on the fact that nobody is interested in doing it.

mannerheim 5y ago

> 2. People who create and distribute these stickers knowing their purpose should go to prison. This would surely not pass constitutional muster.

mgraczyk 5y ago
You might be surprised to learn that the US government regulates things that are much closer to what you consider "speech". For example, > Federal law prohibits the possession with intent to sell or distribute obscenity, to send, ship, or receive obscenity, to import obscenity, and to transport obscenity across state borders for purposes of distribution. https://www.justice.gov/criminal-ceos/citizens-guide-us-federal-law-obscenity https://www.justice.gov/criminal-ceos/citizens-guide-us-fede... A specific recent case: https://www.mtsu.edu/first-amendment/article/167/united-states-v-williams https://www.mtsu.edu/first-amendment/article/167/united-stat... > “offers to engage in illegal transactions are categorically excluded from First Amendment protection.” If it's illegal to posses something, the government can ban offering to sell and distribute it.
- mannerheim 5y ago
  Is federal obscenity law enforced? I doubt it can be enforced either, with the decision in Reno v. ACLU that struck down similar provisions of the Communications Decency Act as violating the First Amendment. > If it's illegal to posses something, the government can ban offering to sell and distribute it. This is begging the question, because first you would have to show that banning the mere possession of adversarial images doesn't violate the First Amendment. Otherwise you can make it illegal to possess books in a way to prevent the sale and distribution of literature.
  - d110af5ccf 5y ago
    > Is federal obscenity law enforced? It seems to vary. Typically not, and when attempts are made court verdicts seem to be mixed. But unfortunately sometimes people do get convicted and the convictions upheld. For example, United States v. Whorley. https://caselaw.findlaw.com/us-4th-circuit/1431669.html https://caselaw.findlaw.com/us-4th-circuit/1431669.html

deleted 5y ago

[deleted]