17 ms·

Adversarial image attacks are no joke

178 points by Hard_Space 5y ago

meiji163 5y ago
Language models have the same defect - they are quite brittle and susceptible to black-box adversarial attacks ( eg arxiv.org/abs/1907.11932 )
bckr 5y ago
I like the cnn-adversarial aesthetic. Psychedelic blobs (sans the swirly demon dog faces), flower paintings and vinyl stickers of organic figures everywhere!
corey_moncure 5y ago
This is bad news for safety critical computer vision systems like Tesla vision.
- amelius 5y ago
  Don't worry, they'll just "convince" the regulators to ignore these problems.
  - pixelgeek 5y ago
    Maybe put a sticker on the conference table that distracts them?
frazbin 5y ago
The really scary thing is that this could be used as an excuse to hide production ML models and even the tech used to generate them. Sounds like we can expect the state-of-the-art AI techniques to be jealously guarded eventually. I guess optimism on the ground is enough to have prevented that so far, but once the scales tip away from sharing and towards exploitation.. well, we know it's largely a one-way process on the 1 decade time scale. Is this the chilling effect that will bring us into the next AI winter?
- jowday 5y ago
  > Sounds like we can expect the state-of-the-art AI techniques to be jealously guarded eventually. This isn’t an eventuality, it’s the current state of the industry.
  - frazbin 5y ago
    Hm is that really true? I thought that there was quite a lot of sharing from industry leaders at the research paper and dataset level, and that these could be used imitate production systems given some hacking. Kinda seemed like the majors were enjoying the benefits of the scrutiny afforded to public scientific research, while keeping their monopoly confined to the silicon/speed/throughput axis. Hence all the free AI software toolkits and also high priced specialty hot-off-the-wafer chips you'll never get.
laura_g 5y ago
The literature has pretty consistently shown that adversarial examples can be found with only black box access (even with truncated prediction vectors), robustness methods are primarily a cat-and-mouse game between attackers and defenders, and the existence of adversarial examples is likely inevitable (https://arxiv.org/pdf/1809.02104.pdf https://arxiv.org/pdf/1809.02104.pdf). The big question that remains is - so what? There's exceedingly few use cases where the existence of adversarial examples causes a security threat. There's a lot of research value in understanding adversarial examples and what that tells us about how models learn, generalize, and retain information, but I am not convinced that these attacks pose a threat remotely close to the amount of attention given.
- owlbite 5y ago
  Self driving cars seem like a dangerous threat vector if an adversarial image can be deployed in such a way as to cause them to commit dangerous maneuvers on demand.
  - laura_g 5y ago
    I completely agree, but that's a very big "if". I'm not terribly familiar with autonomous vehicle driving systems, but my passing understanding is that there are multiple components working together that help make predictions, and these systems do not rely on any single point of failure. The classic example of a sticker on a stop sign is, in my view, more of a dramatization than a real threat surface. Designing an adversarial perturbation on a sticker that can cause misclassifications from particular angles and lighting conditions is possible, but that alone won't cause a vehicle to ignore traffic situations, pedestrians, and other contextual information. Plus, if I wanted to trick a self driving vehicle into not stopping at an intersection, it would be much easier and cheaper for me to just take the stop sign down :)
  - Isinlor 5y ago
    There is plenty of natural "adversarial examples" to worry about. Like billboard with stop sign on it. https://youtu.be/-OdOmU58zOw?t=149 https://youtu.be/-OdOmU58zOw?t=149
mgraczyk 5y ago
As somebody who works on computer vision, my general take on these things is that adversarial examples are like poison. It would be fairly easy to add poison to a water supply or the air intake of a large building and kill a large number of people. This rarely happens though. It's ok that water sources, buildings, and people aren't completely immune to poison. The safety requirement isn't that poison can't hurt. Instead, we rely on weaker protections. We try to make known poisons hard to make, we try to track people who could make them, and we try to make it hard to deliver poison. I believe the same will be true of adversarial examples for vision (and language) models. We can try to make them hard to make, hard to posses anonymously, and hard to deliver. I think this will be much easier with computer vision than with poison, so I'm not worried about it. For example, consider the case of pasting a sticker on a speed limit sign that causes Teslas to swerve off the road. Governments should protect people from this in multiple ways, similarly to how they protect us from poison: 1. People who post these stickers should go to prison. 2. People who create and distribute these stickers knowing their purpose should go to prison. 3. Tesla should be civilly liable for cases where preventing such an incident was possible with known technology. 4. Roads should be modified over time to make it more difficult to do this attack. I think some combination of the above would be enough to make society as comfortable with adversarial example risk as we are with poison risk.
- deleted 5y ago
  [deleted]
- voakbasda 5y ago
  Your proposed laws do not cut out any exemption for research and experimentation, either with existing systems or potential new ones. This level of regulation would create an impossibly high barrier to entry and ensure that only the established players would remain in the marketplace. The last thing that I want to see is yet more regulatory capture, particularly in an industry that has yet to establish a reasonable baseline of success.
  - mgraczyk 5y ago
    None of the things I listed would affect research. Researchers shouldn't be posting these on public highways, and researchers shouldn't be distributing them with the intent to cause harm.
ravi-delia 5y ago
It seems to me theres a difference between attacks that carefully craft an image that slips through the cracks, and an attack that basically exploits the fact that without context, it's hard to figure out what single item is important. If I took a picture of a conch shell on top of my keyboard and sent it to someone, no one would think I was just showing off my keyboard! They'd assume, correctly, that my desk was a mess and I didn't feel like finding a clear surface. That's not to say that either attack is less harmful than the other! If you train an image classifier to find bikers, it's not really wrong or right to say that a picture of a biker qualifies. But if a car stops lest it run over a painted bike on the road, that's obviously bad. The problem is that you aren't trying to recognize bikers, you're trying to avoid obstacles. We just don't train well for that.
- pixelgeek 5y ago
  I don't think that the word 'train' should be used for these systems. We feed then reams of data and effectively cull the ones that don't work but the critical problem is that we judge the effectiveness of an ML system and we actually do know what the ML systems is supposed to be looking for. We feed a system a series of images of bikes and then select the ones that can pick out a bike but we don't know how the bike is being chosen. We know it is picking out bikes but we have no way to predict if the system is picking out bikes or picking out a series of contrasting colour and shadow shapes and could easily be thrown off by anything that contains the same sort of data.
  - NavinF 5y ago
    Thank you for an accurate ELI5 description of the human visual system. Dunno what this “ML” is, I assume it’s some part of the brain? It’s too bad you can’t analyze brains like you can with neural networks. It’s trivial to visualize filters and feature maps or to create heatmaps showing which pixels (shadow shapes?) in a specific image affect the classification output and why (contrasting color?).
    - d110af5ccf 5y ago
      The issue is that a human driver is much more than just a visual cortex. > which pixels (shadow shapes?) in a specific image affect the classification output and why (contrasting color?) Sure, you can watch the Rube Goldberg machine work. It doesn't mean you understand why it works on a conceptual level or have any hope of rigorously quantifying when and how it could fail.
pixelgeek 5y ago
Maybe this is a good example of why ML systems shouldn't be used? Ultimately we don't know how the networks that get created actually make decisions so doesn't that make protecting them from attacks like this impossible?
SavantIdiot 5y ago
There is a fundamental disconnect between what deep vision models can do and what is expected of them. On the one hand, there is a very good reason why mean-average-precision is used to assess detection-classification models: because even people make mistakes. On the other hand, we need to apply the use of these forever imperfect models with care, context, and redundancy. This is why engineers add a dozen other input types to ADAS systems in addition to vision (sonar, lidar, mesh computing, etc). This is why regulation is needed, to prevent less rigorous products from making their way into situations where the can be easily compromised, or worse, deadly.
dontreact 5y ago
If you really wanted to crash cars by altering their visual input, why would you bother with all this complexity? Why not just actually swap the road sign? Why does the existence of these attacks change the threat landscape at all? If people are already not doing "dumb" attacks like just changing/removing road signs why would they start doing them? The risk of messing with road signs and throwing off autonomous vehicles really has less to do with adversarial image attacks and more to do with envisioning an impractically brittle system where the decision to stop is based purely on presence/absence of a stop sign and not on a system that has a more general sense of collision-avoidance and situational awareness (like humans do).| Stepping back more generally, I have still never seen a case where the undetectability of adversarial attacks actually means there is a practical difference to security or safety. If you really think through the impact in the real world, usually the risk is already there: you can just change the input to the image and get bad results, it doesn't affect much that the image is imperceptibly changed. Because the whole point of using an automated vision system is usually that you want to avoid human eyes on the problem.
- goatlover 5y ago
  > Why not just actually swap the road sign? Because you have to physically do it, as opposed to hacking from anywhere else on the planet. > not on a system that has a more general sense of collision-avoidance and situational awareness (like humans do). Are vision systems to that point yet when it comes to driving vehicles? > Because the whole point of using an automated vision system is usually that you want to avoid human eyes on the problem. And the point of hacking an automated system is that it's easier to do that remotely than to cause a human to crash locally.
  - JoshuaDavid 5y ago
    > Because you have to physically do it, as opposed to hacking from anywhere else on the planet. My impression is that the adversarial image attacks in question involve physically placing a sticker on something which will be in the view of self-driving cars -- it's not a remote exploit.
JohnFen 5y ago
I take a large measure of hope from this. I see facial recognition as a large societal threat, and it's nice to know that a defense is possible.
- zcw100 5y ago
  Image cloaking https://sandlab.cs.uchicago.edu/fawkes/ https://sandlab.cs.uchicago.edu/fawkes/
dhosek 5y ago
Whenever these discussions come up, I often think of a time I was driving on a two-lane road in rural Ohio in the 90s and at one point the center stripe curved into the lane (presumably because the driver of the striping truck pulled over without turning off the striper) and I started to curve off the road thanks to subconscious interpretation of the cue. I caught myself before I drove into a corn field, but human vision systems are also susceptible to these sorts of problems.
- donkarma 5y ago
  You caught yourself, didn't you?
OldHand2018 5y ago
I see a completely different attack vector here. Lawyers. If you are selling a product or service that has been trained on a dataset that contains copyrighted photos you don't have permission to use and I can "prove it" enough to get you into court and into the discovery phase, you are screwed. I'll get an injunction that shuts you down while we talk about how much money you have to pay me. And lol, if any of those photos of faces was taken in Illinois, we're going to get the class-action lawyers involved, or bury you with a ton of individual suits from thousands of people. That link at the bottom about a "safe harbor" you get from using old datasets from the Wild West is not going to fly when you start selling.
- laura_g 5y ago
  This would be membership inference attacks - https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=7958568&casa_token=edBGtFGrJlwAAAAA:eLObvr85G6__HNWXaAWmdqcK-ljtTy_eniFBv4BM3EBojJ7pza3n-GxsnOmqC8GFjj5ZCzZb https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=7958568...
  - OldHand2018 5y ago
    Oh excellent. But of course the key addition is handing off this information to lawyers who use it to shut you down and/or extract money from you. If you are using some torrent of a dataset, nobody is indemnifying you, and once you get to the discovery phase of a lawsuit, they are going to know that you intentionally grabbed a dataset you knew you shouldn't have had access to. Treble damages!
- NoGravitas 5y ago
  I dunno, Microsoft seem to think they can get away with training autocomplete on copyrighted source code that they don't have permission to use.
- KingMachiavelli 5y ago
  IIRC simply training on copyrighted material is completely fine or at least you can claim fair use. As long as the market of the copyrighted material is not 'AI data training set' then it should be OK. Essentially scraping images from the internet is OK but using a pirated copyrighted commercial AI data training set is not. (Fair use doesn't necessarily exclude use for a commercial/sold product.) But if the AI model just spits out copyrighted material verbatim then that is still owned by the actual copyright holder.
deleted 5y ago
[deleted]
m3kw9 5y ago
I can propose a non-trivial solution to these problems, that is to have a data cleaner average and ignore certain data like how humans does it. Humans would ignore everything else but the face and maybe the body, and also we don’t examine someone’s follicles either, we basically average.
adolph 5y ago
"Adversarial" communication with a CV model inference process isn't necessarily an attack because it is unintended by the humans associated with the process. It is more akin to using the full range of an API that uses radiation instead of a network port. It could be used to stage a protest by stopping or slowing cars on a freeway or call attention to deteriorating infrastructure by inducing the car to go over potholes instead of avoiding them. Maybe a neighborhood could self-implement traffic calming measures that don't apply to emergency vehicles.
- voldacar 5y ago
  If you are trying to make my car go over potholes without my consent, or in any way do something that I don't want it to, that is adversarial behavior. You are my adversary.
whatever1 5y ago
Is traditional computer vision less susceptible to these? Since the features are human crafted, it sounds to me that the risk would be much lower.
- ativzzz 5y ago
  I would imagine more because those features are probably easier to reverse engineer. Plus traditional CV is weaker in generalized scenarios and is pretty easy to trick or throw off.
draw_down 5y ago
The "adversarial" language around this feels like blame-shifting. If your algorithm thinks a flower is Barack Obama, it's not the flower's fault.
quirkot 5y ago
The key point I see in this is that, given the current ecosystem, attacks are systemic. Plus, given the nature of ML training and datasets, it's expensive to bug fix an attack, if it's even possible. This right here is the real underlying long term danger: > the most popular CV datasets are so embedded in development cycles around the world as to resemble software more than data; software that often hasn’t been notably updated in years
windows2020 5y ago
Some people think that a self driving car needs to perform just well enough to meet some average metric at which point its shortcomings are considered justified. Would you drive with a human that could fail so spectacularly? As least they can rationalize their decision. My opinion is this: we're missing something big. Current weak AI strategies may be sufficient in some domains, but until the essence of consciousness can be defined, general AI (and AI I would trust my life with) is out of the question.
igorkraw 5y ago
I work in this field, I have a project specifically on adversarial examples and I have a strong opinion on this. I personally think worrying about adversarial examples in real life production systems is like worrying about getting the vanilla linux kernel to perform RT critical tasks. It is fundamentally not a burden you should put on that one component alone and is a problem you can only solve with a system approach. And if you do that, it is for all practical purposes already solved: apply multiple, random perturbations to the input, project your perturbed version onto a known,safe image space, and establish consensus. [1] is a work from my university which I like to point towards. Yes this lower accuracy, yes you won't be able to do critical things with this anymore but that's the price you pay for safety. Not getting hyped about CNNs and adopting a fail-safe approach that is only augmented with NNs is (in my humble opinion) why Waymo has 30k miles between disengagements [2] now while Tesla is either going to make me eat this post (not impossible given Andrej Karpathy is much smarter than me) OR are trying to hide the fact that they will never have anything resembling FSD by avoiding to report numbers. [3] is another paper I recommend for anyone wanting to USE CNNs for applications and wants to calmly assess the risk associated with adversarial examples Now, from a research perspective they are fascinating, they highlight weaknesses in our ability to train models,are a valuable tool to train robust CV models in the low data regime and have paved the way towards understanding the types of features learned in CNNs (our neighbours just released this [4] which in my eyes debunked a previously held assumptions that CNNs have a bias towards high frequency features, which is a fascinating result). But for anyone wanting to use the models, you shouldn't worry about them because you shouldn't be using the models for anything critical in a place where an attack can happen anyway. The same way that "what is the best way to encrypt our users passwords so they cannot be stolen" is the wrong way to approach passwords "how can we make the deep neural network in the application critical path robust against targeted attack" is (for now) the wrong way to approach CV. [1] https://arxiv.org/abs/1802.06806 https://arxiv.org/abs/1802.06806 [2] https://www.forbes.com/sites/bradtempleton/2021/02/09/california-robocar-disengagement-reports-reveal-about-tesla-autox-apple-others/ https://www.forbes.com/sites/bradtempleton/2021/02/09/califo... [3]https://arxiv.org/abs/1807.06732 https://arxiv.org/abs/1807.06732 [4] https://proceedings.neurips.cc/paper/2020/hash/1ea97de85eb634d580161c603422437f-Abstract.html https://proceedings.neurips.cc/paper/2020/hash/1ea97de85eb63...
p2p_astroturf 5y ago
Ever since I had the misfortune of learning about hacker kids wanting self-driving cars, I've been saying you can literally put a poster on the side of the road and every car that comes by it will crash. Seems like I'm on the right track. Software has edge cases. Every software engineer knows this. >The second-most frequent complaint is that the adversarial image attack is ‘white box’, meaning that you would need direct access to the training environment or data. The training data will be leaked. Companies are very bad at classifying what is and isn't private information that they need to keep secret. But anyway you probably don't even need the training data.
karaterobot 5y ago
I don't like the idea that self-driving cars were in use, and suddenly became compromised, but only because people could die as a result. Other than that, I can't think of a situation where I would be upset if computer vision was considered too unreliable to be misused by governments or abused by advertisers, which is how it'll go otherwise.
comfysocks 5y ago
When we design classical control systems, the performance limitations are well understood. If we do not exceed the limits, we expect the system to be well behaved. By contrast, DNN/CNN based systems can be a bit of a black box. We can only evaluate performance empirically, not analytically. It is difficult to know where in the input space the failure modes lie. It is difficult to then build the larger system around it, because you do not know were the keep-out zones are. I think a study of the failure modes of CNNs shouldn't be interpreted as an all-or-nothing evaluation of the technology as a whole, but rather a step towards gaining some confidence regarding its reliability. A lot more work needs to be done before I will trust it to drive my car. Regarding the use of CNN's for autonomous driving, I think it is insane that people are trying to do this by trying to solve a VERY hard problem, i.e. making a machine that can do what the human brain does. Your neural net does not have enough labels to account for all possible scenarios. Instead, it would make more sense to redesign the infrastructure in a way that bounds the problem space. The current system is designed for human drivers. We should make a system that is easy to interpret for both human and machine drivers. Of course this infrastructure would benefit all car makers, not just the first mover.
- mgraczyk 5y ago
  The major players are not doing it the way you describe. Tesla's driving system is not a giant model trained to imitate a human brain. There are separate perception, planning, and control algorithms.
  - comfysocks 5y ago
    These other algorithms would be part of the "larger system built around it" that I mention. This larger system has the task of doing what human judgement does.
    - mgraczyk 5y ago
      The "larger system around it" is not trained to "do what a human brain does". That's called "behavioral cloning". The major players do not do that. That's why the systems are more robust than you probably think to failures in perception. It's also why these systems sometimes fail in ways that humans would never fail.

178 points by Hard_Space 5y ago

Language models have the same defect - they are quite brittle and susceptible to black-box adversarial attacks ( eg arxiv.org/abs/1907.11932 )

bckr 5y ago

I like the cnn-adversarial aesthetic. Psychedelic blobs (sans the swirly demon dog faces), flower paintings and vinyl stickers of organic figures everywhere!

corey_moncure 5y ago

This is bad news for safety critical computer vision systems like Tesla vision.

amelius 5y ago
Don't worry, they'll just "convince" the regulators to ignore these problems.
- pixelgeek 5y ago
  Maybe put a sticker on the conference table that distracts them?

frazbin 5y ago

The really scary thing is that this could be used as an excuse to hide production ML models and even the tech used to generate them. Sounds like we can expect the state-of-the-art AI techniques to be jealously guarded eventually. I guess optimism on the ground is enough to have prevented that so far, but once the scales tip away from sharing and towards exploitation.. well, we know it's largely a one-way process on the 1 decade time scale. Is this the chilling effect that will bring us into the next AI winter?

jowday 5y ago
> Sounds like we can expect the state-of-the-art AI techniques to be jealously guarded eventually. This isn’t an eventuality, it’s the current state of the industry.
- frazbin 5y ago
  Hm is that really true? I thought that there was quite a lot of sharing from industry leaders at the research paper and dataset level, and that these could be used imitate production systems given some hacking. Kinda seemed like the majors were enjoying the benefits of the scrutiny afforded to public scientific research, while keeping their monopoly confined to the silicon/speed/throughput axis. Hence all the free AI software toolkits and also high priced specialty hot-off-the-wafer chips you'll never get.

laura_g 5y ago

The literature has pretty consistently shown that adversarial examples can be found with only black box access (even with truncated prediction vectors), robustness methods are primarily a cat-and-mouse game between attackers and defenders, and the existence of adversarial examples is likely inevitable (https://arxiv.org/pdf/1809.02104.pdf https://arxiv.org/pdf/1809.02104.pdf). The big question that remains is - so what? There's exceedingly few use cases where the existence of adversarial examples causes a security threat. There's a lot of research value in understanding adversarial examples and what that tells us about how models learn, generalize, and retain information, but I am not convinced that these attacks pose a threat remotely close to the amount of attention given.

owlbite 5y ago
Self driving cars seem like a dangerous threat vector if an adversarial image can be deployed in such a way as to cause them to commit dangerous maneuvers on demand.
- laura_g 5y ago
  I completely agree, but that's a very big "if". I'm not terribly familiar with autonomous vehicle driving systems, but my passing understanding is that there are multiple components working together that help make predictions, and these systems do not rely on any single point of failure. The classic example of a sticker on a stop sign is, in my view, more of a dramatization than a real threat surface. Designing an adversarial perturbation on a sticker that can cause misclassifications from particular angles and lighting conditions is possible, but that alone won't cause a vehicle to ignore traffic situations, pedestrians, and other contextual information. Plus, if I wanted to trick a self driving vehicle into not stopping at an intersection, it would be much easier and cheaper for me to just take the stop sign down :)
- Isinlor 5y ago
  There is plenty of natural "adversarial examples" to worry about. Like billboard with stop sign on it. https://youtu.be/-OdOmU58zOw?t=149 https://youtu.be/-OdOmU58zOw?t=149

mgraczyk 5y ago

As somebody who works on computer vision, my general take on these things is that adversarial examples are like poison. It would be fairly easy to add poison to a water supply or the air intake of a large building and kill a large number of people. This rarely happens though. It's ok that water sources, buildings, and people aren't completely immune to poison. The safety requirement isn't that poison can't hurt. Instead, we rely on weaker protections. We try to make known poisons hard to make, we try to track people who could make them, and we try to make it hard to deliver poison. I believe the same will be true of adversarial examples for vision (and language) models. We can try to make them hard to make, hard to posses anonymously, and hard to deliver. I think this will be much easier with computer vision than with poison, so I'm not worried about it. For example, consider the case of pasting a sticker on a speed limit sign that causes Teslas to swerve off the road. Governments should protect people from this in multiple ways, similarly to how they protect us from poison: 1. People who post these stickers should go to prison. 2. People who create and distribute these stickers knowing their purpose should go to prison. 3. Tesla should be civilly liable for cases where preventing such an incident was possible with known technology. 4. Roads should be modified over time to make it more difficult to do this attack. I think some combination of the above would be enough to make society as comfortable with adversarial example risk as we are with poison risk.

deleted 5y ago
[deleted]
voakbasda 5y ago
Your proposed laws do not cut out any exemption for research and experimentation, either with existing systems or potential new ones. This level of regulation would create an impossibly high barrier to entry and ensure that only the established players would remain in the marketplace. The last thing that I want to see is yet more regulatory capture, particularly in an industry that has yet to establish a reasonable baseline of success.
- mgraczyk 5y ago
  None of the things I listed would affect research. Researchers shouldn't be posting these on public highways, and researchers shouldn't be distributing them with the intent to cause harm.

ravi-delia 5y ago

It seems to me theres a difference between attacks that carefully craft an image that slips through the cracks, and an attack that basically exploits the fact that without context, it's hard to figure out what single item is important. If I took a picture of a conch shell on top of my keyboard and sent it to someone, no one would think I was just showing off my keyboard! They'd assume, correctly, that my desk was a mess and I didn't feel like finding a clear surface. That's not to say that either attack is less harmful than the other! If you train an image classifier to find bikers, it's not really wrong or right to say that a picture of a biker qualifies. But if a car stops lest it run over a painted bike on the road, that's obviously bad. The problem is that you aren't trying to recognize bikers, you're trying to avoid obstacles. We just don't train well for that.

pixelgeek 5y ago
I don't think that the word 'train' should be used for these systems. We feed then reams of data and effectively cull the ones that don't work but the critical problem is that we judge the effectiveness of an ML system and we actually do know what the ML systems is supposed to be looking for. We feed a system a series of images of bikes and then select the ones that can pick out a bike but we don't know how the bike is being chosen. We know it is picking out bikes but we have no way to predict if the system is picking out bikes or picking out a series of contrasting colour and shadow shapes and could easily be thrown off by anything that contains the same sort of data.
- NavinF 5y ago
  Thank you for an accurate ELI5 description of the human visual system. Dunno what this “ML” is, I assume it’s some part of the brain? It’s too bad you can’t analyze brains like you can with neural networks. It’s trivial to visualize filters and feature maps or to create heatmaps showing which pixels (shadow shapes?) in a specific image affect the classification output and why (contrasting color?).
  - d110af5ccf 5y ago
    The issue is that a human driver is much more than just a visual cortex. > which pixels (shadow shapes?) in a specific image affect the classification output and why (contrasting color?) Sure, you can watch the Rube Goldberg machine work. It doesn't mean you understand why it works on a conceptual level or have any hope of rigorously quantifying when and how it could fail.

pixelgeek 5y ago

Maybe this is a good example of why ML systems shouldn't be used? Ultimately we don't know how the networks that get created actually make decisions so doesn't that make protecting them from attacks like this impossible?

SavantIdiot 5y ago

There is a fundamental disconnect between what deep vision models can do and what is expected of them. On the one hand, there is a very good reason why mean-average-precision is used to assess detection-classification models: because even people make mistakes. On the other hand, we need to apply the use of these forever imperfect models with care, context, and redundancy. This is why engineers add a dozen other input types to ADAS systems in addition to vision (sonar, lidar, mesh computing, etc). This is why regulation is needed, to prevent less rigorous products from making their way into situations where the can be easily compromised, or worse, deadly.

dontreact 5y ago

If you really wanted to crash cars by altering their visual input, why would you bother with all this complexity? Why not just actually swap the road sign? Why does the existence of these attacks change the threat landscape at all? If people are already not doing "dumb" attacks like just changing/removing road signs why would they start doing them? The risk of messing with road signs and throwing off autonomous vehicles really has less to do with adversarial image attacks and more to do with envisioning an impractically brittle system where the decision to stop is based purely on presence/absence of a stop sign and not on a system that has a more general sense of collision-avoidance and situational awareness (like humans do).| Stepping back more generally, I have still never seen a case where the undetectability of adversarial attacks actually means there is a practical difference to security or safety. If you really think through the impact in the real world, usually the risk is already there: you can just change the input to the image and get bad results, it doesn't affect much that the image is imperceptibly changed. Because the whole point of using an automated vision system is usually that you want to avoid human eyes on the problem.

goatlover 5y ago
> Why not just actually swap the road sign? Because you have to physically do it, as opposed to hacking from anywhere else on the planet. > not on a system that has a more general sense of collision-avoidance and situational awareness (like humans do). Are vision systems to that point yet when it comes to driving vehicles? > Because the whole point of using an automated vision system is usually that you want to avoid human eyes on the problem. And the point of hacking an automated system is that it's easier to do that remotely than to cause a human to crash locally.
- JoshuaDavid 5y ago
  > Because you have to physically do it, as opposed to hacking from anywhere else on the planet. My impression is that the adversarial image attacks in question involve physically placing a sticker on something which will be in the view of self-driving cars -- it's not a remote exploit.

JohnFen 5y ago

I take a large measure of hope from this. I see facial recognition as a large societal threat, and it's nice to know that a defense is possible.

zcw100 5y ago
Image cloaking https://sandlab.cs.uchicago.edu/fawkes/ https://sandlab.cs.uchicago.edu/fawkes/

dhosek 5y ago

Whenever these discussions come up, I often think of a time I was driving on a two-lane road in rural Ohio in the 90s and at one point the center stripe curved into the lane (presumably because the driver of the striping truck pulled over without turning off the striper) and I started to curve off the road thanks to subconscious interpretation of the cue. I caught myself before I drove into a corn field, but human vision systems are also susceptible to these sorts of problems.

donkarma 5y ago
You caught yourself, didn't you?

OldHand2018 5y ago

I see a completely different attack vector here. Lawyers. If you are selling a product or service that has been trained on a dataset that contains copyrighted photos you don't have permission to use and I can "prove it" enough to get you into court and into the discovery phase, you are screwed. I'll get an injunction that shuts you down while we talk about how much money you have to pay me. And lol, if any of those photos of faces was taken in Illinois, we're going to get the class-action lawyers involved, or bury you with a ton of individual suits from thousands of people. That link at the bottom about a "safe harbor" you get from using old datasets from the Wild West is not going to fly when you start selling.

laura_g 5y ago
This would be membership inference attacks - https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=7958568&casa_token=edBGtFGrJlwAAAAA:eLObvr85G6__HNWXaAWmdqcK-ljtTy_eniFBv4BM3EBojJ7pza3n-GxsnOmqC8GFjj5ZCzZb https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=7958568...
- OldHand2018 5y ago
  Oh excellent. But of course the key addition is handing off this information to lawyers who use it to shut you down and/or extract money from you. If you are using some torrent of a dataset, nobody is indemnifying you, and once you get to the discovery phase of a lawsuit, they are going to know that you intentionally grabbed a dataset you knew you shouldn't have had access to. Treble damages!
NoGravitas 5y ago
I dunno, Microsoft seem to think they can get away with training autocomplete on copyrighted source code that they don't have permission to use.
KingMachiavelli 5y ago
IIRC simply training on copyrighted material is completely fine or at least you can claim fair use. As long as the market of the copyrighted material is not 'AI data training set' then it should be OK. Essentially scraping images from the internet is OK but using a pirated copyrighted commercial AI data training set is not. (Fair use doesn't necessarily exclude use for a commercial/sold product.) But if the AI model just spits out copyrighted material verbatim then that is still owned by the actual copyright holder.

deleted 5y ago

[deleted]

m3kw9 5y ago

I can propose a non-trivial solution to these problems, that is to have a data cleaner average and ignore certain data like how humans does it. Humans would ignore everything else but the face and maybe the body, and also we don’t examine someone’s follicles either, we basically average.

adolph 5y ago

"Adversarial" communication with a CV model inference process isn't necessarily an attack because it is unintended by the humans associated with the process. It is more akin to using the full range of an API that uses radiation instead of a network port. It could be used to stage a protest by stopping or slowing cars on a freeway or call attention to deteriorating infrastructure by inducing the car to go over potholes instead of avoiding them. Maybe a neighborhood could self-implement traffic calming measures that don't apply to emergency vehicles.

voldacar 5y ago
If you are trying to make my car go over potholes without my consent, or in any way do something that I don't want it to, that is adversarial behavior. You are my adversary.

whatever1 5y ago

Is traditional computer vision less susceptible to these? Since the features are human crafted, it sounds to me that the risk would be much lower.

ativzzz 5y ago
I would imagine more because those features are probably easier to reverse engineer. Plus traditional CV is weaker in generalized scenarios and is pretty easy to trick or throw off.

draw_down 5y ago

The "adversarial" language around this feels like blame-shifting. If your algorithm thinks a flower is Barack Obama, it's not the flower's fault.

quirkot 5y ago

The key point I see in this is that, given the current ecosystem, attacks are systemic. Plus, given the nature of ML training and datasets, it's expensive to bug fix an attack, if it's even possible. This right here is the real underlying long term danger: > the most popular CV datasets are so embedded in development cycles around the world as to resemble software more than data; software that often hasn’t been notably updated in years

windows2020 5y ago

Some people think that a self driving car needs to perform just well enough to meet some average metric at which point its shortcomings are considered justified. Would you drive with a human that could fail so spectacularly? As least they can rationalize their decision. My opinion is this: we're missing something big. Current weak AI strategies may be sufficient in some domains, but until the essence of consciousness can be defined, general AI (and AI I would trust my life with) is out of the question.

igorkraw 5y ago

I work in this field, I have a project specifically on adversarial examples and I have a strong opinion on this. I personally think worrying about adversarial examples in real life production systems is like worrying about getting the vanilla linux kernel to perform RT critical tasks. It is fundamentally not a burden you should put on that one component alone and is a problem you can only solve with a system approach. And if you do that, it is for all practical purposes already solved: apply multiple, random perturbations to the input, project your perturbed version onto a known,safe image space, and establish consensus. [1] is a work from my university which I like to point towards. Yes this lower accuracy, yes you won't be able to do critical things with this anymore but that's the price you pay for safety. Not getting hyped about CNNs and adopting a fail-safe approach that is only augmented with NNs is (in my humble opinion) why Waymo has 30k miles between disengagements [2] now while Tesla is either going to make me eat this post (not impossible given Andrej Karpathy is much smarter than me) OR are trying to hide the fact that they will never have anything resembling FSD by avoiding to report numbers. [3] is another paper I recommend for anyone wanting to USE CNNs for applications and wants to calmly assess the risk associated with adversarial examples Now, from a research perspective they are fascinating, they highlight weaknesses in our ability to train models,are a valuable tool to train robust CV models in the low data regime and have paved the way towards understanding the types of features learned in CNNs (our neighbours just released this [4] which in my eyes debunked a previously held assumptions that CNNs have a bias towards high frequency features, which is a fascinating result). But for anyone wanting to use the models, you shouldn't worry about them because you shouldn't be using the models for anything critical in a place where an attack can happen anyway. The same way that "what is the best way to encrypt our users passwords so they cannot be stolen" is the wrong way to approach passwords "how can we make the deep neural network in the application critical path robust against targeted attack" is (for now) the wrong way to approach CV. [1] https://arxiv.org/abs/1802.06806 https://arxiv.org/abs/1802.06806 [2] https://www.forbes.com/sites/bradtempleton/2021/02/09/california-robocar-disengagement-reports-reveal-about-tesla-autox-apple-others/ https://www.forbes.com/sites/bradtempleton/2021/02/09/califo... [3]https://arxiv.org/abs/1807.06732 https://arxiv.org/abs/1807.06732 [4] https://proceedings.neurips.cc/paper/2020/hash/1ea97de85eb634d580161c603422437f-Abstract.html https://proceedings.neurips.cc/paper/2020/hash/1ea97de85eb63...

p2p_astroturf 5y ago

Ever since I had the misfortune of learning about hacker kids wanting self-driving cars, I've been saying you can literally put a poster on the side of the road and every car that comes by it will crash. Seems like I'm on the right track. Software has edge cases. Every software engineer knows this. >The second-most frequent complaint is that the adversarial image attack is ‘white box’, meaning that you would need direct access to the training environment or data. The training data will be leaked. Companies are very bad at classifying what is and isn't private information that they need to keep secret. But anyway you probably don't even need the training data.

karaterobot 5y ago

I don't like the idea that self-driving cars were in use, and suddenly became compromised, but only because people could die as a result. Other than that, I can't think of a situation where I would be upset if computer vision was considered too unreliable to be misused by governments or abused by advertisers, which is how it'll go otherwise.

comfysocks 5y ago

When we design classical control systems, the performance limitations are well understood. If we do not exceed the limits, we expect the system to be well behaved. By contrast, DNN/CNN based systems can be a bit of a black box. We can only evaluate performance empirically, not analytically. It is difficult to know where in the input space the failure modes lie. It is difficult to then build the larger system around it, because you do not know were the keep-out zones are. I think a study of the failure modes of CNNs shouldn't be interpreted as an all-or-nothing evaluation of the technology as a whole, but rather a step towards gaining some confidence regarding its reliability. A lot more work needs to be done before I will trust it to drive my car. Regarding the use of CNN's for autonomous driving, I think it is insane that people are trying to do this by trying to solve a VERY hard problem, i.e. making a machine that can do what the human brain does. Your neural net does not have enough labels to account for all possible scenarios. Instead, it would make more sense to redesign the infrastructure in a way that bounds the problem space. The current system is designed for human drivers. We should make a system that is easy to interpret for both human and machine drivers. Of course this infrastructure would benefit all car makers, not just the first mover.

mgraczyk 5y ago
The major players are not doing it the way you describe. Tesla's driving system is not a giant model trained to imitate a human brain. There are separate perception, planning, and control algorithms.
- comfysocks 5y ago
  These other algorithms would be part of the "larger system built around it" that I mention. This larger system has the task of doing what human judgement does.
  - mgraczyk 5y ago
    The "larger system around it" is not trained to "do what a human brain does". That's called "behavioral cloning". The major players do not do that. That's why the systems are more robust than you probably think to failures in perception. It's also why these systems sometimes fail in ways that humans would never fail.