5 ms·

This is really interesting, but points out a key importance in training neural nets, which is to design your dataset and training to maximize generalization. Fo

This is really interesting, but points out a key importance in training neural nets, which is to design your dataset and training to maximize generalization. For example, in the case of training a neural network for something that is highly safety critical, like an autonomous vehicle, it's important for vehicle and pedestrian detection to be as generalized as possible. In order to achieve high confidence in all sorts of lighting, weather, and angle conditions, the training data is augmented, which means it is manipulated randomly with color, saturation, contrast, blurring, shifting, mirroring, adding noise, etc. So adding random pixels, to potentially protect against dead pixels is also a great idea.

legohead 9y ago
Walk around in the city with a giant yellow square costume and observe the mayhem...
- blackRust 9y ago
  Dressing up as traffic signs might become a thing...
  - mattigames 9y ago
    I'm pretty sure that's already illegal on many countries.
    - icc97 9y ago
      https://xkcd.com/1958/ https://xkcd.com/1958/
- stochastic_monk 9y ago
  I recently saw a yellow banana-man in a moshpit at a metal show. Don’t put breaking expectations past humans. We are adversarial by nature.
tomrod 9y ago
Jumping on top comment (which I completely agree with) to ask: Why wouldn't a K-fold cross validation enable catching this? I'm curious if the attack adds doubt, in that the prediction algorithm is _close_ to truth but gets confused (likelihood of horse slightly less than dog), versus incorrect certitude (the horse is definitely a dog). One could then attach a weighting, perhaps based on max RGB/CYMK vector norm between two pixels across the image, to the folds' difference in top two certitudes. I don't know, something like that.
- eanzenberg 9y ago
  Because its too computationally expensive. Attacks like these are found when you have access to the model, take an inout image and change its pixel values and see when an anomoly occurs. It’s much faster scoring N images vs. building N models. How do you combat it? Well that’s an open research question. IMO the most promising techniques make the system harder to attack (ensemble models, more complex models, randomizing the input slightly and dropping outliers) but its not a guarantee. Like in security, it would be great to verify a model is safe to 200 years of brute force search for attack, or whatever it may be.
  - alexcnwy 9y ago
    What about applying dropout to the input image
- nightcracker 9y ago
  I don't believe that many people are using K-fold cross validation at all with deep learning, as the computational overhead is massive.
  - tomrod 9y ago
    I must be misspeaking then. I'm not suggesting it run during training, but run the classification algorithm X number of times over an input image with chunks areas removed/suppressed from the data.
argonaut 9y ago
It also doesn't work for adversarial attacks. "Adversarial training" helps somewhat but is still very susceptible to adversarial attacks.
anonytrary 9y ago
It would be interesting to try and generate this kind of data from 3D models: entire intricate scenes, where the camera can go around and take thousands of images in various lightings, weathers and vantage points. ICBW, but I feel like this modeling may have been tried and didn't translate well to real-life at the time. I can't see why it wouldn't be possible with enough modeling and physics precision, though.
Jaxan 9y ago
You make it sound a bit too easy “just add some random colours”. But there are theoretical hardness results from the 90s on learning NN. It is so-called inherently unpredictable. These hardness results became a bit obsolete by big data. I believe that what we see happening with adversarial inputs is a shadow of such hardness results.
scoot 9y ago
This made me realize that Waymo's training data from Google street view imaging and recaptcha based labeling, whilst noisy in many natural ways, is exclusively daytime data. Seems a glaring hole.
haraldurt 9y ago
There have been some fascinating steps taken towards that direction, e.g. [0] and its related work. There they take the approach of transferring input images back and forth domains (think, a street imaged in summer transferred to its winter manifestation and back or a synthetic GTA image to real-world and back are the examples in [0]). Doing this while simultaneously holding the semantic content of the input unchanged with a GAN-type strategy seems to be a way to coerce the neural net's internal representations to capture what we want them to instead of idiosyncrasies of the dataset. [0] https://arxiv.org/abs/1711.03213 https://arxiv.org/abs/1711.03213
logicallee 9y ago
This is a truly excellent point. It applies to people, too: what will someone do if they are driving through the snow for the first time EVER, if they had lived in a temperate climate and just moved? They will go very slow and perhaps it is not an exaggeration to say they will re-learn. Humans are learning constantly. In fact if there were some natural disaster (lava flow) and a human saw another car drive across some set lava on the way out of town (as more lava is rushing toward them) then a human will go ahead and follow, after seeing the other car make it through. If they see another car try to go across but get stuck on, they might take a detour and go find some intact bridge or other way to pass. Actually, what you call "general" might be as much as general intelligence...
icc97 9y ago
Shifting and mirroring is already a standard thing in papers with crop-10 (top-left, top-right, middle, bottom-left, bottom-right) * 2 (reflection on the horizontal axis). Colour shifting is also done via PCA analysis. One thing that humans have is that young children watch the pages of a book turning, so see basic images at all extreme angles.
s3 9y ago
Ian Goodfellow and others have done a lot of great work exploring the properties of adversarial attacks. While all the things you’ve mentioned are great practices and important to avoid overfitting, adversarial attacks are not due to overfitting (nor are they random) and none of the things you described are sufficient as defenses.

legohead 9y ago

Walk around in the city with a giant yellow square costume and observe the mayhem...

blackRust 9y ago
Dressing up as traffic signs might become a thing...
- mattigames 9y ago
  I'm pretty sure that's already illegal on many countries.
  - icc97 9y ago
    https://xkcd.com/1958/ https://xkcd.com/1958/
stochastic_monk 9y ago
I recently saw a yellow banana-man in a moshpit at a metal show. Don’t put breaking expectations past humans. We are adversarial by nature.

tomrod 9y ago

Jumping on top comment (which I completely agree with) to ask: Why wouldn't a K-fold cross validation enable catching this? I'm curious if the attack adds doubt, in that the prediction algorithm is _close_ to truth but gets confused (likelihood of horse slightly less than dog), versus incorrect certitude (the horse is definitely a dog). One could then attach a weighting, perhaps based on max RGB/CYMK vector norm between two pixels across the image, to the folds' difference in top two certitudes. I don't know, something like that.

eanzenberg 9y ago
Because its too computationally expensive. Attacks like these are found when you have access to the model, take an inout image and change its pixel values and see when an anomoly occurs. It’s much faster scoring N images vs. building N models. How do you combat it? Well that’s an open research question. IMO the most promising techniques make the system harder to attack (ensemble models, more complex models, randomizing the input slightly and dropping outliers) but its not a guarantee. Like in security, it would be great to verify a model is safe to 200 years of brute force search for attack, or whatever it may be.
- alexcnwy 9y ago
  What about applying dropout to the input image
nightcracker 9y ago
I don't believe that many people are using K-fold cross validation at all with deep learning, as the computational overhead is massive.
- tomrod 9y ago
  I must be misspeaking then. I'm not suggesting it run during training, but run the classification algorithm X number of times over an input image with chunks areas removed/suppressed from the data.

argonaut 9y ago

It also doesn't work for adversarial attacks. "Adversarial training" helps somewhat but is still very susceptible to adversarial attacks.

anonytrary 9y ago

It would be interesting to try and generate this kind of data from 3D models: entire intricate scenes, where the camera can go around and take thousands of images in various lightings, weathers and vantage points. ICBW, but I feel like this modeling may have been tried and didn't translate well to real-life at the time. I can't see why it wouldn't be possible with enough modeling and physics precision, though.

Jaxan 9y ago

You make it sound a bit too easy “just add some random colours”. But there are theoretical hardness results from the 90s on learning NN. It is so-called inherently unpredictable. These hardness results became a bit obsolete by big data. I believe that what we see happening with adversarial inputs is a shadow of such hardness results.

scoot 9y ago

This made me realize that Waymo's training data from Google street view imaging and recaptcha based labeling, whilst noisy in many natural ways, is exclusively daytime data. Seems a glaring hole.

haraldurt 9y ago

There have been some fascinating steps taken towards that direction, e.g. [0] and its related work. There they take the approach of transferring input images back and forth domains (think, a street imaged in summer transferred to its winter manifestation and back or a synthetic GTA image to real-world and back are the examples in [0]). Doing this while simultaneously holding the semantic content of the input unchanged with a GAN-type strategy seems to be a way to coerce the neural net's internal representations to capture what we want them to instead of idiosyncrasies of the dataset. [0] https://arxiv.org/abs/1711.03213 https://arxiv.org/abs/1711.03213

logicallee 9y ago

This is a truly excellent point. It applies to people, too: what will someone do if they are driving through the snow for the first time EVER, if they had lived in a temperate climate and just moved? They will go very slow and perhaps it is not an exaggeration to say they will re-learn. Humans are learning constantly. In fact if there were some natural disaster (lava flow) and a human saw another car drive across some set lava on the way out of town (as more lava is rushing toward them) then a human will go ahead and follow, after seeing the other car make it through. If they see another car try to go across but get stuck on, they might take a detour and go find some intact bridge or other way to pass. Actually, what you call "general" might be as much as general intelligence...

icc97 9y ago

Shifting and mirroring is already a standard thing in papers with crop-10 (top-left, top-right, middle, bottom-left, bottom-right) * 2 (reflection on the horizontal axis). Colour shifting is also done via PCA analysis. One thing that humans have is that young children watch the pages of a book turning, so see basic images at all extreme angles.

s3 9y ago

Ian Goodfellow and others have done a lot of great work exploring the properties of adversarial attacks. While all the things you’ve mentioned are great practices and important to avoid overfitting, adversarial attacks are not due to overfitting (nor are they random) and none of the things you described are sufficient as defenses.