5 ms·
Jumping on top comment (which I completely agree with) to ask: Why wouldn't a K-fold cross validation enable catching this? I'm curious if the attack adds doub
by tomrod 9y ago
Jumping on top comment (which I completely agree with) to ask:
Why wouldn't a K-fold cross validation enable catching this? I'm curious if the attack adds doubt, in that the prediction algorithm is _close_ to truth but gets confused (likelihood of horse slightly less than dog), versus incorrect certitude (the horse is definitely a dog). One could then attach a weighting, perhaps based on max RGB/CYMK vector norm between two pixels across the image, to the folds' difference in top two certitudes.
I don't know, something like that.
- eanzenberg 9y agoBecause its too computationally expensive. Attacks like these are found when you have access to the model, take an inout image and change its pixel values and see when an anomoly occurs. It’s much faster scoring N images vs. building N models. How do you combat it? Well that’s an open research question. IMO the most promising techniques make the system harder to attack (ensemble models, more complex models, randomizing the input slightly and dropping outliers) but its not a guarantee. Like in security, it would be great to verify a model is safe to 200 years of brute force search for attack, or whatever it may be.
- alexcnwy 9y agoWhat about applying dropout to the input image
- nightcracker 9y agoI don't believe that many people are using K-fold cross validation at all with deep learning, as the computational overhead is massive.
- tomrod 9y agoI must be misspeaking then. I'm not suggesting it run during training, but run the classification algorithm X number of times over an input image with chunks areas removed/suppressed from the data.
- yorwba 9y agoIf I understand correctly, you want to modify an input image at application time to get multiple different classifications, and then compare them to find out how certain the model really is. While that would likely improve results a bit, it would also multiply the model runtime. That's why the other replies directly jump to talking about training data augmentation, since that can give you similar benefits without the runtime penalty. However, random augmentation can't fully protect against adversarial examples. The number of input variables is simply too large, and there are exponentially many directions in which they could be modified. Data augmentation can't cover all of them, and a single modification that confuses the model slightly can be amplified into an adversarial example that causes a total misclassification.
- tomrod 9y ago> That's why the other replies directly jump to talking about training data augmentation, since that can give you similar benefits without the runtime penalty. Exactly. It's a fix that doesn't work, apparently, so that's why I'm thinking towards the runtime. > it would also multiply the model runtime. Predictably so, I would think? Such an approach could scale decently since it's not adding a dimension to the runtime, just a multiple.
- yorwba 9y agoPredictably slow is still slow. You could parallelize it over multiple GPUs, but GPUs are expensive, so nobody is going to do that. Requiring lots of training resources is fine, but inference needs to be cheap. More problematic is that your approach isn't going to actually work, since CNNs are just too flexible (they can learn even completely random labels) and only generalize by accident. No input augmentation technique that doesn't cover every possible modification is going to be robust against adversarial examples, and getting that amount of coverage requires an exponential blowup in runtime. The adversary has the advantage of being able to choose one modification, while the model needs to defend against all of them.
- rjurney 9y ago
- jgalt212 9y agonot as massive as producing a bum model. And most of the overhead in ML is data acquisition, cleaning, normalization and tagging costs.
- avaxzat 9y ago>I'm curious if the attack adds doubt, in that the prediction algorithm is _close_ to truth but gets confused (likelihood of horse slightly less than dog), versus incorrect certitude (the horse is definitely a dog). While I can't speak for this attack in particular, there exist algorithms that can fool a neural network into generating high-confidence incorrect predictions for images that are visually indistinguishable from ones on which the network performs just fine. That's the biggest issue with these adversarial images.