4 ms·

I don't believe that many people are using K-fold cross validation at all with deep learning, as the computational overhead is massive.

tomrod 9y ago
I must be misspeaking then. I'm not suggesting it run during training, but run the classification algorithm X number of times over an input image with chunks areas removed/suppressed from the data.
- yorwba 9y ago
  If I understand correctly, you want to modify an input image at application time to get multiple different classifications, and then compare them to find out how certain the model really is. While that would likely improve results a bit, it would also multiply the model runtime. That's why the other replies directly jump to talking about training data augmentation, since that can give you similar benefits without the runtime penalty. However, random augmentation can't fully protect against adversarial examples. The number of input variables is simply too large, and there are exponentially many directions in which they could be modified. Data augmentation can't cover all of them, and a single modification that confuses the model slightly can be amplified into an adversarial example that causes a total misclassification.
  - tomrod 9y ago
    > That's why the other replies directly jump to talking about training data augmentation, since that can give you similar benefits without the runtime penalty. Exactly. It's a fix that doesn't work, apparently, so that's why I'm thinking towards the runtime. > it would also multiply the model runtime. Predictably so, I would think? Such an approach could scale decently since it's not adding a dimension to the runtime, just a multiple.
    - yorwba 9y ago
      Predictably slow is still slow. You could parallelize it over multiple GPUs, but GPUs are expensive, so nobody is going to do that. Requiring lots of training resources is fine, but inference needs to be cheap. More problematic is that your approach isn't going to actually work, since CNNs are just too flexible (they can learn even completely random labels) and only generalize by accident. No input augmentation technique that doesn't cover every possible modification is going to be robust against adversarial examples, and getting that amount of coverage requires an exponential blowup in runtime. The adversary has the advantage of being able to choose one modification, while the model needs to defend against all of them.
- rjurney 9y ago
  There is a way to do this in Tensorflow, where you try to expand your training data images by altering them, moving them around, etc. However, this is not used with large training sets.
- d4l3k 9y ago
  This is very common. It's an easy way to improve accuracy on a model with a fixed amount of data. They typically crop the image as well as rotate and scale it in order to get a larger data set. It does make the model more robust, but doesn't seem to help much with finding adversarial examples in the model. Generating adversarial examples and training on that might be a better approach to solving this.
jgalt212 9y ago
not as massive as producing a bum model. And most of the overhead in ML is data acquisition, cleaning, normalization and tagging costs.

tomrod 9y ago

I must be misspeaking then. I'm not suggesting it run during training, but run the classification algorithm X number of times over an input image with chunks areas removed/suppressed from the data.

yorwba 9y ago
If I understand correctly, you want to modify an input image at application time to get multiple different classifications, and then compare them to find out how certain the model really is. While that would likely improve results a bit, it would also multiply the model runtime. That's why the other replies directly jump to talking about training data augmentation, since that can give you similar benefits without the runtime penalty. However, random augmentation can't fully protect against adversarial examples. The number of input variables is simply too large, and there are exponentially many directions in which they could be modified. Data augmentation can't cover all of them, and a single modification that confuses the model slightly can be amplified into an adversarial example that causes a total misclassification.
- tomrod 9y ago
  > That's why the other replies directly jump to talking about training data augmentation, since that can give you similar benefits without the runtime penalty. Exactly. It's a fix that doesn't work, apparently, so that's why I'm thinking towards the runtime. > it would also multiply the model runtime. Predictably so, I would think? Such an approach could scale decently since it's not adding a dimension to the runtime, just a multiple.
  - yorwba 9y ago
    Predictably slow is still slow. You could parallelize it over multiple GPUs, but GPUs are expensive, so nobody is going to do that. Requiring lots of training resources is fine, but inference needs to be cheap. More problematic is that your approach isn't going to actually work, since CNNs are just too flexible (they can learn even completely random labels) and only generalize by accident. No input augmentation technique that doesn't cover every possible modification is going to be robust against adversarial examples, and getting that amount of coverage requires an exponential blowup in runtime. The adversary has the advantage of being able to choose one modification, while the model needs to defend against all of them.
rjurney 9y ago
There is a way to do this in Tensorflow, where you try to expand your training data images by altering them, moving them around, etc. However, this is not used with large training sets.
d4l3k 9y ago
This is very common. It's an easy way to improve accuracy on a model with a fixed amount of data. They typically crop the image as well as rotate and scale it in order to get a larger data set. It does make the model more robust, but doesn't seem to help much with finding adversarial examples in the model. Generating adversarial examples and training on that might be a better approach to solving this.

jgalt212 9y ago

not as massive as producing a bum model. And most of the overhead in ML is data acquisition, cleaning, normalization and tagging costs.