3 ms·

This seems like it could dramatically worsen overfitting-like effects for algorithms like CNNs for image processing, where surface statistics of the available d

by mlthoughts2018 8y ago

This seems like it could dramatically worsen overfitting-like effects for algorithms like CNNs for image processing, where surface statistics of the available data set seem to be more responsible for the learned model than any type of “semantic” understanding. If you prespecify what data augmentation you would do, like preregistering the details of a clinical trial, you’ll be less susceptible to a spurious result from this. It seems like especially things like color distribution manipulation would have a potentially very adverse effect that counters any gains from clamping the supervised learning to be “robust” to that color variation. I’m thinking in the spirit of: < https://arxiv.org/abs/1711.11561 https://arxiv.org/abs/1711.11561 >.

nafizh 8y ago
Don't exactly remember the paper, but many of the claims in the paper you mentioned were seemed to be proven false. As such, augmentation like this should, if any, reduce overfitting.
- jjn2009 8y ago
  I agree, as long as you do out of sample testing correctly then using this as a benchmark for finding the correct transformations to perform should reduce overfitting. That said I can see how if done incorrectly you could easily end up overfitting instead.
- mlthoughts2018 8y ago
  Could you provide a link to where the Bengio paper was “disproven”? I would be quite surprised to learn that, especially for the experimental result they have with the low-pass Fourier filter on the training set, and also because the Bengio paper is quite recent.
tlb 8y ago
An example they give is that it doesn't apply shear transforms to CIFAR-10, because there aren't sheared examples in the (test) data. That makes it less indicative of real-world vision performance, given that human vision is fairly robust to shear. An opposite sort of strategy is to apply every augmentation that doesn't increase human error rate on a test sample.
- mlthoughts2018 8y ago
  Sure, I think the approach could work successfully in some cases, but I think specifically allowing it to learn transformations of the input color space would be quite problematic — you are essentially giving it more degrees of surface statistics freedom with which to data mine a better looking overfitting metric during validation steps.

nafizh 8y ago

Don't exactly remember the paper, but many of the claims in the paper you mentioned were seemed to be proven false. As such, augmentation like this should, if any, reduce overfitting.

jjn2009 8y ago
I agree, as long as you do out of sample testing correctly then using this as a benchmark for finding the correct transformations to perform should reduce overfitting. That said I can see how if done incorrectly you could easily end up overfitting instead.
mlthoughts2018 8y ago
Could you provide a link to where the Bengio paper was “disproven”? I would be quite surprised to learn that, especially for the experimental result they have with the low-pass Fourier filter on the training set, and also because the Bengio paper is quite recent.

tlb 8y ago

An example they give is that it doesn't apply shear transforms to CIFAR-10, because there aren't sheared examples in the (test) data. That makes it less indicative of real-world vision performance, given that human vision is fairly robust to shear. An opposite sort of strategy is to apply every augmentation that doesn't increase human error rate on a test sample.

mlthoughts2018 8y ago
Sure, I think the approach could work successfully in some cases, but I think specifically allowing it to learn transformations of the input color space would be quite problematic — you are essentially giving it more degrees of surface statistics freedom with which to data mine a better looking overfitting metric during validation steps.