4 ms·
leopard colors and texture is what the network actually was looking for during training Eh. The network is finding regular patterns. It doesn't know the differ
by nerdponx 8y ago
leopard colors and texture is what the network actually was looking for during training
Eh. The network is finding regular patterns. It doesn't know the difference between actual leopard parts and arbitrary noise in the image. The only reason we believe it's learning something about leopards is that we have fed the network lots of different kinds of leopards in lots of different contexts and positions.
But there are still some soft invariants that apply to images of leopards. These networks are complicated dynamical systems. Adding visual noise that is dissimilar to any kind of visual noise found naturally in leopard photos amounts to an input outside the normal operating range of the system. The effect of something like that, in general, is unpredictable. The fact that these examples exist shouldn't be that surprising. The fact that the output is quantized to one prediction or another only exacerbates the issue.
Is it because the network looks for the combination of certain parameters, and treats the exactness of a combination as the most important factor, more important than it's global context?
Probably. A CNN is an ensemble of many local representations.
So that the combination of the adversarially-modified pixels look like having the most exact ratios between each other, even though their ratios compared to the rest of the pixels is not on par at all - and the network decides that the most exact combination has the most information? Then, why isn't this easily combated by regularization and stuff like dropout?
Someone more knowledgeable than me can probably answer this. I'm not an expert in CNNs. But hopefully the rest is helpful.
- stochastic_monk 8y agoIn light of [0], there is both theoretical and experimental reason to believe that CNNs are mostly picking up fourier decomposition. [0] https://arxiv.org/pdf/1711.11561.pdf https://arxiv.org/pdf/1711.11561.pdf