3 ms·

The adversarial ML arms race seems similar to the rest of the security/privacy arms race, where these endeavors will make recognition stronger, not weaker, sim

The adversarial ML arms race seems similar to the rest of the security/privacy arms race, where these endeavors will make recognition stronger, not weaker, similar to how any other manual red team attacks ultimately get (a) automated and (b) incorporated into blue team's automatic defenses. Hard to see why that wouldn't be the case, esp. for techniques that are general, vs. exploiting bugs in individual models. As long as a person can quickly tell the difference, it's in the grasp of deep learning for ~perception problems, and the economics of the arms race determines the rest of what happens when..

nickvincent 6y ago
Definitely agree that there will be cases in which a computer vision operator ships features specifically intended to create a new automatic defense. One thing that seems unique to technologies that are mostly just statistical learning is that each new manipulation approach can basically widen the distribution of possible inputs. In particular, I'm thinking that as more obfuscation and protest technologies are made public like this, the distribution of "images of faces available for computer vision training" becomes more complex. That is to say, whenever a adversarial tool creates a combination of pixels that's never been see before, if that "new image" can't be reduced back to a familiar image via de-noising or pre-processing, the overall difficulty of computer vision tasks increases. All a long winded way of saying, I think for ML systems, there's a unique opportunity to "stretch the distribution of inputs" that may not exist for other security arms races. Totally agree that economics of the arms race(s) will a huge factor in determining how much an impact obfuscation and protest can have.
- lmeyerov 6y ago
  yeah maybe sql injection is a good analogy: bug bounties for it, then automated fuzzing, and now built-in to frameworks. there a companies and oss here, so building robustness into training sets, tf, is normal . I'm not sure if bet on a new Coverity wrt VC $, but definitely r&d and smaller groups

nickvincent 6y ago

Definitely agree that there will be cases in which a computer vision operator ships features specifically intended to create a new automatic defense. One thing that seems unique to technologies that are mostly just statistical learning is that each new manipulation approach can basically widen the distribution of possible inputs. In particular, I'm thinking that as more obfuscation and protest technologies are made public like this, the distribution of "images of faces available for computer vision training" becomes more complex. That is to say, whenever a adversarial tool creates a combination of pixels that's never been see before, if that "new image" can't be reduced back to a familiar image via de-noising or pre-processing, the overall difficulty of computer vision tasks increases. All a long winded way of saying, I think for ML systems, there's a unique opportunity to "stretch the distribution of inputs" that may not exist for other security arms races. Totally agree that economics of the arms race(s) will a huge factor in determining how much an impact obfuscation and protest can have.

lmeyerov 6y ago
yeah maybe sql injection is a good analogy: bug bounties for it, then automated fuzzing, and now built-in to frameworks. there a companies and oss here, so building robustness into training sets, tf, is normal . I'm not sure if bet on a new Coverity wrt VC $, but definitely r&d and smaller groups