3 ms·

There's a theme in this discussion that ML operators will just train new models on adversarially perturbed data. I don't think this is necessarily true at all!

There's a theme in this discussion that ML operators will just train new models on adversarially perturbed data. I don't think this is necessarily true at all! The proliferation of tools like this and the "LowKey" paper/tool linked below (an awesome paper!) will fundamentally change the distribution of image data that exists. I think that widespread usage of this kind of tool should trend towards increasing the irreducible error of various computer vision tasks (in the same way that long term adoption of mask wearing might change the maximum accuracy of facial recognition). Critically, while right now the people who do something like manipulate their images will probably be very privacy conscious or tech-interested people, tools like this seriously lower the barrier to entry. It's not hard to imagine a browser extension that helps you perturb all images you upload to a particular domain, or something similar.

yumraj 6y ago
> It's not hard to imagine a browser extension that helps you perturb all images you upload to a particular domain, or something similar. Ideally I’d like to see something like this be part of the camera filter itself. Why can’t Apple, if they choose to do so, just add something like this as part of their camera app itself?
- LockAndLol 6y ago
  If it's opensource, there's no need to wait on the business interests of a trillion dollar company to align with your wishes. Camera app developers can be made aware of it and add it to their apps. If there are app developers on HN, they can create pull requests to their favorite apps and add the feature. That's the power of opensource.
  - yumraj 6y ago
    Yes, for you and me, but not for ordinary folks..
lmeyerov 6y ago
The adversarial ML arms race seems similar to the rest of the security/privacy arms race, where these endeavors will make recognition stronger, not weaker, similar to how any other manual red team attacks ultimately get (a) automated and (b) incorporated into blue team's automatic defenses. Hard to see why that wouldn't be the case, esp. for techniques that are general, vs. exploiting bugs in individual models. As long as a person can quickly tell the difference, it's in the grasp of deep learning for ~perception problems, and the economics of the arms race determines the rest of what happens when..
- nickvincent 6y ago
  Definitely agree that there will be cases in which a computer vision operator ships features specifically intended to create a new automatic defense. One thing that seems unique to technologies that are mostly just statistical learning is that each new manipulation approach can basically widen the distribution of possible inputs. In particular, I'm thinking that as more obfuscation and protest technologies are made public like this, the distribution of "images of faces available for computer vision training" becomes more complex. That is to say, whenever a adversarial tool creates a combination of pixels that's never been see before, if that "new image" can't be reduced back to a familiar image via de-noising or pre-processing, the overall difficulty of computer vision tasks increases. All a long winded way of saying, I think for ML systems, there's a unique opportunity to "stretch the distribution of inputs" that may not exist for other security arms races. Totally agree that economics of the arms race(s) will a huge factor in determining how much an impact obfuscation and protest can have.
  - lmeyerov 6y ago
    yeah maybe sql injection is a good analogy: bug bounties for it, then automated fuzzing, and now built-in to frameworks. there a companies and oss here, so building robustness into training sets, tf, is normal . I'm not sure if bet on a new Coverity wrt VC $, but definitely r&d and smaller groups

yumraj 6y ago

> It's not hard to imagine a browser extension that helps you perturb all images you upload to a particular domain, or something similar. Ideally I’d like to see something like this be part of the camera filter itself. Why can’t Apple, if they choose to do so, just add something like this as part of their camera app itself?

LockAndLol 6y ago
If it's opensource, there's no need to wait on the business interests of a trillion dollar company to align with your wishes. Camera app developers can be made aware of it and add it to their apps. If there are app developers on HN, they can create pull requests to their favorite apps and add the feature. That's the power of opensource.
- yumraj 6y ago
  Yes, for you and me, but not for ordinary folks..

lmeyerov 6y ago

The adversarial ML arms race seems similar to the rest of the security/privacy arms race, where these endeavors will make recognition stronger, not weaker, similar to how any other manual red team attacks ultimately get (a) automated and (b) incorporated into blue team's automatic defenses. Hard to see why that wouldn't be the case, esp. for techniques that are general, vs. exploiting bugs in individual models. As long as a person can quickly tell the difference, it's in the grasp of deep learning for ~perception problems, and the economics of the arms race determines the rest of what happens when..

nickvincent 6y ago
Definitely agree that there will be cases in which a computer vision operator ships features specifically intended to create a new automatic defense. One thing that seems unique to technologies that are mostly just statistical learning is that each new manipulation approach can basically widen the distribution of possible inputs. In particular, I'm thinking that as more obfuscation and protest technologies are made public like this, the distribution of "images of faces available for computer vision training" becomes more complex. That is to say, whenever a adversarial tool creates a combination of pixels that's never been see before, if that "new image" can't be reduced back to a familiar image via de-noising or pre-processing, the overall difficulty of computer vision tasks increases. All a long winded way of saying, I think for ML systems, there's a unique opportunity to "stretch the distribution of inputs" that may not exist for other security arms races. Totally agree that economics of the arms race(s) will a huge factor in determining how much an impact obfuscation and protest can have.
- lmeyerov 6y ago
  yeah maybe sql injection is a good analogy: bug bounties for it, then automated fuzzing, and now built-in to frameworks. there a companies and oss here, so building robustness into training sets, tf, is normal . I'm not sure if bet on a new Coverity wrt VC $, but definitely r&d and smaller groups