6 ms·

It's hard to see this as much anything but snake oil. While the technique allegedly allows you to target multiple models, it still requires you to know what adv

It's hard to see this as much anything but snake oil. While the technique allegedly allows you to target multiple models, it still requires you to know what adversarial model you're targeting before the fact. If the adversary sees there's visible artifacts they'll just change a few parameters or use a different model and be good to go none the less. In short, while this might be academically interesting, I can't help but feel that this is a futile field to work in. A good way to see why is to look at it this way: a skilled human can manually manipulate an image so that it isn't possible to tell that the image has been manipulated. It is then a matter of teaching the model to do the same, which might take time or be computationally prohibitive for now, but eventually it will be done.

danuker 5y ago
> I can't help but feel that this is a futile field to work in. Considering that training a model costs thousands of dollars, and that people reuse models instead of training from scratch, I somewhat disagree with this in general, for now. But this specific technique implies the attacker sees the result, which means they can try different models, until one does not produce artifacts.
- hwers 5y ago
  Models come in a broad spectrum and while it's true that gpt-3 costs a ton, many do not. Any mobile ready snapchat esque real time deep fake filter will typically be super fast and cheap to train (since the model has to be lightweight to run on a phone). There's also transfer learning from an expensive model which could easily be modified to remove whatever 'architecture flagging' is built into the structure.
sigmoid10 5y ago
A few weeks ago, there was a post here about that guy who deepfaked female profile pictures. He showed that the visible artifacts go away by merely scaling up the number of parameters in the model. Give it a few years and these fake images will be impossible to detect unless we agree on some way of cryptographically signing images from real cameras.
- sideshowb 5y ago
  I'm not hopeful that the latter will do much. Just do your deepfake, point a camera at your screen and press the shutter button to get it cryptographically signed..? Ok, so that's tongue in cheek and we'd see some artifacts there but the general principle works. You could intercept the signal from the CCD, or just extract the signing key from the camera's ROM, etc etc.
  - sigmoid10 5y ago
    You can already do that today to fool basic image forensics. Security is never absolute, but if you make the hassle big enough it might suffice to protect the general population. Just like almost all smartphones don't let you mess with the wifi card or spoof your mac address unless you root them. If that wasn't the case, you'd see way more hostapd-wpe attacks these days, since every kid could do it.
    - rspeele 5y ago
      In the case of image trustworthiness, it might actually be better for it to be accessible to the general population, if it's doable at all. Then the public knows that images are unreliable evidence.
- foobar33333 5y ago
  Cryptographic keys embedded in hardware worked really well for the DRM industry. Eventually someone will work out how to extract the key, then create malware to spread around android devices to send back millions of valid keys. What do you do then? Blacklist millions of real peoples cameras and prevent them from using the internet?
  - sigmoid10 5y ago
    You only need a method of key invalidation and renewal, that's all. Ssl certificates have been facing this problem for years. The threat model is equal to someone infecting millions of devices and then sending back banking data, so it's not like people aren't working on mitigating that stuff.
deleted 5y ago
[deleted]
Closi 5y ago
This is fine, we will just train another model to detect which model is being targeted. Then if there is an adversarial model built for that, we will just create a model to detect images that have been changed to hide the model they were built with. Then we will build a model for the adversarial AI on top of that and so on.

danuker 5y ago

> I can't help but feel that this is a futile field to work in. Considering that training a model costs thousands of dollars, and that people reuse models instead of training from scratch, I somewhat disagree with this in general, for now. But this specific technique implies the attacker sees the result, which means they can try different models, until one does not produce artifacts.

hwers 5y ago
Models come in a broad spectrum and while it's true that gpt-3 costs a ton, many do not. Any mobile ready snapchat esque real time deep fake filter will typically be super fast and cheap to train (since the model has to be lightweight to run on a phone). There's also transfer learning from an expensive model which could easily be modified to remove whatever 'architecture flagging' is built into the structure.

sigmoid10 5y ago

A few weeks ago, there was a post here about that guy who deepfaked female profile pictures. He showed that the visible artifacts go away by merely scaling up the number of parameters in the model. Give it a few years and these fake images will be impossible to detect unless we agree on some way of cryptographically signing images from real cameras.

sideshowb 5y ago
I'm not hopeful that the latter will do much. Just do your deepfake, point a camera at your screen and press the shutter button to get it cryptographically signed..? Ok, so that's tongue in cheek and we'd see some artifacts there but the general principle works. You could intercept the signal from the CCD, or just extract the signing key from the camera's ROM, etc etc.
- sigmoid10 5y ago
  You can already do that today to fool basic image forensics. Security is never absolute, but if you make the hassle big enough it might suffice to protect the general population. Just like almost all smartphones don't let you mess with the wifi card or spoof your mac address unless you root them. If that wasn't the case, you'd see way more hostapd-wpe attacks these days, since every kid could do it.
  - rspeele 5y ago
    In the case of image trustworthiness, it might actually be better for it to be accessible to the general population, if it's doable at all. Then the public knows that images are unreliable evidence.
foobar33333 5y ago
Cryptographic keys embedded in hardware worked really well for the DRM industry. Eventually someone will work out how to extract the key, then create malware to spread around android devices to send back millions of valid keys. What do you do then? Blacklist millions of real peoples cameras and prevent them from using the internet?
- sigmoid10 5y ago
  You only need a method of key invalidation and renewal, that's all. Ssl certificates have been facing this problem for years. The threat model is equal to someone infecting millions of devices and then sending back banking data, so it's not like people aren't working on mitigating that stuff.

deleted 5y ago

[deleted]

Closi 5y ago

This is fine, we will just train another model to detect which model is being targeted. Then if there is an adversarial model built for that, we will just create a model to detect images that have been changed to hide the model they were built with. Then we will build a model for the adversarial AI on top of that and so on.