4 ms·

We need software to detect deep fakes

by leoh 7y ago

EGreg 7y ago
That’s just an adversarial network. The deepfakes will become so good that we won’t be able to tell anyway. It’s like AlphaZero playing against itself and you hoping that the “defensive” AlphaZero is able to win the game every time. In reality, each side will win sometimes but will have absolutely no way of explaining why or how. So we will have no idea anymore what is fake or not, just whether the detector said it was for some reason that time. Basically, it’s game over very soon after you do this arms race. See CAPTCHAs being solved etc.
- Too 7y ago
  So we are back to the polygraph, we don't know it's accuracy so there is no way to tell if a human is lying or not. Taking it to the extreme, video evidence falls back to being no more valuable than eye witnesses.
neuralzen 7y ago
The problem here is it is a cat and mouse game. Software algorithms that can detect deep fakes can also be used to patch detectable flaws in deep fakes to evade detection, at least until new detection approaches get better but the cycle repeats on future generated content.
kfuwbi2640 7y ago
I’ve seen a number of attempts to identify deepfakes and other forms of manipulated images using AI. This seems like a fool’s errand since it becomes a never ending adversarial AI arms race. Instead, I haven’t seen a proposal for a system I think could work well. Camera and phone manufacturers could have their devices cryptographically sign each photo or video taken. And that’s it. From that starting place, you can build a system on top of it to verify that the image on the site you’re reading is authentic. What am I missing that makes this an invalid approach? I do understand that this would require manufacturers to implement, but it seems achievable to get them onboard. I even think you get one company like Apple to do this and it’s enough traction for the rest of the industry to have to follow suit.
- imron 7y ago
  > This seems like a fool’s errand since it becomes a never ending adversarial AI arms race. Mission accomplished? https://xkcd.com/810/ https://xkcd.com/810/
- zrm 7y ago
  Such a system would require all devices to be secure against key extraction. Otherwise the attacker need only choose the most vulnerable device, extract a signing key from it and sign their deepfakes with it. It would also allow any device manufacturer to sign anything they like, as well as anyone who can coerce a device manufacturer to do so.
  - kfuwbi2640 7y ago
    Apologies for a late response here (by HN standards where conversations last only a number of hours). I agree that an attacker could compromise weaker devices and sign their deep fakes with them. But then hopefully those keys Or that manufacturer would be blacklisted. In my mind, a company like Hauwei could implement this, but I as a consumer of media wouldn’t necessarily trust photos from their devices. But photos signed by an iPhone where Apple has a better privacy record, I could trust more. Thanks for replying though, this does help me understand the challenges in a system like this.
    - zrm 7y ago
      It's not really a matter of privacy record. In general manufacturers don't do it on purpose. For example, it was discovered that it's possible to extract keys from Intel SGX enclaves using certain speculative execution vulnerabilities. Intel SGX predates Spectre. It isn't a category of vulnerability they knew existed when they were designing it. Vulnerabilities are regularly discovered in almost everything, iPhones included. Diligent vendors are quick to patch them, but an attacker only needs to wait until the next vulnerability is discovered and then extract the signing keys from a device that hasn't been patched yet. You also have no way of knowing which keys they are -- if a million devices of a particular model have a known vulnerability then any attacker could extract the keys from any of them, and even blacklisting all of them (which would tend to dissatisfy their innocent owners) still wouldn't save you from an attacker using an unpublished vulnerability against a device you don't even know is vulnerable. To put this another way, this is basically the same class of technology that Hollywood uses for DRM. Now, how many Hollywood movies can you say have not been pirated by anyone?
YetAnotherNick 7y ago
https://www.kaggle.com/c/deepfake-detection-challenge https://www.kaggle.com/c/deepfake-detection-challenge > AWS, Facebook, Microsoft, the Partnership on AI’s Media Integrity Steering Committee, and academics have come together to build the Deepfake Detection Challenge (DFDC). The goal of the challenge is to spur researchers around the world to build innovative new technologies that can help detect deepfakes and manipulated media. This has a prize pool of $1 million.

We need software to detect deep fakes

by leoh 7y ago

We need software to detect deep fakes

That’s just an adversarial network. The deepfakes will become so good that we won’t be able to tell anyway. It’s like AlphaZero playing against itself and you hoping that the “defensive” AlphaZero is able to win the game every time. In reality, each side will win sometimes but will have absolutely no way of explaining why or how. So we will have no idea anymore what is fake or not, just whether the detector said it was for some reason that time. Basically, it’s game over very soon after you do this arms race. See CAPTCHAs being solved etc.

Too 7y ago
So we are back to the polygraph, we don't know it's accuracy so there is no way to tell if a human is lying or not. Taking it to the extreme, video evidence falls back to being no more valuable than eye witnesses.

neuralzen 7y ago

The problem here is it is a cat and mouse game. Software algorithms that can detect deep fakes can also be used to patch detectable flaws in deep fakes to evade detection, at least until new detection approaches get better but the cycle repeats on future generated content.

kfuwbi2640 7y ago

I’ve seen a number of attempts to identify deepfakes and other forms of manipulated images using AI. This seems like a fool’s errand since it becomes a never ending adversarial AI arms race. Instead, I haven’t seen a proposal for a system I think could work well. Camera and phone manufacturers could have their devices cryptographically sign each photo or video taken. And that’s it. From that starting place, you can build a system on top of it to verify that the image on the site you’re reading is authentic. What am I missing that makes this an invalid approach? I do understand that this would require manufacturers to implement, but it seems achievable to get them onboard. I even think you get one company like Apple to do this and it’s enough traction for the rest of the industry to have to follow suit.

imron 7y ago
> This seems like a fool’s errand since it becomes a never ending adversarial AI arms race. Mission accomplished? https://xkcd.com/810/ https://xkcd.com/810/
zrm 7y ago
Such a system would require all devices to be secure against key extraction. Otherwise the attacker need only choose the most vulnerable device, extract a signing key from it and sign their deepfakes with it. It would also allow any device manufacturer to sign anything they like, as well as anyone who can coerce a device manufacturer to do so.
- kfuwbi2640 7y ago
  Apologies for a late response here (by HN standards where conversations last only a number of hours). I agree that an attacker could compromise weaker devices and sign their deep fakes with them. But then hopefully those keys Or that manufacturer would be blacklisted. In my mind, a company like Hauwei could implement this, but I as a consumer of media wouldn’t necessarily trust photos from their devices. But photos signed by an iPhone where Apple has a better privacy record, I could trust more. Thanks for replying though, this does help me understand the challenges in a system like this.
  - zrm 7y ago
    It's not really a matter of privacy record. In general manufacturers don't do it on purpose. For example, it was discovered that it's possible to extract keys from Intel SGX enclaves using certain speculative execution vulnerabilities. Intel SGX predates Spectre. It isn't a category of vulnerability they knew existed when they were designing it. Vulnerabilities are regularly discovered in almost everything, iPhones included. Diligent vendors are quick to patch them, but an attacker only needs to wait until the next vulnerability is discovered and then extract the signing keys from a device that hasn't been patched yet. You also have no way of knowing which keys they are -- if a million devices of a particular model have a known vulnerability then any attacker could extract the keys from any of them, and even blacklisting all of them (which would tend to dissatisfy their innocent owners) still wouldn't save you from an attacker using an unpublished vulnerability against a device you don't even know is vulnerable. To put this another way, this is basically the same class of technology that Hollywood uses for DRM. Now, how many Hollywood movies can you say have not been pirated by anyone?

YetAnotherNick 7y ago

https://www.kaggle.com/c/deepfake-detection-challenge https://www.kaggle.com/c/deepfake-detection-challenge > AWS, Facebook, Microsoft, the Partnership on AI’s Media Integrity Steering Committee, and academics have come together to build the Deepfake Detection Challenge (DFDC). The goal of the challenge is to spur researchers around the world to build innovative new technologies that can help detect deepfakes and manipulated media. This has a prize pool of $1 million.