4 ms·

"Despite this promising figure, the team notes there are still several limitations to the approach. Among them is the fact that these deviations could be fixed

by kuu 6y ago

"Despite this promising figure, the team notes there are still several limitations to the approach. Among them is the fact that these deviations could be fixed with editing software and that the image must present a clear view of the eye for the technique to work" So interesting idea but really easy for the "deepfakers creators" to fix it

codetrotter 6y ago
> these deviations could be fixed with editing software Or even yet, couldn’t this also be part of the training of the deepfake models? So that when it’s learning it’ll try to achieve a high realism score as judged by the algorithm that they use in this other tool?
- mckirk 6y ago
  That is always the problem with this fight, yes. In a sense it's a meta-GAN, with other teams building better discriminators that can then be used to further train the generator. This works 'out of the box' if the discriminator is differentiable, but even if that's not the case you could use Reinforcement Learning to achieve the same, I suppose. If you have the man power, you could even train humans to spot deepfakes and then use them as discriminators to improve the fakes in turn... yay! Be vary once reCAPTCHA asks you to 'spot the fake'.
  - Someone 6y ago
    Indeed. It’s similar to paper money counterfeiting. There, they use two strategies to prevent that: 1) Do not tell the counterfeiters how you discriminate their products from the real ones. 2) Keep the technology you use out of the hands of the counterfeiters. For this problem, for 2), I think the cat is out of the bag. A with money, keeping tech out of the hands of government-size entities is impossible, but keeping it out of the hands of small players is doable. Here, doing that that would require locking down all computers in the hands of small players to the level iOS does it. Not impossible, but won’t happen overnight, and I don’t think we want to go there. That leaves 1). That’s a bit at odds with “publish or perish”, but I guess the likes of NSA personnel aren’t in that rat race, so it might be possible for the NSA to provide an online “fake or real?” tool. Of course, that would mean the likes of the NSA could lie when asked whether their fake was real and when asked whether a real image they don’t want to be known as real was real, so we would need multiple, independent such parties.
    - macca321 6y ago
      In the long run, I wonder if all video will be signed via blockchain-or-other-trusted-3rd-party. You'd need everything from the camera to the the editing software to support it though.
      - CPLX 6y ago
        Until reality has a signing mechanism it’s not really going to fix anything. Otherwise you can just feed whatever video you have in mind into the first stage of that chain you’ve created and say you were there when it happened.
        mckirk 6y ago
        Unless somebody builds a kind of 'pseudo-random-generator' chain where the internal state is huge, changes substantially each tick and is very expensive to compute, so that it becomes infeasible to store or recompute (a large number of) previous states for anyone but the most powerful players. You could then use that internal state as a private key to sign what you want and only keep the corresponding public key and a signed date+time to proof the veracity. The public keys themselves would need to form a blockchain in the traditional sense, so you can verify their integrity without the previous internal states. If you wanted to go all-in with the verification, you'd probably also want to show something containing the date+time in the video, together with some complicated physical effects that are hard to fake. Well, just thinking out loud here, but it seems like it might work :D
      - eivarv 6y ago
        That's still just a shortcut to a faulty thought-process. We can't inherently trust content on the basis of its medium.
      - cesarb 6y ago
        I vaguely recall reading somewhere several years ago about some cameras which signed the photos they produced, so one could prove they came directly from the camera and weren't edited. IIRC, it was in an article about these signatures being broken, though I don't recall the details (perhaps it was by dumping the signing key from the camera firmware?).
- kuu 6y ago
  Yes, I was just quoting the article :)
jcims 6y ago
Yes going to definitely take an ensemble of detection signals to really build any confidence one way or the other. Then ultimately it’s going to be left as a confidence value which humans are going to be free to override based on their own assessment of the content.

by kuu 6y ago

codetrotter 6y ago

> these deviations could be fixed with editing software Or even yet, couldn’t this also be part of the training of the deepfake models? So that when it’s learning it’ll try to achieve a high realism score as judged by the algorithm that they use in this other tool?

mckirk 6y ago
That is always the problem with this fight, yes. In a sense it's a meta-GAN, with other teams building better discriminators that can then be used to further train the generator. This works 'out of the box' if the discriminator is differentiable, but even if that's not the case you could use Reinforcement Learning to achieve the same, I suppose. If you have the man power, you could even train humans to spot deepfakes and then use them as discriminators to improve the fakes in turn... yay! Be vary once reCAPTCHA asks you to 'spot the fake'.
- Someone 6y ago
  Indeed. It’s similar to paper money counterfeiting. There, they use two strategies to prevent that: 1) Do not tell the counterfeiters how you discriminate their products from the real ones. 2) Keep the technology you use out of the hands of the counterfeiters. For this problem, for 2), I think the cat is out of the bag. A with money, keeping tech out of the hands of government-size entities is impossible, but keeping it out of the hands of small players is doable. Here, doing that that would require locking down all computers in the hands of small players to the level iOS does it. Not impossible, but won’t happen overnight, and I don’t think we want to go there. That leaves 1). That’s a bit at odds with “publish or perish”, but I guess the likes of NSA personnel aren’t in that rat race, so it might be possible for the NSA to provide an online “fake or real?” tool. Of course, that would mean the likes of the NSA could lie when asked whether their fake was real and when asked whether a real image they don’t want to be known as real was real, so we would need multiple, independent such parties.
  - macca321 6y ago
    In the long run, I wonder if all video will be signed via blockchain-or-other-trusted-3rd-party. You'd need everything from the camera to the the editing software to support it though.
    - CPLX 6y ago
      Until reality has a signing mechanism it’s not really going to fix anything. Otherwise you can just feed whatever video you have in mind into the first stage of that chain you’ve created and say you were there when it happened.
      - mckirk 6y ago
        Unless somebody builds a kind of 'pseudo-random-generator' chain where the internal state is huge, changes substantially each tick and is very expensive to compute, so that it becomes infeasible to store or recompute (a large number of) previous states for anyone but the most powerful players. You could then use that internal state as a private key to sign what you want and only keep the corresponding public key and a signed date+time to proof the veracity. The public keys themselves would need to form a blockchain in the traditional sense, so you can verify their integrity without the previous internal states. If you wanted to go all-in with the verification, you'd probably also want to show something containing the date+time in the video, together with some complicated physical effects that are hard to fake. Well, just thinking out loud here, but it seems like it might work :D
    - eivarv 6y ago
      That's still just a shortcut to a faulty thought-process. We can't inherently trust content on the basis of its medium.
    - cesarb 6y ago
      I vaguely recall reading somewhere several years ago about some cameras which signed the photos they produced, so one could prove they came directly from the camera and weren't edited. IIRC, it was in an article about these signatures being broken, though I don't recall the details (perhaps it was by dumping the signing key from the camera firmware?).
kuu 6y ago
Yes, I was just quoting the article :)

jcims 6y ago

Yes going to definitely take an ensemble of detection signals to really build any confidence one way or the other. Then ultimately it’s going to be left as a confidence value which humans are going to be free to override based on their own assessment of the content.