10 ms·

Computer tool spots deepfakes via tiny reflections in the eyes

kuu 6y ago
"Despite this promising figure, the team notes there are still several limitations to the approach. Among them is the fact that these deviations could be fixed with editing software and that the image must present a clear view of the eye for the technique to work" So interesting idea but really easy for the "deepfakers creators" to fix it
- codetrotter 6y ago
  > these deviations could be fixed with editing software Or even yet, couldn’t this also be part of the training of the deepfake models? So that when it’s learning it’ll try to achieve a high realism score as judged by the algorithm that they use in this other tool?
  - mckirk 6y ago
    That is always the problem with this fight, yes. In a sense it's a meta-GAN, with other teams building better discriminators that can then be used to further train the generator. This works 'out of the box' if the discriminator is differentiable, but even if that's not the case you could use Reinforcement Learning to achieve the same, I suppose. If you have the man power, you could even train humans to spot deepfakes and then use them as discriminators to improve the fakes in turn... yay! Be vary once reCAPTCHA asks you to 'spot the fake'.
    - Someone 6y ago
      Indeed. It’s similar to paper money counterfeiting. There, they use two strategies to prevent that: 1) Do not tell the counterfeiters how you discriminate their products from the real ones. 2) Keep the technology you use out of the hands of the counterfeiters. For this problem, for 2), I think the cat is out of the bag. A with money, keeping tech out of the hands of government-size entities is impossible, but keeping it out of the hands of small players is doable. Here, doing that that would require locking down all computers in the hands of small players to the level iOS does it. Not impossible, but won’t happen overnight, and I don’t think we want to go there. That leaves 1). That’s a bit at odds with “publish or perish”, but I guess the likes of NSA personnel aren’t in that rat race, so it might be possible for the NSA to provide an online “fake or real?” tool. Of course, that would mean the likes of the NSA could lie when asked whether their fake was real and when asked whether a real image they don’t want to be known as real was real, so we would need multiple, independent such parties.
      - macca321 6y ago
        In the long run, I wonder if all video will be signed via blockchain-or-other-trusted-3rd-party. You'd need everything from the camera to the the editing software to support it though.
        CPLX 6y ago
        Until reality has a signing mechanism it’s not really going to fix anything. Otherwise you can just feed whatever video you have in mind into the first stage of that chain you’ve created and say you were there when it happened.
        mckirk 6y ago
        Unless somebody builds a kind of 'pseudo-random-generator' chain where the internal state is huge, changes substantially each tick and is very expensive to compute, so that it becomes infeasible to store or recompute (a large number of) previous states for anyone but the most powerful players. You could then use that internal state as a private key to sign what you want and only keep the corresponding public key and a signed date+time to proof the veracity. The public keys themselves would need to form a blockchain in the traditional sense, so you can verify their integrity without the previous internal states. If you wanted to go all-in with the verification, you'd probably also want to show something containing the date+time in the video, together with some complicated physical effects that are hard to fake. Well, just thinking out loud here, but it seems like it might work :D
        eivarv 6y ago
        That's still just a shortcut to a faulty thought-process. We can't inherently trust content on the basis of its medium.
        cesarb 6y ago
        I vaguely recall reading somewhere several years ago about some cameras which signed the photos they produced, so one could prove they came directly from the camera and weren't edited. IIRC, it was in an article about these signatures being broken, though I don't recall the details (perhaps it was by dumping the signing key from the camera firmware?).
  - kuu 6y ago
    Yes, I was just quoting the article :)
- jcims 6y ago
  Yes going to definitely take an ensemble of detection signals to really build any confidence one way or the other. Then ultimately it’s going to be left as a confidence value which humans are going to be free to override based on their own assessment of the content.
jcims 6y ago
I’m curious how Eulerian Video Magnification would fare as a DeepFake signal. https://people.csail.mit.edu/mrub/evm/ https://people.csail.mit.edu/mrub/evm/
- bsenftner 6y ago
  In the enterprise class FR software I work on, EVM is one of the methods used for "liveness detection" in live video - no pulse, or impossible-be-be-alive pulses get flagged.
  - jcims 6y ago
    Nice! It struck me as a very useful analytical method for video in general.
Mizza 6y ago
Can every improvement in deep-fake detection be incorporated into the discriminator step of the deep-fake generator?
- timlardner 6y ago
  Yup. For those unfamiliar: https://en.wikipedia.org/wiki/Generative_adversarial_network https://en.wikipedia.org/wiki/Generative_adversarial_network
- suvelx 6y ago
  For freely available and/or published detectors. Probably yes. But you can hinder it's use quite easily if you have a private/commercial implementation by just not doing realtime detection.
rkachowski 6y ago
Light reflections would have to be one of the easiest artifacts to fix given the depth of work on lighting in computer graphics, no?
- faeyanpiraat 6y ago
  You would have to realistically model the subject’s surroundings, which can be quite challenging if the camera only points at one direction the whole time.
eivarv 6y ago
Trying to "solve" deepfakes with recognition is a dead end, as it's an arms-race between generators and detectors. We stop treating video as if its some perfect representation of reality that's inherently trustworthy, and learn from confronting similar issues with other media earlier in history. Focus on media literacy, critical thinking and digital literacy.
- WanderPanda 6y ago
  > Focus on media literacy, critical thinking and digital literacy. 1. That is not the easy solution for the politicians 2. That is not the easy solution for the individual (needs to question himself) = Not gonna happen I believe “The hard way” does only appeal to a small minority (maybe not in HN :D )
  - brobdingnagians 6y ago
    Indeed, most politicians don't really want you to focus on critical thinking. It would solve some of their problems, and create lots more problems for them. But for yourself, it is becoming more and more important to really think through what is being presented to you in whatever media form and analyze whether it is credible. The print form of a deep fake is plain old well constructed word-lies, which is why we have had the phrase "don't believe everything you read on the internet" for awhile. There are people who do, but we live better lives if we don't. Deep fakes aren't really a new problem, they are just a different form of an old problem.
    - EGreg 6y ago
      Actually once the politicians speeches are being deepfaked they will BEG the public to make sure there is a digital watermark on the video before trusting it. And then of course the keys to the watermarking will be leaked :)
    - WanderPanda 6y ago
      That “don't believe everything you read on the internet” example is nice, I hope society comes up with a similar immune response for other things
williesleg 6y ago
Patched this week, not anymore pal.
mooneater 6y ago
These fake-detector tools will only pave the way for flawless deepfakes. Im not sure what the solutions are to the societal problem it opens up. Media will get a lot stranger.
WildParser 6y ago
I once took one of these online-tests for spotting deep fakes. By marking anything that has reflections in the eyes as fake I got only 1 answer wrong (the real guy there looked really strange).
castis 6y ago
Cool cool, new automated tests for deepfake libraries.
auganov 6y ago
I wonder if this style of profile picture will simply go out of fashion. When someone who's not obviously well known has this close-up nondescript professional shot on regular social media, I check it out to see if it's a fake. 95% of the time there are obvious artifacts. It's already so slanted to a point where people make themselves look fake by using this style rather than the other way around.
- imbnwa 6y ago
  I mean I can't rememeber the last time a Uber/Lyft driver's profile pic didn't seem to also double as one of their Tinder pics. America is going to float away into the ether of mere appearances where everyone's constantly disappointed with... reality.
lamename 6y ago
Maybe a naive question: Would it be possible and useful to have some type of checksum system for videos? (Perhaps we already do). This might not defend against all deep fakes, but at least it could in certain cases with more...official videos. I'm not sure how this could defend e.g. a person from a deep fake someone made of them without their knowledge. But I'd like to hear what people who are more knowledgable than me think.
- deft 6y ago
  NFTs solve the deepfake problem.
zupreme 6y ago
Clearly this had to happen, in due course, but these publicly-announced efforts only serve to teach deepfake makers to produce better (i.e. more believable) content. The interesting place this could take us to is one within which video evidence of a crime is no longer sufficient evidence to convict, making charges like police brutality and domestic abuse even harder to consistently prove than they are now.
douglaswlance 6y ago
Won't there always be a better dataset for deep-fake detectors than there will be for generators?
- aparsons 6y ago
  Other way around, no?
  - douglaswlance 6y ago
    The problem with the other way around is that there is no definitive measure of "convincingness."
m3kw9 6y ago
Deep fakes is a massively scaled up problem of actual people faking it on video.
MR4D 6y ago
This is like bug testing for deep fakes. Eventually all the bugs will be ironed out and we’ll see basically perfect deep fakes. Might take several years, but it’s coming.
morelandjs 6y ago
I’ve always wondered if physics can be useful in this context. Emulators will often approximate a physical system in such a way that it looks indistinguishable, but the emulator may grossly violate a fundamental physical law. An example of this is training neutral networks to emulate the hydrodynamic equations of motion. The emulator temperature profile may look like the real thing, but it could also grossly violate total conservation of energy. I’m curious if we can apply such “physics tests” to deep fakes to search for violations.
- ravi-delia 6y ago
  It seems like on some level GANs should already be immune to that kind of detection right? If there was that severe of a telltail sign, the adversary would pick up on it. It feels like these networks are optimizing for fooling us, but really they're optimizing for fooling each other, and they're good enough at detection that we get fooled too.
michaelmrose 6y ago
Can't we separate the hardware that runs the camera from the user accessible part of the device and have it cryptographically sign the created images/videos removing the need to spot the differences?
spyder 6y ago
AI's response: Thanks, added it to the loss function to optimize for. :D

kuu 6y ago

"Despite this promising figure, the team notes there are still several limitations to the approach. Among them is the fact that these deviations could be fixed with editing software and that the image must present a clear view of the eye for the technique to work" So interesting idea but really easy for the "deepfakers creators" to fix it

codetrotter 6y ago
> these deviations could be fixed with editing software Or even yet, couldn’t this also be part of the training of the deepfake models? So that when it’s learning it’ll try to achieve a high realism score as judged by the algorithm that they use in this other tool?
- mckirk 6y ago
  That is always the problem with this fight, yes. In a sense it's a meta-GAN, with other teams building better discriminators that can then be used to further train the generator. This works 'out of the box' if the discriminator is differentiable, but even if that's not the case you could use Reinforcement Learning to achieve the same, I suppose. If you have the man power, you could even train humans to spot deepfakes and then use them as discriminators to improve the fakes in turn... yay! Be vary once reCAPTCHA asks you to 'spot the fake'.
  - Someone 6y ago
    Indeed. It’s similar to paper money counterfeiting. There, they use two strategies to prevent that: 1) Do not tell the counterfeiters how you discriminate their products from the real ones. 2) Keep the technology you use out of the hands of the counterfeiters. For this problem, for 2), I think the cat is out of the bag. A with money, keeping tech out of the hands of government-size entities is impossible, but keeping it out of the hands of small players is doable. Here, doing that that would require locking down all computers in the hands of small players to the level iOS does it. Not impossible, but won’t happen overnight, and I don’t think we want to go there. That leaves 1). That’s a bit at odds with “publish or perish”, but I guess the likes of NSA personnel aren’t in that rat race, so it might be possible for the NSA to provide an online “fake or real?” tool. Of course, that would mean the likes of the NSA could lie when asked whether their fake was real and when asked whether a real image they don’t want to be known as real was real, so we would need multiple, independent such parties.
    - macca321 6y ago
      In the long run, I wonder if all video will be signed via blockchain-or-other-trusted-3rd-party. You'd need everything from the camera to the the editing software to support it though.
      - CPLX 6y ago
        Until reality has a signing mechanism it’s not really going to fix anything. Otherwise you can just feed whatever video you have in mind into the first stage of that chain you’ve created and say you were there when it happened.
        mckirk 6y ago
        Unless somebody builds a kind of 'pseudo-random-generator' chain where the internal state is huge, changes substantially each tick and is very expensive to compute, so that it becomes infeasible to store or recompute (a large number of) previous states for anyone but the most powerful players. You could then use that internal state as a private key to sign what you want and only keep the corresponding public key and a signed date+time to proof the veracity. The public keys themselves would need to form a blockchain in the traditional sense, so you can verify their integrity without the previous internal states. If you wanted to go all-in with the verification, you'd probably also want to show something containing the date+time in the video, together with some complicated physical effects that are hard to fake. Well, just thinking out loud here, but it seems like it might work :D
      - eivarv 6y ago
        That's still just a shortcut to a faulty thought-process. We can't inherently trust content on the basis of its medium.
      - cesarb 6y ago
        I vaguely recall reading somewhere several years ago about some cameras which signed the photos they produced, so one could prove they came directly from the camera and weren't edited. IIRC, it was in an article about these signatures being broken, though I don't recall the details (perhaps it was by dumping the signing key from the camera firmware?).
- kuu 6y ago
  Yes, I was just quoting the article :)
jcims 6y ago
Yes going to definitely take an ensemble of detection signals to really build any confidence one way or the other. Then ultimately it’s going to be left as a confidence value which humans are going to be free to override based on their own assessment of the content.

jcims 6y ago

I’m curious how Eulerian Video Magnification would fare as a DeepFake signal. https://people.csail.mit.edu/mrub/evm/ https://people.csail.mit.edu/mrub/evm/

bsenftner 6y ago
In the enterprise class FR software I work on, EVM is one of the methods used for "liveness detection" in live video - no pulse, or impossible-be-be-alive pulses get flagged.
- jcims 6y ago
  Nice! It struck me as a very useful analytical method for video in general.

Mizza 6y ago

Can every improvement in deep-fake detection be incorporated into the discriminator step of the deep-fake generator?

timlardner 6y ago
Yup. For those unfamiliar: https://en.wikipedia.org/wiki/Generative_adversarial_network https://en.wikipedia.org/wiki/Generative_adversarial_network
suvelx 6y ago
For freely available and/or published detectors. Probably yes. But you can hinder it's use quite easily if you have a private/commercial implementation by just not doing realtime detection.

rkachowski 6y ago

Light reflections would have to be one of the easiest artifacts to fix given the depth of work on lighting in computer graphics, no?

faeyanpiraat 6y ago
You would have to realistically model the subject’s surroundings, which can be quite challenging if the camera only points at one direction the whole time.

eivarv 6y ago

Trying to "solve" deepfakes with recognition is a dead end, as it's an arms-race between generators and detectors. We stop treating video as if its some perfect representation of reality that's inherently trustworthy, and learn from confronting similar issues with other media earlier in history. Focus on media literacy, critical thinking and digital literacy.

WanderPanda 6y ago
> Focus on media literacy, critical thinking and digital literacy. 1. That is not the easy solution for the politicians 2. That is not the easy solution for the individual (needs to question himself) = Not gonna happen I believe “The hard way” does only appeal to a small minority (maybe not in HN :D )
- brobdingnagians 6y ago
  Indeed, most politicians don't really want you to focus on critical thinking. It would solve some of their problems, and create lots more problems for them. But for yourself, it is becoming more and more important to really think through what is being presented to you in whatever media form and analyze whether it is credible. The print form of a deep fake is plain old well constructed word-lies, which is why we have had the phrase "don't believe everything you read on the internet" for awhile. There are people who do, but we live better lives if we don't. Deep fakes aren't really a new problem, they are just a different form of an old problem.
  - EGreg 6y ago
    Actually once the politicians speeches are being deepfaked they will BEG the public to make sure there is a digital watermark on the video before trusting it. And then of course the keys to the watermarking will be leaked :)
  - WanderPanda 6y ago
    That “don't believe everything you read on the internet” example is nice, I hope society comes up with a similar immune response for other things

williesleg 6y ago

Patched this week, not anymore pal.

mooneater 6y ago

These fake-detector tools will only pave the way for flawless deepfakes. Im not sure what the solutions are to the societal problem it opens up. Media will get a lot stranger.

WildParser 6y ago

I once took one of these online-tests for spotting deep fakes. By marking anything that has reflections in the eyes as fake I got only 1 answer wrong (the real guy there looked really strange).

castis 6y ago

Cool cool, new automated tests for deepfake libraries.

auganov 6y ago

I wonder if this style of profile picture will simply go out of fashion. When someone who's not obviously well known has this close-up nondescript professional shot on regular social media, I check it out to see if it's a fake. 95% of the time there are obvious artifacts. It's already so slanted to a point where people make themselves look fake by using this style rather than the other way around.

imbnwa 6y ago
I mean I can't rememeber the last time a Uber/Lyft driver's profile pic didn't seem to also double as one of their Tinder pics. America is going to float away into the ether of mere appearances where everyone's constantly disappointed with... reality.

lamename 6y ago

Maybe a naive question: Would it be possible and useful to have some type of checksum system for videos? (Perhaps we already do). This might not defend against all deep fakes, but at least it could in certain cases with more...official videos. I'm not sure how this could defend e.g. a person from a deep fake someone made of them without their knowledge. But I'd like to hear what people who are more knowledgable than me think.

deft 6y ago
NFTs solve the deepfake problem.

zupreme 6y ago

Clearly this had to happen, in due course, but these publicly-announced efforts only serve to teach deepfake makers to produce better (i.e. more believable) content. The interesting place this could take us to is one within which video evidence of a crime is no longer sufficient evidence to convict, making charges like police brutality and domestic abuse even harder to consistently prove than they are now.

douglaswlance 6y ago

Won't there always be a better dataset for deep-fake detectors than there will be for generators?

aparsons 6y ago
Other way around, no?
- douglaswlance 6y ago
  The problem with the other way around is that there is no definitive measure of "convincingness."

m3kw9 6y ago

Deep fakes is a massively scaled up problem of actual people faking it on video.

MR4D 6y ago

This is like bug testing for deep fakes. Eventually all the bugs will be ironed out and we’ll see basically perfect deep fakes. Might take several years, but it’s coming.

morelandjs 6y ago

I’ve always wondered if physics can be useful in this context. Emulators will often approximate a physical system in such a way that it looks indistinguishable, but the emulator may grossly violate a fundamental physical law. An example of this is training neutral networks to emulate the hydrodynamic equations of motion. The emulator temperature profile may look like the real thing, but it could also grossly violate total conservation of energy. I’m curious if we can apply such “physics tests” to deep fakes to search for violations.

ravi-delia 6y ago
It seems like on some level GANs should already be immune to that kind of detection right? If there was that severe of a telltail sign, the adversary would pick up on it. It feels like these networks are optimizing for fooling us, but really they're optimizing for fooling each other, and they're good enough at detection that we get fooled too.

michaelmrose 6y ago

Can't we separate the hardware that runs the camera from the user accessible part of the device and have it cryptographically sign the created images/videos removing the need to spot the differences?

spyder 6y ago

AI's response: Thanks, added it to the loss function to optimize for. :D