5 ms·

What a deepfake video of Mark Zuckerberg reveals about how we're manipulated

42 points by colinprince 7y ago

techrich 7y ago
Its a crap deep fake, and sounds nothing like him.
- jedberg 7y ago
  And yet, almost everyone will fall for it, because they have no idea what he sounds like.
- cujic9 7y ago
  You're right. His real voice is slightly more robotic.
segmondy 7y ago
Most people think deepfake will be something massive like this. I suspect it's going to be very subtle. With a few words and phrases changed. Imagine a politician saying, "I'm for low taxes" and it's changed to "I'm for high taxes" Just one word that changes everything and has a big enough impact.
- mlb_hn 7y ago
  Or imagine it at scale. It seems there are two uses of deepfake which have separate effects: 1) High profile low volume content that trends (article) 2) Low profile high volume content spam (your point) The second one seems like a much harder problem than the first (but both matter). Everyone already gets dozens of spam/phishing emails and links a day and not much has been done about it. I have no idea what the impact of deepfakes on the latter even looks like.
- shostack 7y ago
  It isn't just the ability to change a message subtly and play both sides, it is the ability to deploy that at scale to highly siloed and divided audiences. This happens to be a known tactic of Russian active measures btw. We've seen the impact of that politically, but I worry what will come during an actual large-scale military conflict when the battlefield very much becomes digital and attempting to subvert civilians to the cause.
- undersuit 7y ago
  As computing power grows these attacks will become something that can be performed in real-time on a single target. You could hack a CEO to introduce literal fake news that they would then act on, or change the instructions on a bomb-building video so that a "required ingredient" is exclusively sourced from a local store with direct contact with the FBI. Did your friend on Skype really tell you to meet them on Main and Broadway at 4PM?
- 6gvONxR4sf7o 7y ago
  Or even more subtle. A tired looking politician says "I'm for, um, low taxes" and it's changed to a slightly more handsome version saying slightly more charismatically, "I'm for low taxes."
- johannes1234321 7y ago
  A single word won't work, if you have any context. A thing which works to some degree is some more subtle editing (see the recent case of the slow Pelossi video) making people appear drunk or sick can be powerful.
jedberg 7y ago
The 2020 election is going to be interesting with deepfake tech getting better and better. Either everyone will be fooled, or everyone will quickly doubt any video ever presented. Either way, the candidates will need to find a new way to spread their propaganda (I mean campaign messaging) that people trust, if they stop trusting video.
- jsonne 7y ago
  When everyone can scream fake news we'll have to take politicians at their word. While President Trump was the first he won't be the last to use this tactic, and it may be a step towards debate turning to policy again rather than personal background gotchas etc.
  - jedberg 7y ago
    > and it may be a step towards debate turning to policy again rather than personal background gotchas etc. That would be a nice change! I think that's also the most optimist view. A pessimist would say that the end is no one trusts anything and we get more fractured and partisan.
    - jsonne 7y ago
      Realistically it'll probably be a bit of column A and a bit of column B haha
jsonne 7y ago
2 big impacts I think deepfakes will have. 1. Internet mobs will be de-clawed and any random person on the street that may have had their private information used against them now has instant plausible deniability. This is a big step towards valuing due process and personal privacy as digging into people's personal histories will become far less valuable than before if it can simple be handwaved away. 2. This could be a step back from the balkanization of the media. If anything can be faked than personal reputation and organizational reputation will be become far more valuable when it comes to the spread of stories.
- aassddffasdf 7y ago
  Regarding #1: not so with widely deployed cryptographic provenance. Also, AI can detect deep fakes and image manipulation.
- Reedx 7y ago
  The problem is that mobs don't care about what's real. They're incentivized to believe whatever reinforces their existing narrative.
- aiyodev 7y ago
  Unfortunately, history proves this wrong. The average person on the street won’t have plausible deniability because deepfakes won’t be good enough to avoid detection. But rich and powerful people and governments will be able to create undetectable deepfakes to fool the public while denying such a thing is possible.
- 6gvONxR4sf7o 7y ago
  Regarding point 2, I thought that at first, but am now less certain. If fake stuff is everywhere, it's true. But if people then gravitate to reputable sources, there's less reason to make fake stuff, decreasing the amount of fake stuff until the incentive towards reputation is gone again. There's going to be an equilibrium in there somewhere, but that may be weirder and less predictable than either end of the spectrum.
lifeisstillgood 7y ago
So this is the future - we can't trust anything and so we trust nothing but our pre-existing biases Bugger that. This is a tech-created problem it can be tech-solved. 1. Photo manipulation leaves traces on the pixels, and this must be worse. 2. cameras can hash / sign every other frame or similar - we can build a chain of hardware to image that can be followed - and of an image or video does not have a CA cert chain it should be as mis-trusted as a web site without TLS 3. Err - there must be other ways.
- was_boring 7y ago
  I have a friend who was working on #1 specifically. They got out of it after they realized it was just playing whack-a-mole. This is honestly one of the very few use cases of cryptography signing a video or picture that I can see. But that would mean amateur journalists or the common person might be excluded from being a trusted source with video evidence.
  - lifeisstillgood 7y ago
    Why? If my iphone camera signs every 24th frame (seems a conservative estimate of speed to do hash and sign), using a combination of the apple camera CA public key and my private key in the secure enclave than we can be reasonably sure that the raw footage now on my site came from a genuine iphone and from the phone I registered (somewhere) All we are doing is trusting that the video images came from a given piece of hardware that we can trust belongs to me. Whether I can be trusted is the same age old problem we have always had But this simply makes it exponentially harder to fake or alter digitally any image (or at least look stupid when someone says "Citation Needed") - if you cannot produce the original raw footage questions will be asked why. I can imagine this being something wanted on police bodycams quite soon
- keithwhor 7y ago
  Riffing on your thoughts: Killer blockchain app: Couldn't we just use blockchain to protect against video deepfakes? Hash every frame as the video is recording, then create an immutable ledger of frames. All "official" videos must be able to be verified with said ledger, and the ledger itself can only be verified with vendor-specific key licensed by a central authority. Perhaps two -- say, Apple has one for iPhone and they must verify as well. This could certainly be the billion dollar opportunity blockchain fanatics have been looking for.
danShumway 7y ago
I'm going to make a moderate to strong-ish claim that deepfakes will not be a substantially serious issue for Democracy or reporting. I'm not certain, but I feel reasonably confident. - We've already had sophisticated image editing capabilities for a long time, and it hasn't substantially changed how newspapers, blogs, and social media accounts share photos. In particular, I see screencaps of Twitter threads all the time, even though they're trivial to fake. For whatever reason, people already seem to be OK with this. Maybe image editing did upset everything and I'm just too young to remember, but in either case, it doesn't look to me like the world fell apart. - We've had (relatively) sophisticated audio editing capabilities for a somewhat lesser period of time, and that hasn't changed the landscape of how I see audio reported on. Reporters still use audio as evidence in articles, we haven't seen a wide spread of proprietary chains of trust, anything like that. - We've seen decent evidence that deepfakes aren't required to currently fake video content to the point that you can deceive people. Deceptive cutting, slowdown, etc... seems to be enough. For people who want reasons to distrust a video, those methods also seem to be adequate. I regularly encounter people who'll look at body-cam footage and say, "well, you don't know, it doesn't show the full context." I'm skeptical deepfakes will make this situation worse -- at most I suspect they'll just further divide us into groups who research sources and groups who don't. - Deepfakes are getting better, and I keep on getting told that they're eventually going to be perfect, but again, in practice I can't help but compare it to Photoshop. It turned out that with Photoshop, people's ability to spot fakes improved at roughly the same rate as the technology, to the point where I don't think it would be trivial for me today to make a fake photo of a politician that wouldn't get quickly identified as such on Reddit or Twitter. I've seen some really impressive AI generated content that blows me away, but right now I can still tell that it's all fake. I haven't seen anything that can consistently produce results that fool me beyond the most straightforward, boring use-cases. And I'm not an expert on this stuff, if it's good it should be able to fool me. In theory, deepfakes are concerning, and I get why people are concerned about them. The technology will get better, it will be easier and easier for anyone to make a video saying anything they want. In theory, I agree with all of that. I just don't see a lot of practical evidence that it's going to be any better or worse than what we have right now. If I hadn't grown up in an era where people were making what to me sounded like similar claims about Photoshop, I would be more worried.
wuliwong 7y ago
I kind of suspect that fake videos will not be much more of an issue than photoshopped images. I don't recall with certainty but I believe there was some fear surrounding the ease at which photographs could be manipulated in the digital age. Some combination of a general growing awareness of the ease at which photos could be altered along with continued advancements in the ability to determine the authenticity of photographs has rendered faking of photographs relatively harmless. In fact, I think the feedback loop has resulted in not being a huge market for faking images at least for political gain. I am not referring to photoshopping images of models and things to just make them look more attractive. Also, although it is very easy to doctor an image, in general people have still managed to keep a level of intuition which allows them to not suspect every image they see as being fake.
vernie 7y ago
Why didn't they use voice cloning?
alwaysanagenda 7y ago
I've been following the press about deep fakes for a while now, and it's been full-on negative, gloom and doom, a potential threat to every democracy on the planet, so on and so forth, in that fashion. I suspect this is a smoke screen for something coming. There will be an event where a very real video emerges, likely of a political elite, and its contents will be so horrific and damming that the only defense will be to claim that it is a deep fake. A large chunk of the public, now well primed for this, will accept the argument, even though many people will go to great lengths to show how the video is actually authentic. But the jury has been poisoned already. The waters are muddied. There will be no conversation, and the mainstream, already pumping out story after story about how awful these deep fakes are, will continue to take up this position, and the sleeping masses will continue to sleep. Keep your wits about you. The truth wants to come out.
lurquer 7y ago
The Rules of Evidence in Federal Court (and the States) make it fairly easy to introduce photo/video/audio. This needs to change. The rules regarding admissibility of this type of evidence were drafted when fakes were difficult. The treatment of emails, however, does not inspire much hope. That is, in most courts, a printout of an email is treated -- as a practical matter -- similarly to a document signed by the author. Very frustrating. There is nothing magical about a piece of paper with an email header... you can create same in two minutes with a word processor. And don't even Get me started with text messages... try to introduce an unsigned typed scrap of paper and you'll get laughed out of court; but, put the text into something that looks like a screenshot of a phone and, voila, instantly admissible as an authentic text message.

42 points by colinprince 7y ago

techrich 7y ago

Its a crap deep fake, and sounds nothing like him.

jedberg 7y ago
And yet, almost everyone will fall for it, because they have no idea what he sounds like.
cujic9 7y ago
You're right. His real voice is slightly more robotic.

segmondy 7y ago

Most people think deepfake will be something massive like this. I suspect it's going to be very subtle. With a few words and phrases changed. Imagine a politician saying, "I'm for low taxes" and it's changed to "I'm for high taxes" Just one word that changes everything and has a big enough impact.

mlb_hn 7y ago
Or imagine it at scale. It seems there are two uses of deepfake which have separate effects: 1) High profile low volume content that trends (article) 2) Low profile high volume content spam (your point) The second one seems like a much harder problem than the first (but both matter). Everyone already gets dozens of spam/phishing emails and links a day and not much has been done about it. I have no idea what the impact of deepfakes on the latter even looks like.
shostack 7y ago
It isn't just the ability to change a message subtly and play both sides, it is the ability to deploy that at scale to highly siloed and divided audiences. This happens to be a known tactic of Russian active measures btw. We've seen the impact of that politically, but I worry what will come during an actual large-scale military conflict when the battlefield very much becomes digital and attempting to subvert civilians to the cause.
undersuit 7y ago
As computing power grows these attacks will become something that can be performed in real-time on a single target. You could hack a CEO to introduce literal fake news that they would then act on, or change the instructions on a bomb-building video so that a "required ingredient" is exclusively sourced from a local store with direct contact with the FBI. Did your friend on Skype really tell you to meet them on Main and Broadway at 4PM?
6gvONxR4sf7o 7y ago
Or even more subtle. A tired looking politician says "I'm for, um, low taxes" and it's changed to a slightly more handsome version saying slightly more charismatically, "I'm for low taxes."
johannes1234321 7y ago
A single word won't work, if you have any context. A thing which works to some degree is some more subtle editing (see the recent case of the slow Pelossi video) making people appear drunk or sick can be powerful.

jedberg 7y ago

The 2020 election is going to be interesting with deepfake tech getting better and better. Either everyone will be fooled, or everyone will quickly doubt any video ever presented. Either way, the candidates will need to find a new way to spread their propaganda (I mean campaign messaging) that people trust, if they stop trusting video.

jsonne 7y ago
When everyone can scream fake news we'll have to take politicians at their word. While President Trump was the first he won't be the last to use this tactic, and it may be a step towards debate turning to policy again rather than personal background gotchas etc.
- jedberg 7y ago
  > and it may be a step towards debate turning to policy again rather than personal background gotchas etc. That would be a nice change! I think that's also the most optimist view. A pessimist would say that the end is no one trusts anything and we get more fractured and partisan.
  - jsonne 7y ago
    Realistically it'll probably be a bit of column A and a bit of column B haha

jsonne 7y ago

2 big impacts I think deepfakes will have. 1. Internet mobs will be de-clawed and any random person on the street that may have had their private information used against them now has instant plausible deniability. This is a big step towards valuing due process and personal privacy as digging into people's personal histories will become far less valuable than before if it can simple be handwaved away. 2. This could be a step back from the balkanization of the media. If anything can be faked than personal reputation and organizational reputation will be become far more valuable when it comes to the spread of stories.

aassddffasdf 7y ago
Regarding #1: not so with widely deployed cryptographic provenance. Also, AI can detect deep fakes and image manipulation.
Reedx 7y ago
The problem is that mobs don't care about what's real. They're incentivized to believe whatever reinforces their existing narrative.
aiyodev 7y ago
Unfortunately, history proves this wrong. The average person on the street won’t have plausible deniability because deepfakes won’t be good enough to avoid detection. But rich and powerful people and governments will be able to create undetectable deepfakes to fool the public while denying such a thing is possible.
6gvONxR4sf7o 7y ago
Regarding point 2, I thought that at first, but am now less certain. If fake stuff is everywhere, it's true. But if people then gravitate to reputable sources, there's less reason to make fake stuff, decreasing the amount of fake stuff until the incentive towards reputation is gone again. There's going to be an equilibrium in there somewhere, but that may be weirder and less predictable than either end of the spectrum.

lifeisstillgood 7y ago

So this is the future - we can't trust anything and so we trust nothing but our pre-existing biases Bugger that. This is a tech-created problem it can be tech-solved. 1. Photo manipulation leaves traces on the pixels, and this must be worse. 2. cameras can hash / sign every other frame or similar - we can build a chain of hardware to image that can be followed - and of an image or video does not have a CA cert chain it should be as mis-trusted as a web site without TLS 3. Err - there must be other ways.

was_boring 7y ago
I have a friend who was working on #1 specifically. They got out of it after they realized it was just playing whack-a-mole. This is honestly one of the very few use cases of cryptography signing a video or picture that I can see. But that would mean amateur journalists or the common person might be excluded from being a trusted source with video evidence.
- lifeisstillgood 7y ago
  Why? If my iphone camera signs every 24th frame (seems a conservative estimate of speed to do hash and sign), using a combination of the apple camera CA public key and my private key in the secure enclave than we can be reasonably sure that the raw footage now on my site came from a genuine iphone and from the phone I registered (somewhere) All we are doing is trusting that the video images came from a given piece of hardware that we can trust belongs to me. Whether I can be trusted is the same age old problem we have always had But this simply makes it exponentially harder to fake or alter digitally any image (or at least look stupid when someone says "Citation Needed") - if you cannot produce the original raw footage questions will be asked why. I can imagine this being something wanted on police bodycams quite soon
keithwhor 7y ago
Riffing on your thoughts: Killer blockchain app: Couldn't we just use blockchain to protect against video deepfakes? Hash every frame as the video is recording, then create an immutable ledger of frames. All "official" videos must be able to be verified with said ledger, and the ledger itself can only be verified with vendor-specific key licensed by a central authority. Perhaps two -- say, Apple has one for iPhone and they must verify as well. This could certainly be the billion dollar opportunity blockchain fanatics have been looking for.

danShumway 7y ago

I'm going to make a moderate to strong-ish claim that deepfakes will not be a substantially serious issue for Democracy or reporting. I'm not certain, but I feel reasonably confident. - We've already had sophisticated image editing capabilities for a long time, and it hasn't substantially changed how newspapers, blogs, and social media accounts share photos. In particular, I see screencaps of Twitter threads all the time, even though they're trivial to fake. For whatever reason, people already seem to be OK with this. Maybe image editing did upset everything and I'm just too young to remember, but in either case, it doesn't look to me like the world fell apart. - We've had (relatively) sophisticated audio editing capabilities for a somewhat lesser period of time, and that hasn't changed the landscape of how I see audio reported on. Reporters still use audio as evidence in articles, we haven't seen a wide spread of proprietary chains of trust, anything like that. - We've seen decent evidence that deepfakes aren't required to currently fake video content to the point that you can deceive people. Deceptive cutting, slowdown, etc... seems to be enough. For people who want reasons to distrust a video, those methods also seem to be adequate. I regularly encounter people who'll look at body-cam footage and say, "well, you don't know, it doesn't show the full context." I'm skeptical deepfakes will make this situation worse -- at most I suspect they'll just further divide us into groups who research sources and groups who don't. - Deepfakes are getting better, and I keep on getting told that they're eventually going to be perfect, but again, in practice I can't help but compare it to Photoshop. It turned out that with Photoshop, people's ability to spot fakes improved at roughly the same rate as the technology, to the point where I don't think it would be trivial for me today to make a fake photo of a politician that wouldn't get quickly identified as such on Reddit or Twitter. I've seen some really impressive AI generated content that blows me away, but right now I can still tell that it's all fake. I haven't seen anything that can consistently produce results that fool me beyond the most straightforward, boring use-cases. And I'm not an expert on this stuff, if it's good it should be able to fool me. In theory, deepfakes are concerning, and I get why people are concerned about them. The technology will get better, it will be easier and easier for anyone to make a video saying anything they want. In theory, I agree with all of that. I just don't see a lot of practical evidence that it's going to be any better or worse than what we have right now. If I hadn't grown up in an era where people were making what to me sounded like similar claims about Photoshop, I would be more worried.

wuliwong 7y ago

I kind of suspect that fake videos will not be much more of an issue than photoshopped images. I don't recall with certainty but I believe there was some fear surrounding the ease at which photographs could be manipulated in the digital age. Some combination of a general growing awareness of the ease at which photos could be altered along with continued advancements in the ability to determine the authenticity of photographs has rendered faking of photographs relatively harmless. In fact, I think the feedback loop has resulted in not being a huge market for faking images at least for political gain. I am not referring to photoshopping images of models and things to just make them look more attractive. Also, although it is very easy to doctor an image, in general people have still managed to keep a level of intuition which allows them to not suspect every image they see as being fake.

vernie 7y ago

Why didn't they use voice cloning?

alwaysanagenda 7y ago

I've been following the press about deep fakes for a while now, and it's been full-on negative, gloom and doom, a potential threat to every democracy on the planet, so on and so forth, in that fashion. I suspect this is a smoke screen for something coming. There will be an event where a very real video emerges, likely of a political elite, and its contents will be so horrific and damming that the only defense will be to claim that it is a deep fake. A large chunk of the public, now well primed for this, will accept the argument, even though many people will go to great lengths to show how the video is actually authentic. But the jury has been poisoned already. The waters are muddied. There will be no conversation, and the mainstream, already pumping out story after story about how awful these deep fakes are, will continue to take up this position, and the sleeping masses will continue to sleep. Keep your wits about you. The truth wants to come out.

lurquer 7y ago

The Rules of Evidence in Federal Court (and the States) make it fairly easy to introduce photo/video/audio. This needs to change. The rules regarding admissibility of this type of evidence were drafted when fakes were difficult. The treatment of emails, however, does not inspire much hope. That is, in most courts, a printout of an email is treated -- as a practical matter -- similarly to a document signed by the author. Very frustrating. There is nothing magical about a piece of paper with an email header... you can create same in two minutes with a word processor. And don't even Get me started with text messages... try to introduce an unsigned typed scrap of paper and you'll get laughed out of court; but, put the text into something that looks like a screenshot of a phone and, voila, instantly admissible as an authentic text message.