11 ms·

Malicious Deep Fake Prohibition Act of 2018 [pdf]

121 points by Hard_Space 8y ago

orbifold 8y ago
Just yesterday it occurred to me that Deep Fakes could be used to create some of the most potent disinformation campaigns ever. Already now we have had memes with misattributed quotes and photos edited to associate Clinton with the devil, satanic rituals and so on. Think of how much more impactful slightly off quotes and speeches would be.
- samcday 8y ago
  Yes, it definitely has the potential to become a major issue. So much so that the DoD is already working on tools to detect deepfakes [1]. Of course this will very likely devolve into an arms race akin to antivirus vendors vs malware developers. The fundamental problem is going to be that many people seem to actually like being in their comfortable echo chamber. So, if they're presented with a video that reaffirms their hunch that the Clintons run a child trafficking ring from under a pizza store, those people seem to be unlikely to venture out and do the necessary due diligence to verify the information they're being given :( [1]: https://medium.com/mit-technology-review/the-defense-department-has-produced-the-first-tools-for-catching-deepfakes-db03998f8b65 https://medium.com/mit-technology-review/the-defense-departm...
  - carry_bit 8y ago
    > those people seem to be unlikely to venture out and do the necessary due diligence to verify the information they're being given Especially since people in those circles have for the last year been pushing the idea that the attention on deep fakes is preemptive cover for such a video rumored to be out there.
    - mmjaa 8y ago
      And .. what if it is preemptive cover? What do we do about that circumstance? Like, seriously, this is shaping up to be a cyberwar nobody wants to admit. Either we can trust video sources from now on, or we can't - how is it going to be possible for TPTB to continue to divert public attention with video, if we can no longer trust our own eyes?
      - vokep 8y ago
        Hardware-based signing-while-recording, and only consider previously recorded self-verified unsigned recordings legitimate, otherwise unsigned=assumefake.
        Bartweiss 8y ago
        > only consider previously recorded self-verified unsigned recordings legitimate, otherwise unsigned=assumefake Can you clarify this part, especially "self-verified unsigned"? I don't think I understand it - presumably you could generate camera-and-video-specific signatures which an edit couldn't reproduce, but it's not obvious to me how you would verify them without access to the original camera. As far as sign-while-recording, I agree. We'll quickly end up in a world where video of ongoing events is signed as it's taken. The simplest tactic I can think of is submitting a hash of the video file to some public store, which at least authenticates when it was shot. For an event with a well-known time (a speech, public protest, etc), that should suffice to prove the footage isn't faked, and corroboration of multiple videos should prove it's not an outright fabrication (e.g. pre-rendering a whole different speech). For time-nonspecific events like a video of a suspicious meeting, this wouldn't be enough. A hardware signature might prove that a video was real, but we'd still face a significant change in that unsourced videos floating around the web couldn't be treated as convincing. We're already getting there with photos, of course, but at the moment a clear, high-res video is pretty trustworthy without a source.
- rookonaut 8y ago
  You can watch this video of Obama calling Trump names already: https://www.youtube.com/watch?v=cQ54GDm1eL0 https://www.youtube.com/watch?v=cQ54GDm1eL0 Those times have already begun.
- skummetmaelk 8y ago
  Whether it is done or not doesn't even matter. It also does not matter whether deep faking videos is made illegal. All that matters is that it is possible to do so, and supporters of political candidate A will claim with possible deniability that political candidate A never said "X". While also claiming that political candidate B definitely said "Y", even though this may be false. Soon there will be no way to prove whether or not the videos are actually real and people will believe whatever fits their preconceived notions.
- Zecar 8y ago
  > Already now we have had memes with misattributed quotes and photos edited to associate Clinton with the devil, satanic rituals and so on Right, like my favorite one is the idea that she took $150 million from Russia in exchange for a shipment of our uranium. Those funds went to the Clinton Foundation a _charity_. Sheesh.
  - arcticfox 8y ago
    > Those funds went to the Clinton Foundation a _charity_. Sheesh While independent investigations have concluded there was no wrongdoing, the reason above is really no defense. The Trump Foundation, for example, has shown how nonprofits can be manipulated to the benefit of the founders.
    - travisoneill1 8y ago
      No _proof_ of wrongdoing. They also failed to come up with any plausible reason for the payment other than wrongdoing.
  - Cthulhu_ 8y ago
    Don't all rich people stash their billions in charities to avoid taxes anyway?
  - Chris_Chambers 8y ago
    Wow, you typed "sheesh" with such conviction that you finally persuaded me to believe that the "charity" is not essentially her family's personal slush fund. Pack it up my fellow alt-right neo-Nazis, looks like we got dunked on by an exasperated use of an exclamation yet again.
  - bacondude3 8y ago
    I don't know the details of this Clinton-uranium-$$ story, but I can think of a few questions to ask... 1. If you were going to give somebody an illegal bribe, would you write "bribe" in the For: line on the check? (no, you would hide or disguise your bribe) 2. What plausible, legitimate reasons are there for Russia to donate to the Clinton Foundation? 3. Is charitable giving to anybody, especially in quantities exceeding $25-50 million, normal for Russia? Cthulhu_ and arcticfox are definitely both right that charities are used by many rich people to avoid taxes and just because it's a charity doesn't mean the owner isn't benefiting from it. I'd appreciate hearing your response. Not saying it happened one way or the other, just making some observations based on the tiny bit I've heard about this. Edit: Regarding #2, a quick search turned up [1]. I don't know anything about the Borgen Project, but a quick glance at the page doesn't look too suspicious. So, assuming that page is correct and that Russia donated similar quantities in 2013 as they did the same year the uranium incident happened, they donated over 50% to the Clinton Foundation. If those assumptions are fair and that actually is the case, it sure sounds suspicious to me. [1]: https://borgenproject.org/russias-charitable-giving/ https://borgenproject.org/russias-charitable-giving/
- taneq 8y ago
  Just yesterday? The very first examples we saw of this kind of tech was heads of state being puppeted to say things. That and porn are about all its been used for.
- mc32 8y ago
  Deep fakes lack context. There are no witnesses, etc. So they’ll be most “effective” in trying to establish something in the past. It’ll be harder to produce the context where it’s believable in the near past (present minus up to a day a month a year?). So they may be useful in sowing distrust in someone by establishing they did something untoward in the past, but less likely to incite immediate reaction about something happening now, or recent past, once people become used to the possibility.
  - Casseres 8y ago
    People will "remember" things that didn't happen.
    - pbhjpbhj 8y ago
      People already remember things that didn't happen; and public figures already get maligned by news outlets lying about what they did and where they went. Even when the situations are revealed to be lies.
  - michaelt 8y ago
    Presumably you'd fake things it would be plausible the politician was trying to keep secret. So you'd expect there to be no witnesses. For example you'd fake hidden camera footage of politicians beating prostitutes and accepting brown envelopes from russian agents.
- netsharc 8y ago
  There was a "middle finger scandal" in 2015 where German journalists found a clip of the man who was to be the Greek finance minister giving a speech and saying "[Greece] should stick the finger to Germany" and doing the gesture as well, and of course stoking the typical "outrage culture". Strangely a German TV satire team have claimed they doctored the video to make it look like he's giving the finger: https://www.youtube.com/watch?v=Vx-1LQu6mAE https://www.youtube.com/watch?v=Vx-1LQu6mAE (although I stil wonder if they did the opposite: doctor the gesture out of the video, IMO the gesture itself was par for the course for him and perfectly acceptable).
- sigi45 8y ago
  In theory you would need to visit all events in person again as you will no longer be able to trust anything around you. We would need to invent something like public/private key auth which signs every videostream and which allows you to trust specific sources like reporters from NYT for example.
- twic 8y ago
  We had a crude version of this a few months ago: https://www.marketwatch.com/story/video-of-acosta-incident-posted-by-white-house-press-secretary-contains-extra-frames-2018-11-08 https://www.marketwatch.com/story/video-of-acosta-incident-p...
- cwkoss 8y ago
  IIRC, the majority Pizzagate was centered around Clinton's campaign chair John Podesta, not the candidate herself. John and Tony Podesta collect some very creepy art - google it. Certainly not evidence of a crime, but fairly-public figures closely connected to a presidential campaign collecting art of children bound and in morbid situations raises some eyebrows for sure.
- scotty79 8y ago
  Ok. One generation might be a bit confused thinking that what they see on video is what actually happened. But further generations as the technology becomes common will just intuitively understand that video by itself proves nothing and it was most likely made for the lulz than recorded as it happened. Video of Trump saying something already proves nothing especially not that he ever said it because now he says he never did. And all that even without deep fake technology. People care less and less about history of reality. They become aware that it's to large extent unknowable drowned in narratives.
naringas 8y ago
sooner or later they have to make a full length feature film using this technology
- taneq 8y ago
  I thought there already was (some of) one. Starring Nicholas Cage, as everyone.
  - gbrown 8y ago
    That's amazing.
- Cthulhu_ 8y ago
  Forrest Gump comes to mind, and this was what, 20 years ago?
- naringas 8y ago
  an *entire full length movie
michaelt 8y ago
Does anyone know how this act would interact with the first amendment? It's my understanding that the first amendment is extremely broad - if donating money can be counted as protected speech, surely 'parody' videos of politicians would also be?
- Cthulhu_ 8y ago
  Well making a deep fake is legal, however using it for illegal activities is not - freedom of speech is not freedom of consequences. You're free to make a bomb threat and nobody will silence you if you do, but it's likely the FBI will knock on your door. It's not about what you say (or in this case produce), it's about your intent.
  - dTal 8y ago
    Not to disagree with your statement about intent, but... >freedom of speech is not freedom of consequences ...that's not a very convincing form of free speech. What exactly does "free speech" mean then, if not freedom from consequences? It's not as if it's even possible to silence someone before the fact. "You are free to criticise comrade Stalin! You are also free to go to gulag!"
    - paulgb 8y ago
      >freedom of speech is not freedom of consequences Usually when I see that statement it’s to make the point that freedom of speech (in the first amendment sense) protects you from government consequences, but not societal consequences. The government won’t stop you from saying something offensive, but your country club can kick you out for saying it. I agree with you that this sense doesn’t really apply here, though.
    - zaarn 8y ago
      Merriam-Webster defines freedom of speech as "the legal right to express one's opinions freely". Creating a deep fake of a celebrity hardly counts as expressing your opinions and if you use it for illegal activities that's far out of scope.
daenz 8y ago
IANAL, but based on the wording, it seems like it's only unlawful to create w/ intent to distribute, or to knowingly distribute, a deep fake IFF you're facilitating unlawful conduct. So for example, blackmail? Does libel count? Does the very act of distributing a deep fake count as libel against the subject? Can any lawyers chime in?
jawns 8y ago
From the definitions section of the act: > The term 'deep fake' means an audiovisual record created or altered in a manner that the record would falsely appear to a reasonable observer to be an authentic record of the actual speech or conduct of an individual. This limits the scope of the act to prohibiting deep fakes that are not explicitly labeled as such. If the recording (or, potentially, links to the recording) contains any type of disclaimer, that should be sufficient to establish that no reasonable observer would consider it to be authentic. But I wonder if this provides enough protection to victims in the case of deep fakes where a person's face is grafted onto sexual content or other content that might embarrass the person, whether or not a disclaimer exists. For instance, if someone puts an acquaintance's face into a sex scene and distributes it online, then even if there's a huge "This is a deep fake" scrolling disclaimer across the video, that person may still feel as if they have been defamed.
- londons_explore 8y ago
  It seems quite a constraint on creativity to say 'You can't make something look legit when it isn't'.
  - gotocake 8y ago
    It’s not a new concept, and it applies to the difference between a painter copying a Van Gogh (acceptable) and trying to pass that work off as the genuine article (illegal). In short, it’s the line between creative expression and forgery.
  - vorpalhex 8y ago
    Or, you have to at least put a disclaimer on it if it's likely to cause confusion. This isn't new. > All Characters and events in this show -- even those based on real people -- are entirely fictional. All celebrity voices are impersonated... poorly. The following ...
  - pjc50 8y ago
    As others have pointed out, "making something look legit when it isn't" is a good description of fraud, and films have long had disclaimers in the credits that they are fictional.
bitL 8y ago
It's funny how a very simple DL architecture any beginner could prepare ended up in a prohibition act - essentially training two convolutional autoencoders, one on the face one wants to replace (extracting dataset by e.g. tracking face with OpenCV in target video), the second on many pictures of the face one wants to replace (e.g. scraping from Internet or known videos), then essentially gluing together encoder from one and decoder from the other, assuming latent variables match, and suddenly a fake video is done. IMO if you really want to control deep fakes, you should use a blockchain and track all processing steps, starting from image/video acquisition. I don't want to give anyone ideas, but they will do it anyway.
- dieblur 8y ago
  This doesn't even begin to make sense. You can't force people to register any creation on the blockchain and there can't be a main "only real things allowed" chain. If we want to shoehorn a blockchain into every possible problem ever, lets begin with ones that could actually work.
  - bitL 8y ago
    The idea is to enforce equipment manufacturers to prepare uniquely identifiable IDs in all imagery/footage their equipment produces, entice all software vendors to register all operations in a blockchain and store a reference to it in metadata; then reject any footage that can't have all steps verified as "fake". How is that any different than tracking production inputs in logistic chains using blockchain?
    - xamuel 8y ago
      Nothing can stop me from pointing a camera at my computer monitor and recording a verified video of a non-verified video.
      - Skrillex 8y ago
        I believe that the usual response to this is that the devices include some kind of unmodifiable time and location stamping, although that argument spirals out of control pretty quickly.
c0nfused 8y ago
It would seem to ban only international or interstate created deep fakes. This appears to prop up the locally created deep fake market. I'm only half kidding, it seems like the immediate defense would be: I made it at home or in a data center in my home state thus the statute does not apply.
- grigjd3 8y ago
  I suspect this limitation is because the supreme Court has traditionally held the federal government to such limitations in many areas.
  - the_pwner224 8y ago
    The US Constitution only gives the Federal government the power to regulate international and interstate commerce (and of course other powers too, but those aren't relevant here). Article I section 8: https://www.archives.gov/founding-docs/constitution-transcript#toc-section-8- https://www.archives.gov/founding-docs/constitution-transcri... The Bill of Rights (first 10 amendments to the Constitution) explicitly states that all powers not explicitly granted to the Federal government are retained by the states and by the people. So that's probably why this limitation exists. Of course, the Federal government has really stretched the meaning of "To regulate Commerce ... among the several states" and has gained quite a bit of power out of that line. But it's still a limitation.
jdpedrie 8y ago
"Deep Fakes" in general seem to be already covered by existing libel laws, though that has some problems. Libel is generally a civil matter, and is extremely difficult to prosecute. There is precedent, even in the US, for criminal libel laws[0]. Perhaps following that path (combined with continued work on detection and defeat of the technology) would be preferable. Since libel and defamation are well-defined and have a long history of jurisprudence, many of the constitutional and legal issues which would have to be answered in opening a new avenue of speech restriction could be contained within the existing contexts. I'd even think leaving it in the civil arena would make sense until it became a problem worthy of criminalization. Legislating a matter before it arises is almost never a good idea, from either a prudential or principled standpoint. [0] https://www.washingtonpost.com/news/volokh-conspiracy/wp/2014/05/13/impersonating-someone-online-with-intent-to-injure-his-reputation-is-a-crime-in-new-york/ https://www.washingtonpost.com/news/volokh-conspiracy/wp/201...
- Bartweiss 8y ago
  Aside from the civil versus criminal distinction, it seems significant that this statute judges harm differently than libel laws. Libel/slander requires reputational damage to an individual, while this is about facilitating criminal/tortious conduct in general. That seems to cover some fairly significant non-libel cases, like knowingly creating a video for someone to use as an alibi. I agree that it still seems like premature legislation, though. Even from the viewpoint of a random programmer, it's both overbroad (this covers any video editing, not just 'deepfakes' as commonly understood) and incomplete (what happens when no recognizable people are represented, but an edit is still used to facilitate criminal action?) If doctored video becomes as believable as authentic video, it's going to be a major upheaval, returning us to a world where seeing isn't believing. Probably not completely; there will be an obvious market for hard-to-fake authentication, even in forms as simple as registering a video hash with a trusted source as soon as it's taken. But the departure from a world where a high-res video of an event is reliable proof is going to be a very big change, and I seriously doubt any law written today will productively adjust for it.
beager 8y ago
With respect to the U.S. Congress, I don't think there is sufficient language to make something like this sufficient without also being overly broad. I'm looking at this, specifically: 10 ‘‘(2) the term ‘deep fake’ means an audiovisual 11 record created or altered in a manner that the 12 record would falsely appear to a reasonable observer 13 to be an authentic record of the actual speech or 14 conduct of an individual; and If that's the case, any sort of creative editing, even just quick cuts, could fall under this (see: any primetime or cable news, any TV campaign ad, the quick cuts of Obama where it looks like he's singing Never Gonna Give You Up, etc). And, not to get on the US politics slant, but a law like this could be weaponized against political foes—basically, label everything you don't like as "fake news" and prosecute it under this law. Additionally, if you look at this and just say "well, we all know what a deep fake is, so your point is moot," I will say, somewhat at the risk of contradicting myself, that maybe the language needs to be forward-thinking to cover whatever the next "deep fake" is. In my opinion, the sort of clause above would be better written like: The term "Computer-generated audiovisual impersonation" means an audiovisual record created or altered by computer generation in a manner that the record would falsely appear to a reasonable observer to be an authentic record of the actual speech or conduct of an individual;
- e40 8y ago
  If that's the case, any sort of creative editing, even just quick cuts, could fall under this Maybe that wouldn't be so bad. This technique has been used to deceive untold times.
  - kbenson 8y ago
    I agree. A world where edited interviews must air with a disclaimer that it has been edited to remove certain portions might be extremely beneficial.
    - mcbits 8y ago
      Almost every prerecorded interview would carry the disclaimer, rendering it meaningless. Though it would be fun to see "fake reaction" every time they splice in footage of the interviewer nodding, smiling, scowling, etc.
echelon 8y ago
Jeez, yet another annoying new law encroaching on something fun I do as a hobby. I'm really sick of our lawmakers doing this when our existing legal code seems sufficient to punish abuse. Does this mean I should take down my Donald Trump text to speech engine [1]? Or consult a lawyer? It yields really poor quality (right now), and I doubt any reasonable person would consider it to be actual audio from Trump. Does this prohibit me from improving it? I was about to train an ML model on my samples and switch to parametric generation. [1] http://trumped.com http://trumped.com
donatj 8y ago
I mean deep fakes clearly seems protected by the first amendment, particularly strongly in the case of parody. That's not too say however they are clear from libel law.
- hopler 8y ago
  So are they protected or not?
  - donatj 8y ago
    Basically my argument is that they're protected from criminal but not civil charges.
whocares123 8y ago
Us civilians cant do it. But bet your ass government can. Isnt right.
pseingatl 8y ago
The crime is not well-defined. Meanwhile, there's this out without further explanation: "No person shall be held liable under this section for any activity protected by the First Amendment to the Constitution of the United States.’’
- curuinor 8y ago
  Lots of legislation effectively leaves it to courts to hash the actual specifics out. Courts can only rule on specific examples, so it prevents a genre of legal stupidity.
rjf72 8y ago
I'm curious about peoples' views on something. It's clear that in the future we will be able to create fakes that are effectively indistinguishable from reality. The audio in this 'Trump speech' [1] is already remarkable. So there are two ways we can go from here. The first is to try to maintain faith in multimedia. What you see or hear is probably real because we try to pass a bunch of laws making it a really bad thing to try to impersonate people. The second is to go the route of internet speech today. If somebody claims to be somebody of note, you generally would not believe them without extensive proof. And so what they say does not reflect upon the person they claim to be. If video/audio manipulation tech was allowed without constraint, this would eventually become the same for general audio/video. Having no trust in what you see or hear is not a great thing. But at the same time, I think there's a very good argument to be made for the fact that people are already far too susceptible to fake information because we, even before the 'real' advent of deep fake type technology, are still teetering on the precipice between believable and not. For instance this [2] famous image of "animal testing" that keeps going viral on social media every couple of years. It has nothing to do with animal testing, but people are naive. Deep fakes would throw us well off that precipice to the point that I think we'd see substantially increased amounts of scrutiny given to misinformation. The downside here is of course we'd also see substantially increased amounts of scrutiny given to legitimate information, though I'm not entirely sure I see that as a negative. [1] - https://www.youtube.com/watch?v=7Gpc_artOYI https://www.youtube.com/watch?v=7Gpc_artOYI [2] - https://speakingofresearch.com/2014/02/27/fact-into-fiction-why-context-matters-with-animal-images/ https://speakingofresearch.com/2014/02/27/fact-into-fiction-...
adolph 8y ago
I'd bet that this act would paradoxically increase the power of "deep fakes" in that the problem isn't the existence of "deep fakes" but each person's reaction to them. The prevailing culture of knee-jerk hot-takes already thrives on selectively edited video and fresh discoveries of forgotten pasts. If it isn't inoculated into thoughtful skepticism by steady revelations of various fakes it will become more susceptible to the few left un-prosecuted under this law.
cpr 8y ago
In other words: deep state sets the stage for forthcoming shocking real videos of major players. Yes, yes, downvote into oblivion, but remember this when it happens...
ataturk 8y ago
Let's be honest, the scuttlebutt over "Deep Fakes" is because there are blackmail videos of our elite committing unspeakable acts and they are trying to play this off as "fake videos" when in fact the videos are real. It seems that you don't achieve the higher echelons of power without someone holding your leash. Interesting, no?
pbhjpbhj 8y ago
How does one prove the fake is fake. There's going to be development of "live notarised" data streams, so a camera feed gets certified, etc.. People are going to use deep-fake tech as an excuse -- someone faked me, I'm not racist/sexist/fascist/... . How can you show the real is real. I'm imagining a future where you can choose the actors in the lead roles of your films; great for narcissists!
- tlynchpin 8y ago
  Heinlein explored this idea of Fair Witness as a profession in his 1961 novel Stranger in a Strange Land. https://lccn.loc.gov/61011702 https://lccn.loc.gov/61011702
saint_abroad 8y ago
> (2) the term ‘deep fake’ means an audiovisual record created or altered in a manner that the record would falsely appear to a reasonable observer to be an authentic record of the actual speech or conduct of an individual; I see nothing of this act that restricts the applicability of so broad a definition of "deep-fake" so as not to cover the activities of Hollywood, specifically the CGI mapping of actors onto body doubles such as that performed in "The Crow" (1994). Yet another deeply troubling knee-jerk reaction of an act that promises to catch "just the bad".
- cwkoss 8y ago
  Conversely, I think that we may soon need regulation around Hollywood's right to digitally reincarnate dead actors for use in their commercial film productions. The idea of Universal CGI'ing George Carlin into saying politically-tinged speech he would disagree with would be a travesty. And I'm not sure how that would be prevented, save his estate successfully suing Universal.
randyrand 8y ago
This concerns 'records', but I wonder if a program that can do a realtime overlay would solve the moral/possible legal issues.

121 points by Hard_Space 8y ago

Just yesterday it occurred to me that Deep Fakes could be used to create some of the most potent disinformation campaigns ever. Already now we have had memes with misattributed quotes and photos edited to associate Clinton with the devil, satanic rituals and so on. Think of how much more impactful slightly off quotes and speeches would be.

samcday 8y ago
Yes, it definitely has the potential to become a major issue. So much so that the DoD is already working on tools to detect deepfakes [1]. Of course this will very likely devolve into an arms race akin to antivirus vendors vs malware developers. The fundamental problem is going to be that many people seem to actually like being in their comfortable echo chamber. So, if they're presented with a video that reaffirms their hunch that the Clintons run a child trafficking ring from under a pizza store, those people seem to be unlikely to venture out and do the necessary due diligence to verify the information they're being given :( [1]: https://medium.com/mit-technology-review/the-defense-department-has-produced-the-first-tools-for-catching-deepfakes-db03998f8b65 https://medium.com/mit-technology-review/the-defense-departm...
- carry_bit 8y ago
  > those people seem to be unlikely to venture out and do the necessary due diligence to verify the information they're being given Especially since people in those circles have for the last year been pushing the idea that the attention on deep fakes is preemptive cover for such a video rumored to be out there.
  - mmjaa 8y ago
    And .. what if it is preemptive cover? What do we do about that circumstance? Like, seriously, this is shaping up to be a cyberwar nobody wants to admit. Either we can trust video sources from now on, or we can't - how is it going to be possible for TPTB to continue to divert public attention with video, if we can no longer trust our own eyes?
    - vokep 8y ago
      Hardware-based signing-while-recording, and only consider previously recorded self-verified unsigned recordings legitimate, otherwise unsigned=assumefake.
      - Bartweiss 8y ago
        > only consider previously recorded self-verified unsigned recordings legitimate, otherwise unsigned=assumefake Can you clarify this part, especially "self-verified unsigned"? I don't think I understand it - presumably you could generate camera-and-video-specific signatures which an edit couldn't reproduce, but it's not obvious to me how you would verify them without access to the original camera. As far as sign-while-recording, I agree. We'll quickly end up in a world where video of ongoing events is signed as it's taken. The simplest tactic I can think of is submitting a hash of the video file to some public store, which at least authenticates when it was shot. For an event with a well-known time (a speech, public protest, etc), that should suffice to prove the footage isn't faked, and corroboration of multiple videos should prove it's not an outright fabrication (e.g. pre-rendering a whole different speech). For time-nonspecific events like a video of a suspicious meeting, this wouldn't be enough. A hardware signature might prove that a video was real, but we'd still face a significant change in that unsourced videos floating around the web couldn't be treated as convincing. We're already getting there with photos, of course, but at the moment a clear, high-res video is pretty trustworthy without a source.
rookonaut 8y ago
You can watch this video of Obama calling Trump names already: https://www.youtube.com/watch?v=cQ54GDm1eL0 https://www.youtube.com/watch?v=cQ54GDm1eL0 Those times have already begun.
skummetmaelk 8y ago
Whether it is done or not doesn't even matter. It also does not matter whether deep faking videos is made illegal. All that matters is that it is possible to do so, and supporters of political candidate A will claim with possible deniability that political candidate A never said "X". While also claiming that political candidate B definitely said "Y", even though this may be false. Soon there will be no way to prove whether or not the videos are actually real and people will believe whatever fits their preconceived notions.
Zecar 8y ago
> Already now we have had memes with misattributed quotes and photos edited to associate Clinton with the devil, satanic rituals and so on Right, like my favorite one is the idea that she took $150 million from Russia in exchange for a shipment of our uranium. Those funds went to the Clinton Foundation a _charity_. Sheesh.
- arcticfox 8y ago
  > Those funds went to the Clinton Foundation a _charity_. Sheesh While independent investigations have concluded there was no wrongdoing, the reason above is really no defense. The Trump Foundation, for example, has shown how nonprofits can be manipulated to the benefit of the founders.
  - travisoneill1 8y ago
    No _proof_ of wrongdoing. They also failed to come up with any plausible reason for the payment other than wrongdoing.
- Cthulhu_ 8y ago
  Don't all rich people stash their billions in charities to avoid taxes anyway?
- Chris_Chambers 8y ago
  Wow, you typed "sheesh" with such conviction that you finally persuaded me to believe that the "charity" is not essentially her family's personal slush fund. Pack it up my fellow alt-right neo-Nazis, looks like we got dunked on by an exasperated use of an exclamation yet again.
- bacondude3 8y ago
  I don't know the details of this Clinton-uranium-$$ story, but I can think of a few questions to ask... 1. If you were going to give somebody an illegal bribe, would you write "bribe" in the For: line on the check? (no, you would hide or disguise your bribe) 2. What plausible, legitimate reasons are there for Russia to donate to the Clinton Foundation? 3. Is charitable giving to anybody, especially in quantities exceeding $25-50 million, normal for Russia? Cthulhu_ and arcticfox are definitely both right that charities are used by many rich people to avoid taxes and just because it's a charity doesn't mean the owner isn't benefiting from it. I'd appreciate hearing your response. Not saying it happened one way or the other, just making some observations based on the tiny bit I've heard about this. Edit: Regarding #2, a quick search turned up [1]. I don't know anything about the Borgen Project, but a quick glance at the page doesn't look too suspicious. So, assuming that page is correct and that Russia donated similar quantities in 2013 as they did the same year the uranium incident happened, they donated over 50% to the Clinton Foundation. If those assumptions are fair and that actually is the case, it sure sounds suspicious to me. [1]: https://borgenproject.org/russias-charitable-giving/ https://borgenproject.org/russias-charitable-giving/
taneq 8y ago
Just yesterday? The very first examples we saw of this kind of tech was heads of state being puppeted to say things. That and porn are about all its been used for.
mc32 8y ago
Deep fakes lack context. There are no witnesses, etc. So they’ll be most “effective” in trying to establish something in the past. It’ll be harder to produce the context where it’s believable in the near past (present minus up to a day a month a year?). So they may be useful in sowing distrust in someone by establishing they did something untoward in the past, but less likely to incite immediate reaction about something happening now, or recent past, once people become used to the possibility.
- Casseres 8y ago
  People will "remember" things that didn't happen.
  - pbhjpbhj 8y ago
    People already remember things that didn't happen; and public figures already get maligned by news outlets lying about what they did and where they went. Even when the situations are revealed to be lies.
- michaelt 8y ago
  Presumably you'd fake things it would be plausible the politician was trying to keep secret. So you'd expect there to be no witnesses. For example you'd fake hidden camera footage of politicians beating prostitutes and accepting brown envelopes from russian agents.
netsharc 8y ago
There was a "middle finger scandal" in 2015 where German journalists found a clip of the man who was to be the Greek finance minister giving a speech and saying "[Greece] should stick the finger to Germany" and doing the gesture as well, and of course stoking the typical "outrage culture". Strangely a German TV satire team have claimed they doctored the video to make it look like he's giving the finger: https://www.youtube.com/watch?v=Vx-1LQu6mAE https://www.youtube.com/watch?v=Vx-1LQu6mAE (although I stil wonder if they did the opposite: doctor the gesture out of the video, IMO the gesture itself was par for the course for him and perfectly acceptable).
sigi45 8y ago
In theory you would need to visit all events in person again as you will no longer be able to trust anything around you. We would need to invent something like public/private key auth which signs every videostream and which allows you to trust specific sources like reporters from NYT for example.
twic 8y ago
We had a crude version of this a few months ago: https://www.marketwatch.com/story/video-of-acosta-incident-posted-by-white-house-press-secretary-contains-extra-frames-2018-11-08 https://www.marketwatch.com/story/video-of-acosta-incident-p...
cwkoss 8y ago
IIRC, the majority Pizzagate was centered around Clinton's campaign chair John Podesta, not the candidate herself. John and Tony Podesta collect some very creepy art - google it. Certainly not evidence of a crime, but fairly-public figures closely connected to a presidential campaign collecting art of children bound and in morbid situations raises some eyebrows for sure.
scotty79 8y ago
Ok. One generation might be a bit confused thinking that what they see on video is what actually happened. But further generations as the technology becomes common will just intuitively understand that video by itself proves nothing and it was most likely made for the lulz than recorded as it happened. Video of Trump saying something already proves nothing especially not that he ever said it because now he says he never did. And all that even without deep fake technology. People care less and less about history of reality. They become aware that it's to large extent unknowable drowned in narratives.

naringas 8y ago

sooner or later they have to make a full length feature film using this technology

taneq 8y ago
I thought there already was (some of) one. Starring Nicholas Cage, as everyone.
- gbrown 8y ago
  That's amazing.
Cthulhu_ 8y ago
Forrest Gump comes to mind, and this was what, 20 years ago?
naringas 8y ago
an *entire full length movie

michaelt 8y ago

Does anyone know how this act would interact with the first amendment? It's my understanding that the first amendment is extremely broad - if donating money can be counted as protected speech, surely 'parody' videos of politicians would also be?

Cthulhu_ 8y ago
Well making a deep fake is legal, however using it for illegal activities is not - freedom of speech is not freedom of consequences. You're free to make a bomb threat and nobody will silence you if you do, but it's likely the FBI will knock on your door. It's not about what you say (or in this case produce), it's about your intent.
- dTal 8y ago
  Not to disagree with your statement about intent, but... >freedom of speech is not freedom of consequences ...that's not a very convincing form of free speech. What exactly does "free speech" mean then, if not freedom from consequences? It's not as if it's even possible to silence someone before the fact. "You are free to criticise comrade Stalin! You are also free to go to gulag!"
  - paulgb 8y ago
    >freedom of speech is not freedom of consequences Usually when I see that statement it’s to make the point that freedom of speech (in the first amendment sense) protects you from government consequences, but not societal consequences. The government won’t stop you from saying something offensive, but your country club can kick you out for saying it. I agree with you that this sense doesn’t really apply here, though.
  - zaarn 8y ago
    Merriam-Webster defines freedom of speech as "the legal right to express one's opinions freely". Creating a deep fake of a celebrity hardly counts as expressing your opinions and if you use it for illegal activities that's far out of scope.

daenz 8y ago

IANAL, but based on the wording, it seems like it's only unlawful to create w/ intent to distribute, or to knowingly distribute, a deep fake IFF you're facilitating unlawful conduct. So for example, blackmail? Does libel count? Does the very act of distributing a deep fake count as libel against the subject? Can any lawyers chime in?

jawns 8y ago

From the definitions section of the act: > The term 'deep fake' means an audiovisual record created or altered in a manner that the record would falsely appear to a reasonable observer to be an authentic record of the actual speech or conduct of an individual. This limits the scope of the act to prohibiting deep fakes that are not explicitly labeled as such. If the recording (or, potentially, links to the recording) contains any type of disclaimer, that should be sufficient to establish that no reasonable observer would consider it to be authentic. But I wonder if this provides enough protection to victims in the case of deep fakes where a person's face is grafted onto sexual content or other content that might embarrass the person, whether or not a disclaimer exists. For instance, if someone puts an acquaintance's face into a sex scene and distributes it online, then even if there's a huge "This is a deep fake" scrolling disclaimer across the video, that person may still feel as if they have been defamed.

londons_explore 8y ago
It seems quite a constraint on creativity to say 'You can't make something look legit when it isn't'.
- gotocake 8y ago
  It’s not a new concept, and it applies to the difference between a painter copying a Van Gogh (acceptable) and trying to pass that work off as the genuine article (illegal). In short, it’s the line between creative expression and forgery.
- vorpalhex 8y ago
  Or, you have to at least put a disclaimer on it if it's likely to cause confusion. This isn't new. > All Characters and events in this show -- even those based on real people -- are entirely fictional. All celebrity voices are impersonated... poorly. The following ...
- pjc50 8y ago
  As others have pointed out, "making something look legit when it isn't" is a good description of fraud, and films have long had disclaimers in the credits that they are fictional.

bitL 8y ago

It's funny how a very simple DL architecture any beginner could prepare ended up in a prohibition act - essentially training two convolutional autoencoders, one on the face one wants to replace (extracting dataset by e.g. tracking face with OpenCV in target video), the second on many pictures of the face one wants to replace (e.g. scraping from Internet or known videos), then essentially gluing together encoder from one and decoder from the other, assuming latent variables match, and suddenly a fake video is done. IMO if you really want to control deep fakes, you should use a blockchain and track all processing steps, starting from image/video acquisition. I don't want to give anyone ideas, but they will do it anyway.

dieblur 8y ago
This doesn't even begin to make sense. You can't force people to register any creation on the blockchain and there can't be a main "only real things allowed" chain. If we want to shoehorn a blockchain into every possible problem ever, lets begin with ones that could actually work.
- bitL 8y ago
  The idea is to enforce equipment manufacturers to prepare uniquely identifiable IDs in all imagery/footage their equipment produces, entice all software vendors to register all operations in a blockchain and store a reference to it in metadata; then reject any footage that can't have all steps verified as "fake". How is that any different than tracking production inputs in logistic chains using blockchain?
  - xamuel 8y ago
    Nothing can stop me from pointing a camera at my computer monitor and recording a verified video of a non-verified video.
    - Skrillex 8y ago
      I believe that the usual response to this is that the devices include some kind of unmodifiable time and location stamping, although that argument spirals out of control pretty quickly.

c0nfused 8y ago

It would seem to ban only international or interstate created deep fakes. This appears to prop up the locally created deep fake market. I'm only half kidding, it seems like the immediate defense would be: I made it at home or in a data center in my home state thus the statute does not apply.

grigjd3 8y ago
I suspect this limitation is because the supreme Court has traditionally held the federal government to such limitations in many areas.
- the_pwner224 8y ago
  The US Constitution only gives the Federal government the power to regulate international and interstate commerce (and of course other powers too, but those aren't relevant here). Article I section 8: https://www.archives.gov/founding-docs/constitution-transcript#toc-section-8- https://www.archives.gov/founding-docs/constitution-transcri... The Bill of Rights (first 10 amendments to the Constitution) explicitly states that all powers not explicitly granted to the Federal government are retained by the states and by the people. So that's probably why this limitation exists. Of course, the Federal government has really stretched the meaning of "To regulate Commerce ... among the several states" and has gained quite a bit of power out of that line. But it's still a limitation.

jdpedrie 8y ago

"Deep Fakes" in general seem to be already covered by existing libel laws, though that has some problems. Libel is generally a civil matter, and is extremely difficult to prosecute. There is precedent, even in the US, for criminal libel laws[0]. Perhaps following that path (combined with continued work on detection and defeat of the technology) would be preferable. Since libel and defamation are well-defined and have a long history of jurisprudence, many of the constitutional and legal issues which would have to be answered in opening a new avenue of speech restriction could be contained within the existing contexts. I'd even think leaving it in the civil arena would make sense until it became a problem worthy of criminalization. Legislating a matter before it arises is almost never a good idea, from either a prudential or principled standpoint. [0] https://www.washingtonpost.com/news/volokh-conspiracy/wp/2014/05/13/impersonating-someone-online-with-intent-to-injure-his-reputation-is-a-crime-in-new-york/ https://www.washingtonpost.com/news/volokh-conspiracy/wp/201...

Bartweiss 8y ago
Aside from the civil versus criminal distinction, it seems significant that this statute judges harm differently than libel laws. Libel/slander requires reputational damage to an individual, while this is about facilitating criminal/tortious conduct in general. That seems to cover some fairly significant non-libel cases, like knowingly creating a video for someone to use as an alibi. I agree that it still seems like premature legislation, though. Even from the viewpoint of a random programmer, it's both overbroad (this covers any video editing, not just 'deepfakes' as commonly understood) and incomplete (what happens when no recognizable people are represented, but an edit is still used to facilitate criminal action?) If doctored video becomes as believable as authentic video, it's going to be a major upheaval, returning us to a world where seeing isn't believing. Probably not completely; there will be an obvious market for hard-to-fake authentication, even in forms as simple as registering a video hash with a trusted source as soon as it's taken. But the departure from a world where a high-res video of an event is reliable proof is going to be a very big change, and I seriously doubt any law written today will productively adjust for it.

beager 8y ago

With respect to the U.S. Congress, I don't think there is sufficient language to make something like this sufficient without also being overly broad. I'm looking at this, specifically: 10 ‘‘(2) the term ‘deep fake’ means an audiovisual 11 record created or altered in a manner that the 12 record would falsely appear to a reasonable observer 13 to be an authentic record of the actual speech or 14 conduct of an individual; and If that's the case, any sort of creative editing, even just quick cuts, could fall under this (see: any primetime or cable news, any TV campaign ad, the quick cuts of Obama where it looks like he's singing Never Gonna Give You Up, etc). And, not to get on the US politics slant, but a law like this could be weaponized against political foes—basically, label everything you don't like as "fake news" and prosecute it under this law. Additionally, if you look at this and just say "well, we all know what a deep fake is, so your point is moot," I will say, somewhat at the risk of contradicting myself, that maybe the language needs to be forward-thinking to cover whatever the next "deep fake" is. In my opinion, the sort of clause above would be better written like: The term "Computer-generated audiovisual impersonation" means an audiovisual record created or altered by computer generation in a manner that the record would falsely appear to a reasonable observer to be an authentic record of the actual speech or conduct of an individual;

e40 8y ago
If that's the case, any sort of creative editing, even just quick cuts, could fall under this Maybe that wouldn't be so bad. This technique has been used to deceive untold times.
- kbenson 8y ago
  I agree. A world where edited interviews must air with a disclaimer that it has been edited to remove certain portions might be extremely beneficial.
  - mcbits 8y ago
    Almost every prerecorded interview would carry the disclaimer, rendering it meaningless. Though it would be fun to see "fake reaction" every time they splice in footage of the interviewer nodding, smiling, scowling, etc.

echelon 8y ago

Jeez, yet another annoying new law encroaching on something fun I do as a hobby. I'm really sick of our lawmakers doing this when our existing legal code seems sufficient to punish abuse. Does this mean I should take down my Donald Trump text to speech engine [1]? Or consult a lawyer? It yields really poor quality (right now), and I doubt any reasonable person would consider it to be actual audio from Trump. Does this prohibit me from improving it? I was about to train an ML model on my samples and switch to parametric generation. [1] http://trumped.com http://trumped.com

donatj 8y ago

I mean deep fakes clearly seems protected by the first amendment, particularly strongly in the case of parody. That's not too say however they are clear from libel law.

hopler 8y ago
So are they protected or not?
- donatj 8y ago
  Basically my argument is that they're protected from criminal but not civil charges.

whocares123 8y ago

Us civilians cant do it. But bet your ass government can. Isnt right.

pseingatl 8y ago

The crime is not well-defined. Meanwhile, there's this out without further explanation: "No person shall be held liable under this section for any activity protected by the First Amendment to the Constitution of the United States.’’

curuinor 8y ago
Lots of legislation effectively leaves it to courts to hash the actual specifics out. Courts can only rule on specific examples, so it prevents a genre of legal stupidity.

rjf72 8y ago

I'm curious about peoples' views on something. It's clear that in the future we will be able to create fakes that are effectively indistinguishable from reality. The audio in this 'Trump speech' [1] is already remarkable. So there are two ways we can go from here. The first is to try to maintain faith in multimedia. What you see or hear is probably real because we try to pass a bunch of laws making it a really bad thing to try to impersonate people. The second is to go the route of internet speech today. If somebody claims to be somebody of note, you generally would not believe them without extensive proof. And so what they say does not reflect upon the person they claim to be. If video/audio manipulation tech was allowed without constraint, this would eventually become the same for general audio/video. Having no trust in what you see or hear is not a great thing. But at the same time, I think there's a very good argument to be made for the fact that people are already far too susceptible to fake information because we, even before the 'real' advent of deep fake type technology, are still teetering on the precipice between believable and not. For instance this [2] famous image of "animal testing" that keeps going viral on social media every couple of years. It has nothing to do with animal testing, but people are naive. Deep fakes would throw us well off that precipice to the point that I think we'd see substantially increased amounts of scrutiny given to misinformation. The downside here is of course we'd also see substantially increased amounts of scrutiny given to legitimate information, though I'm not entirely sure I see that as a negative. [1] - https://www.youtube.com/watch?v=7Gpc_artOYI https://www.youtube.com/watch?v=7Gpc_artOYI [2] - https://speakingofresearch.com/2014/02/27/fact-into-fiction-why-context-matters-with-animal-images/ https://speakingofresearch.com/2014/02/27/fact-into-fiction-...

adolph 8y ago

I'd bet that this act would paradoxically increase the power of "deep fakes" in that the problem isn't the existence of "deep fakes" but each person's reaction to them. The prevailing culture of knee-jerk hot-takes already thrives on selectively edited video and fresh discoveries of forgotten pasts. If it isn't inoculated into thoughtful skepticism by steady revelations of various fakes it will become more susceptible to the few left un-prosecuted under this law.

cpr 8y ago

In other words: deep state sets the stage for forthcoming shocking real videos of major players. Yes, yes, downvote into oblivion, but remember this when it happens...

ataturk 8y ago

Let's be honest, the scuttlebutt over "Deep Fakes" is because there are blackmail videos of our elite committing unspeakable acts and they are trying to play this off as "fake videos" when in fact the videos are real. It seems that you don't achieve the higher echelons of power without someone holding your leash. Interesting, no?

pbhjpbhj 8y ago

How does one prove the fake is fake. There's going to be development of "live notarised" data streams, so a camera feed gets certified, etc.. People are going to use deep-fake tech as an excuse -- someone faked me, I'm not racist/sexist/fascist/... . How can you show the real is real. I'm imagining a future where you can choose the actors in the lead roles of your films; great for narcissists!

tlynchpin 8y ago
Heinlein explored this idea of Fair Witness as a profession in his 1961 novel Stranger in a Strange Land. https://lccn.loc.gov/61011702 https://lccn.loc.gov/61011702

saint_abroad 8y ago

> (2) the term ‘deep fake’ means an audiovisual record created or altered in a manner that the record would falsely appear to a reasonable observer to be an authentic record of the actual speech or conduct of an individual; I see nothing of this act that restricts the applicability of so broad a definition of "deep-fake" so as not to cover the activities of Hollywood, specifically the CGI mapping of actors onto body doubles such as that performed in "The Crow" (1994). Yet another deeply troubling knee-jerk reaction of an act that promises to catch "just the bad".

cwkoss 8y ago
Conversely, I think that we may soon need regulation around Hollywood's right to digitally reincarnate dead actors for use in their commercial film productions. The idea of Universal CGI'ing George Carlin into saying politically-tinged speech he would disagree with would be a travesty. And I'm not sure how that would be prevented, save his estate successfully suing Universal.

randyrand 8y ago

This concerns 'records', but I wonder if a program that can do a realtime overlay would solve the moral/possible legal issues.