5 ms·

To those who ask about the ethics of releasing something like this, I'd say that this technology already exists, and bad actors probably already can get access

by sva_ 4y ago

To those who ask about the ethics of releasing something like this, I'd say that this technology already exists, and bad actors probably already can get access if they really want to and are sophisticated enough. Making this available to the general public will spread awareness of the existence of such tools, and can then possibly have a preventive effect.

belter 4y ago
You are right. I saw some guy that looked like Matt Damon trying to sell me some crypto coins...
- karmakurtisaani 4y ago
  It would be so ridiculously crazy that an established A-list actor would willingly promote those crypto scams. Must have been a deep fake!
- HWR_14 4y ago
  For what it's worth, actors seem to be securing themselves against this using IP rights.
bko 4y ago
I agree with everything you said, but I we shouldn't deny that opportunistic bad actors don't exist. Or it might get on their radar and be exploited. Open source tools also tend to be better maintained, documented and reliable, so the bad guys will have a better tool. That being said, bringing it to light also has benefits like you said. If the tool is out in the open and state of the art techniques are used, technology to detect its use will also benefit.
blagie 4y ago
As someone with a stalker, I can't emphasize this enough. A stalker will go to all sorts of lengths to do bizarre shit. People don't believe it. I would guess governments will do some equivalent there-of. Democratizing access to things -- including bad things -- has a preventative effect: 1) I can guard against things I know about 2) People take me seriously if something has been democratized The worst-case scenario is if my stalker got her hands on something like deep fake technology before the police / prosecutor / jury didn't knew it existed. I'd probably be in jail by now if something like that ever happened. She's tried to frame me twice before. Fortunately, they were transparent. She'll try again. Best case scenario is that no one has access to this stuff. Worst case scenario is only a select group have access, and most people don't know about it. Universal access is somewhere in between.
- RobertRoberts 4y ago
  I want to second this, as there are so many people that just simply can't believe how much time and energy some people will put into destroying someone else's life. And when you ask for help, people think you are the insane one because they simply can't believe your story about the insanity of someone else. I hope you find relief sometime from your stalker. I found it (not a stalker exactly) from letting the person burn themselves with their behavior so many times without me doing or saying anything in return to them (my strategy of non-direct conflict, and it worked for me), that eventually they ran out of people to manipulate and fool.
  - Moru 4y ago
    A worse situation: What if you ARE insane, it's just that this time it's not your imagination running wild, it's actually someone stalking you? People that have some mental problem, whatever it is, have a much harder time since no one will believe them. It's always just "I know you are feeling bad right now..." even though the diagnose is ADHD or something else irrelevant.
- HWR_14 4y ago
  Your "worst case" depends greatly on who the select group is. Is it movie studios making 9 figure budgets, or is it any obsessed person who can figure out how to find and install software. Obviously, it's hard to imagine many situations like that, but you can imagine a process that required a 8-figure quantum supercomputer.
  - shostack 4y ago
    This is too myopic a view. Think about how far graphics have come on mobile devices let alone high end gaming machines in the past 5 years. This technology will eventually be accessible in the palm of your hand via a powerful enough device or the cloud. It's more a question of when than if.
    - deleted 4y ago
      [deleted]
oliver910 4y ago
I agree that raising awareness that tools like this are possible is important and that sufficiently advanced actors can do this anyway, however I don't think in this case releasing pre-trained weights to the general public is responsible. This could probably be used to help bypass crypto exchange KYC for moneylaundering purposes. I'm not sure what the best access model is - email us with a good reason to get access to the weights perhaps - but what alarms me is there seems to be no consideration for misuse or responsible release at all.
Karliss 4y ago
Even without deepfakes any kind of system relying on a person (or computer) not being tricked by webcam video seems quite questionable. People could still be tricked with a spliced video fragments of the real person or makeup especially if the set of face expressions used during "liveness check" is known ahead of time.
- wussboy 4y ago
  I try to imagine how society will deal with this. What if deep fakes are so perfect that anyone can generate real-time footage of anyone else doing anything? As a society we’d need to move out of the virtual and back into the physical. Would that be such a bad thing? I suspect this perfect deep fake technology might be a real boon to society.
  - haswell 4y ago
    I’ve often wondered if it’s possible to put modern tech “back in the box”. At first, this seems impossible. It permeates our daily lives in so many ways. Continued development towards a tech dystopia might be the only way. But I’m worried it will require the dystopia part before people will wake up and accept that moving back to the physical world might be required. The next few decades will certainly be interesting, at the very least.
light_hue_1 4y ago
That's like saying that nuclear weapons exist, and bad actors can potentially get them, so let's lower the bar so that anyone can. Making such tools accessible is reprehensible. It will lead to more bad actors, to less trust in media and in any objective reality, and more erosion of our institutions and society. There is absolutely no reason whatsoever for this. It's unethical and frankly downright evil.
- teakettle42 4y ago
  On par with nuclear weapons? Downright evil? You’re being absurd. The technology exists, and it’ll get better. Pretending that it doesn’t exist or banning it won’t make it go away — it’ll just be used by the least scrupulous and most powerful. Disruptive efforts like this are most upsetting to anxiety-ridden people who think that if they could just control things firmly enough, everyone and everything will be safe. That kind of thinking doesn’t actually work, though, and it produces a stiflingly rigid, oppressive society that deserves to be upset occasionally.
  - thingification 4y ago
    I think you describe the conventional wisdom well here. But: are those really the only options? Paranoid stop-the-world wishful thinking or all-new-trends-are-inevitable? I realise I'm almost certainly wildly misrepresenting your point of view with the latter label! I don't really mean to ascribe it to you. But I think the conversation often ends up tacitly as a debate between those two positions, both of which are mistakes. But one other option -- as an example -- is to find more complicated ways to stop doing <bad use of tech>. A striking historical example to me because it's so old it can hardly be said to have been an invention that needed to be responded to -- rather just an apparent invitable fact of life: For the longest time, it seems everyday brutal violence on a level we find hard to imagine was common. At the time, that would have been obviously inevitable. Nothing else had ever happened. Then it basically stopped -- turns out it never was inevitable. Why? We didn't for example ban knives as a technology, but we did a lot of things over the centuries that, to a good numerical approximation, stopped us using them for violence (banning them in certain situations is only one, I suspect not the most important and maybe not even necessary).
verisimi 4y ago
This technology certainly already exists and has probably been around for a long time. https://youtu.be/CpAdOi1Vo5s?t=3786 https://youtu.be/CpAdOi1Vo5s?t=3786
oldstrangers 4y ago
- sva_ 4y ago
avivo 4y ago
Ease of access does matter. It only buys time, but that can provide the time needed to create countermeasures and ideally make those very accessible—somewhat similar to responsible vulnerability disclosure. This piece goes into more detail: https://aviv.medium.com/the-path-to-deepfake-harm-da4effb541bd https://aviv.medium.com/the-path-to-deepfake-harm-da4effb541... (excerpt from a working paper, part of which was presented at NeurIPs).
roughly 4y ago
I'm reminded of Firesheep - https://en.wikipedia.org/wiki/Firesheep https://en.wikipedia.org/wiki/Firesheep - which came out in 2010. It wrapped session hijacking on WiFi in an easily usable interface. The technique and the vulnerability wasn't anything new, but the extension raised awareness in a big way and really sparked a big push for getting SSL deployed, enabled, and defaulted everywhere.
deleted 4y ago
[deleted]
gambler 4y ago
The entirety of deep fake technology was developed mostly in mainstream academia using "raising awareness" as an excuse. Paper after paper, model after model, repository after repository. Every single time the excuse was "if we don't do it, someone else will". This was going on for years and the explanation is absolutely laughable. Without countless human-hours put into this by academia, it's pretty obvious that this technology would be nowhere near its current state. Maybe some select military research agencies could develop something analogous. Currently this is accessible to literally every crook and prankster with internet access. Also, the notion that "raising awareness" is going to prevent deep fakes from being used in practice shows complete and utter disconnect from reality. Most people who are skeptical are already aware how imminently fakeable all the media really is. Most people who still are unaware will remain so, no matter how many GitHub repositories some dipshits will publish.

by sva_ 4y ago

belter 4y ago

You are right. I saw some guy that looked like Matt Damon trying to sell me some crypto coins...

karmakurtisaani 4y ago
It would be so ridiculously crazy that an established A-list actor would willingly promote those crypto scams. Must have been a deep fake!
HWR_14 4y ago
For what it's worth, actors seem to be securing themselves against this using IP rights.

bko 4y ago

I agree with everything you said, but I we shouldn't deny that opportunistic bad actors don't exist. Or it might get on their radar and be exploited. Open source tools also tend to be better maintained, documented and reliable, so the bad guys will have a better tool. That being said, bringing it to light also has benefits like you said. If the tool is out in the open and state of the art techniques are used, technology to detect its use will also benefit.

blagie 4y ago

As someone with a stalker, I can't emphasize this enough. A stalker will go to all sorts of lengths to do bizarre shit. People don't believe it. I would guess governments will do some equivalent there-of. Democratizing access to things -- including bad things -- has a preventative effect: 1) I can guard against things I know about 2) People take me seriously if something has been democratized The worst-case scenario is if my stalker got her hands on something like deep fake technology before the police / prosecutor / jury didn't knew it existed. I'd probably be in jail by now if something like that ever happened. She's tried to frame me twice before. Fortunately, they were transparent. She'll try again. Best case scenario is that no one has access to this stuff. Worst case scenario is only a select group have access, and most people don't know about it. Universal access is somewhere in between.

RobertRoberts 4y ago
I want to second this, as there are so many people that just simply can't believe how much time and energy some people will put into destroying someone else's life. And when you ask for help, people think you are the insane one because they simply can't believe your story about the insanity of someone else. I hope you find relief sometime from your stalker. I found it (not a stalker exactly) from letting the person burn themselves with their behavior so many times without me doing or saying anything in return to them (my strategy of non-direct conflict, and it worked for me), that eventually they ran out of people to manipulate and fool.
- Moru 4y ago
  A worse situation: What if you ARE insane, it's just that this time it's not your imagination running wild, it's actually someone stalking you? People that have some mental problem, whatever it is, have a much harder time since no one will believe them. It's always just "I know you are feeling bad right now..." even though the diagnose is ADHD or something else irrelevant.
HWR_14 4y ago
Your "worst case" depends greatly on who the select group is. Is it movie studios making 9 figure budgets, or is it any obsessed person who can figure out how to find and install software. Obviously, it's hard to imagine many situations like that, but you can imagine a process that required a 8-figure quantum supercomputer.
- shostack 4y ago
  This is too myopic a view. Think about how far graphics have come on mobile devices let alone high end gaming machines in the past 5 years. This technology will eventually be accessible in the palm of your hand via a powerful enough device or the cloud. It's more a question of when than if.
  - deleted 4y ago
    [deleted]

oliver910 4y ago

I agree that raising awareness that tools like this are possible is important and that sufficiently advanced actors can do this anyway, however I don't think in this case releasing pre-trained weights to the general public is responsible. This could probably be used to help bypass crypto exchange KYC for moneylaundering purposes. I'm not sure what the best access model is - email us with a good reason to get access to the weights perhaps - but what alarms me is there seems to be no consideration for misuse or responsible release at all.

Karliss 4y ago

Even without deepfakes any kind of system relying on a person (or computer) not being tricked by webcam video seems quite questionable. People could still be tricked with a spliced video fragments of the real person or makeup especially if the set of face expressions used during "liveness check" is known ahead of time.

wussboy 4y ago
I try to imagine how society will deal with this. What if deep fakes are so perfect that anyone can generate real-time footage of anyone else doing anything? As a society we’d need to move out of the virtual and back into the physical. Would that be such a bad thing? I suspect this perfect deep fake technology might be a real boon to society.
- haswell 4y ago
  I’ve often wondered if it’s possible to put modern tech “back in the box”. At first, this seems impossible. It permeates our daily lives in so many ways. Continued development towards a tech dystopia might be the only way. But I’m worried it will require the dystopia part before people will wake up and accept that moving back to the physical world might be required. The next few decades will certainly be interesting, at the very least.

light_hue_1 4y ago

That's like saying that nuclear weapons exist, and bad actors can potentially get them, so let's lower the bar so that anyone can. Making such tools accessible is reprehensible. It will lead to more bad actors, to less trust in media and in any objective reality, and more erosion of our institutions and society. There is absolutely no reason whatsoever for this. It's unethical and frankly downright evil.

teakettle42 4y ago
On par with nuclear weapons? Downright evil? You’re being absurd. The technology exists, and it’ll get better. Pretending that it doesn’t exist or banning it won’t make it go away — it’ll just be used by the least scrupulous and most powerful. Disruptive efforts like this are most upsetting to anxiety-ridden people who think that if they could just control things firmly enough, everyone and everything will be safe. That kind of thinking doesn’t actually work, though, and it produces a stiflingly rigid, oppressive society that deserves to be upset occasionally.
- thingification 4y ago
  I think you describe the conventional wisdom well here. But: are those really the only options? Paranoid stop-the-world wishful thinking or all-new-trends-are-inevitable? I realise I'm almost certainly wildly misrepresenting your point of view with the latter label! I don't really mean to ascribe it to you. But I think the conversation often ends up tacitly as a debate between those two positions, both of which are mistakes. But one other option -- as an example -- is to find more complicated ways to stop doing <bad use of tech>. A striking historical example to me because it's so old it can hardly be said to have been an invention that needed to be responded to -- rather just an apparent invitable fact of life: For the longest time, it seems everyday brutal violence on a level we find hard to imagine was common. At the time, that would have been obviously inevitable. Nothing else had ever happened. Then it basically stopped -- turns out it never was inevitable. Why? We didn't for example ban knives as a technology, but we did a lot of things over the centuries that, to a good numerical approximation, stopped us using them for violence (banning them in certain situations is only one, I suspect not the most important and maybe not even necessary).

verisimi 4y ago

This technology certainly already exists and has probably been around for a long time. https://youtu.be/CpAdOi1Vo5s?t=3786 https://youtu.be/CpAdOi1Vo5s?t=3786

oldstrangers 4y ago

sva_ 4y ago

avivo 4y ago

Ease of access does matter. It only buys time, but that can provide the time needed to create countermeasures and ideally make those very accessible—somewhat similar to responsible vulnerability disclosure. This piece goes into more detail: https://aviv.medium.com/the-path-to-deepfake-harm-da4effb541bd https://aviv.medium.com/the-path-to-deepfake-harm-da4effb541... (excerpt from a working paper, part of which was presented at NeurIPs).

roughly 4y ago

I'm reminded of Firesheep - https://en.wikipedia.org/wiki/Firesheep https://en.wikipedia.org/wiki/Firesheep - which came out in 2010. It wrapped session hijacking on WiFi in an easily usable interface. The technique and the vulnerability wasn't anything new, but the extension raised awareness in a big way and really sparked a big push for getting SSL deployed, enabled, and defaulted everywhere.

deleted 4y ago

[deleted]

gambler 4y ago

The entirety of deep fake technology was developed mostly in mainstream academia using "raising awareness" as an excuse. Paper after paper, model after model, repository after repository. Every single time the excuse was "if we don't do it, someone else will". This was going on for years and the explanation is absolutely laughable. Without countless human-hours put into this by academia, it's pretty obvious that this technology would be nowhere near its current state. Maybe some select military research agencies could develop something analogous. Currently this is accessible to literally every crook and prankster with internet access. Also, the notion that "raising awareness" is going to prevent deep fakes from being used in practice shows complete and utter disconnect from reality. Most people who are skeptical are already aware how imminently fakeable all the media really is. Most people who still are unaware will remain so, no matter how many GitHub repositories some dipshits will publish.