8 ms·

I'm probably being captain obvious here, but if this is what's being released for free, I wonder how much better a polished commercial version does, and when we

I'm probably being captain obvious here, but if this is what's being released for free, I wonder how much better a polished commercial version does, and when we reach the point where we can't trust anything we see anymore. It doesn't even have to be super perfect, even reaching the point where it takes experts about two weeks to determine if something's real or not might already be long enough to do great damage. From a technical standpoint I think this is very impressive, and I'm also interested in creative/artsy use of this. Their "replace trees by houses" example is pretty dull but gives a good glimpse at what can be done.

TomMarius 8y ago
I'm sure the point is already crossed. People on 4chan are making fake porn that is nearly indistinguishable - and they are complete amateurs.
- pbhjpbhj 8y ago
  Are you sure they're amateurs, not professionals don't it for fun (/pro boner/!?!).
  - TomMarius 8y ago
    I meant something a little different. Any random dude with a random mid-level PC can download the software and produce amazing results without any special hardware or knowledge. I recommend you to try it out for yourself.
    - TeMPOraL 8y ago
      Was there a huge progress made since news about deepfakes broke early this year? Around March/April this year, I actually did download the TensorFlow toolkit for face transfer that was used by /r/deepfakes people and tested it out (there were samples of photos of politicians included); the results were, at best, worse that I could do in 2 minutes in Gimp. Maybe they could get better if I had an expensive GPU farm at my disposal, but I'm pretty doubtful - given that the news died down pretty quickly, and no reasonable-quality faked pictures or videos were reported ever since.
      - TomMarius 8y ago
        It's more about experimenting with your training data and other configuration. What I've heard, getting great results takes time - but it's possible. Most of the focus of the community is on porn, so I can imagine not so many journalists are checking the newest results and reporting about them.
albertgoeswoof 8y ago
If you compare OSS or free software to commercial software generally, I don’t think there are that many massive gaps. It’s mostly polish and small incremental improvements, but the underlying tech is mostly the same. Why would that be different in this case?
- akiselev 8y ago
  The gaps are massive in several electrical/silicon CAD verticals and simulation. Due to IP secrecy, no OSS Verilog/VHDL synthesis alternative exists for Vivado/ISE/Quartus and the Intel/Xilinx line of FPGAs. I don't think any practical alternatives exist for Mentor Graphics' line of silicon design or simulation software, nor have I seen any OS software capable of complex mixed domain simulations like COMSOL or Ansys - many of the pieces exist, but it takes a lot of work to verify that algorithmic physical models actually work together.
- have_faith 8y ago
  Adobe have definitely bested most oss competitors in their space. Although with the amount of man power at their disposal it would be hard to beat.
  - sjwright 8y ago
    The vast bulk of Adobe's advantage is in UX, not technical algorithms. Which makes perfect sense because that tends to be the case with most F/OSS software—technically brilliant but with an face only a programmer could love. Yes, Adobe do have some remarkable algorithms that would be difficult to replicate (e.g. heal brush and content aware fill) but these are a small minority of Adobe's software advantage. The one that irritates me the most is vector drawing programs: open source programs (and even paid competitors) just can't touch Adobe Illustrator for the sort of work I do. I'm sure at least 50 percent of it is familiarity and muscle memory, but I've desperately tried switching to a few different options like Inkscape or Affinity and left wildly disappointed.
    - homarp 8y ago
satori99 8y ago
It could also lead to crimes like Blackmail becoming extinct. It would be hard to hold incriminating recordings of anyone over them if near-perfect audio and video synthesis was common. Especially for public figures with lots training data available.
- iforgotpassword 8y ago
  That's equally horrifying in a different way.
  - garmaine 8y ago
    Also, map images pulled from facebook to the bodies of pornstars. The creepiness and invasion of ... personal image(?) this enables is horrifying.
    - cube2222 8y ago
      That's already happened. Search for deep fakes. We already have face substitution in videos which is working surprisingly well sometimes. There's been r/deepfakes where around 30% of the content was porn with swapped faces. It was banned though.
      - deleted 8y ago
        [deleted]
      - thanatropism 8y ago
        It was pretty limited though. For split seconds videos seemed real but the seams soon enough showed. Fake celebrity porn has been on the internet since 1996 at the very least. It's always been crummy; but porn in general requires a thick suspension of disbelief and an intense focus in a partial object of desire (what Lacan calls the objet petit a) that blurs everything else.
- flurdy 8y ago
  Maybe but only after extensive proliferation. Initially, every Tom, Dick & Harry script kiddie will try to blackmail people whilst the public and justice system is not really aware of the technology possible. When eventually enough of the public is made aware to not trust photos and videos anymore then these types of blackmail attempts would be less effective. But there will still be less informed gullible targets.
tnolet 8y ago
Isn’t this something a lookalike and some good make up could already do for ages?
- dgzl 8y ago
  That would require an entire team's worth of skill, preparation and work. This could be done with a few already existing medias, and a single person behind a computer.
davedx 8y ago
Yeah, the Tweet I found this from had a similar sentiment: https://twitter.com/PiratePartyINT/status/1042964668078116864 https://twitter.com/PiratePartyINT/status/104296466807811686... "Starting now, we cannot trust video or audio evidence. The ramifications for our legal & political systems will not be known for many years"
- bsenftner 8y ago
  "Starting now"? How's about 5 to 10 years ago. This is being released free now, which indicates to me this is now disposable tech and the authors have much, much better in their labs.
  - currymj 8y ago
    Probably not; the GAN technique wasn't even published until 2014, and this is based on other work that's been done even more recently. The field moves fast and is more open than you might expect (because it's still so academic).
  - p1esk 8y ago
    Usually it's the other way around with ML researchers - what they show is better than what they have in the lab :)
  - andybak 8y ago
    Not sure it works this way except in extremely specific areas of research (and I'm not even sure about that). Publicly available research is usually the cutting edge in most areas of knowledge.
dgzl 8y ago
Honestly, you just need to convince one person to execute a successful social engineering attack.
yonkshi 8y ago
AI can also be used to identify these fake videos, and they are probably better than human identification. There will be a rise of AI forensics. The type of neural network they use (GAN) works by having two networks battling each other, one tries to generate fake, and another (discriminator) tries to identify fake, it's a constant arms race. As the generator gets better, the discriminator also gets better. Which means, if the fake video is this good, there must be a discriminator network that identifies fakes just as good. We did a similar project using GAN, generating images from a text description. You can see the progression of generator and discriminator battling each other, and both get better with time. https://github.com/yonkshi/text2imageNet https://github.com/yonkshi/text2imageNet
- nickbecker 8y ago
  This is how their model was trained, but I think what you've said may not quite be the case. Because the discriminator (D) and generator (G) usually compete in a minimax game, the equilibrium probability of D correctly classifying an image as fake tends to 1/2 (ignoring distributional factors). If the competing networks have enough capacity and can be stably trained, then in theory they will reach equilibrium as the data distribution from G converges to the actual data distribution. If this is the case, then the discriminator correctly identifies fake videos with a probability of 1/2. They may not reach equilibrium (making D > 0.5), but it's not clear that the discriminator itself is a panacea for identifying fake videos/images.
- lambdadmitry 8y ago
  The whole GAN thing is useless though if neither of the models have a concept of symmetry or that those pixels do represent actual solid objects in space. Modern neural nets can't represent abstract spatial and semantical concepts or reason, so that videos are full of glaring perspective inconsistencies and large-scale artifacts. So nope, neither AI can help to "identify fake videos", nor do we need AI for that.
- jcims 8y ago
  The problem is that it's a game of superhuman cat and mouse. We'll have systems arguing with each other and no way to tell which is correct. If, for example, someone is able to get a copy of the forensic AI model, they can train their decoder/generator to work against it until the results pass as legitimate. With no human ability to argue with the results of the forensics AI, we'll just trust it and pass it off as truth. If you can clone the jury and conduct a quadrillion private trials, your chance of success in court is going to increase substantially. Things gonna get creepy.
malmsteen 8y ago
Yeah but it was already the case with CGI. IT's true that it's going to be easier and easier to do fake porn, fake speeches, fake voice recording, fake vides....
colordrops 8y ago
We reached that point of not being able to trust any media a while ago. It's just getting easier and more widely distributed.
jorgemf 8y ago
Probably Nvidia is not interesting in this type of tools and this is more like "hey, look what you can do with a lot of Nvidia cards, buy us a lot of them and you will do this and much more"

TomMarius 8y ago

I'm sure the point is already crossed. People on 4chan are making fake porn that is nearly indistinguishable - and they are complete amateurs.

pbhjpbhj 8y ago
Are you sure they're amateurs, not professionals don't it for fun (/pro boner/!?!).
- TomMarius 8y ago
  I meant something a little different. Any random dude with a random mid-level PC can download the software and produce amazing results without any special hardware or knowledge. I recommend you to try it out for yourself.
  - TeMPOraL 8y ago
    Was there a huge progress made since news about deepfakes broke early this year? Around March/April this year, I actually did download the TensorFlow toolkit for face transfer that was used by /r/deepfakes people and tested it out (there were samples of photos of politicians included); the results were, at best, worse that I could do in 2 minutes in Gimp. Maybe they could get better if I had an expensive GPU farm at my disposal, but I'm pretty doubtful - given that the news died down pretty quickly, and no reasonable-quality faked pictures or videos were reported ever since.
    - TomMarius 8y ago
      It's more about experimenting with your training data and other configuration. What I've heard, getting great results takes time - but it's possible. Most of the focus of the community is on porn, so I can imagine not so many journalists are checking the newest results and reporting about them.

albertgoeswoof 8y ago

If you compare OSS or free software to commercial software generally, I don’t think there are that many massive gaps. It’s mostly polish and small incremental improvements, but the underlying tech is mostly the same. Why would that be different in this case?

akiselev 8y ago
The gaps are massive in several electrical/silicon CAD verticals and simulation. Due to IP secrecy, no OSS Verilog/VHDL synthesis alternative exists for Vivado/ISE/Quartus and the Intel/Xilinx line of FPGAs. I don't think any practical alternatives exist for Mentor Graphics' line of silicon design or simulation software, nor have I seen any OS software capable of complex mixed domain simulations like COMSOL or Ansys - many of the pieces exist, but it takes a lot of work to verify that algorithmic physical models actually work together.
have_faith 8y ago
Adobe have definitely bested most oss competitors in their space. Although with the amount of man power at their disposal it would be hard to beat.
- sjwright 8y ago
  The vast bulk of Adobe's advantage is in UX, not technical algorithms. Which makes perfect sense because that tends to be the case with most F/OSS software—technically brilliant but with an face only a programmer could love. Yes, Adobe do have some remarkable algorithms that would be difficult to replicate (e.g. heal brush and content aware fill) but these are a small minority of Adobe's software advantage. The one that irritates me the most is vector drawing programs: open source programs (and even paid competitors) just can't touch Adobe Illustrator for the sort of work I do. I'm sure at least 50 percent of it is familiarity and muscle memory, but I've desperately tried switching to a few different options like Inkscape or Affinity and left wildly disappointed.
  - homarp 8y ago

satori99 8y ago

It could also lead to crimes like Blackmail becoming extinct. It would be hard to hold incriminating recordings of anyone over them if near-perfect audio and video synthesis was common. Especially for public figures with lots training data available.

iforgotpassword 8y ago
That's equally horrifying in a different way.
- garmaine 8y ago
  Also, map images pulled from facebook to the bodies of pornstars. The creepiness and invasion of ... personal image(?) this enables is horrifying.
  - cube2222 8y ago
    That's already happened. Search for deep fakes. We already have face substitution in videos which is working surprisingly well sometimes. There's been r/deepfakes where around 30% of the content was porn with swapped faces. It was banned though.
    - deleted 8y ago
      [deleted]
    - thanatropism 8y ago
      It was pretty limited though. For split seconds videos seemed real but the seams soon enough showed. Fake celebrity porn has been on the internet since 1996 at the very least. It's always been crummy; but porn in general requires a thick suspension of disbelief and an intense focus in a partial object of desire (what Lacan calls the objet petit a) that blurs everything else.
flurdy 8y ago
Maybe but only after extensive proliferation. Initially, every Tom, Dick & Harry script kiddie will try to blackmail people whilst the public and justice system is not really aware of the technology possible. When eventually enough of the public is made aware to not trust photos and videos anymore then these types of blackmail attempts would be less effective. But there will still be less informed gullible targets.

tnolet 8y ago

Isn’t this something a lookalike and some good make up could already do for ages?

dgzl 8y ago
That would require an entire team's worth of skill, preparation and work. This could be done with a few already existing medias, and a single person behind a computer.

davedx 8y ago

Yeah, the Tweet I found this from had a similar sentiment: https://twitter.com/PiratePartyINT/status/1042964668078116864 https://twitter.com/PiratePartyINT/status/104296466807811686... "Starting now, we cannot trust video or audio evidence. The ramifications for our legal & political systems will not be known for many years"

bsenftner 8y ago
"Starting now"? How's about 5 to 10 years ago. This is being released free now, which indicates to me this is now disposable tech and the authors have much, much better in their labs.
- currymj 8y ago
  Probably not; the GAN technique wasn't even published until 2014, and this is based on other work that's been done even more recently. The field moves fast and is more open than you might expect (because it's still so academic).
- p1esk 8y ago
  Usually it's the other way around with ML researchers - what they show is better than what they have in the lab :)
- andybak 8y ago
  Not sure it works this way except in extremely specific areas of research (and I'm not even sure about that). Publicly available research is usually the cutting edge in most areas of knowledge.

dgzl 8y ago

Honestly, you just need to convince one person to execute a successful social engineering attack.

yonkshi 8y ago

AI can also be used to identify these fake videos, and they are probably better than human identification. There will be a rise of AI forensics. The type of neural network they use (GAN) works by having two networks battling each other, one tries to generate fake, and another (discriminator) tries to identify fake, it's a constant arms race. As the generator gets better, the discriminator also gets better. Which means, if the fake video is this good, there must be a discriminator network that identifies fakes just as good. We did a similar project using GAN, generating images from a text description. You can see the progression of generator and discriminator battling each other, and both get better with time. https://github.com/yonkshi/text2imageNet https://github.com/yonkshi/text2imageNet

nickbecker 8y ago
This is how their model was trained, but I think what you've said may not quite be the case. Because the discriminator (D) and generator (G) usually compete in a minimax game, the equilibrium probability of D correctly classifying an image as fake tends to 1/2 (ignoring distributional factors). If the competing networks have enough capacity and can be stably trained, then in theory they will reach equilibrium as the data distribution from G converges to the actual data distribution. If this is the case, then the discriminator correctly identifies fake videos with a probability of 1/2. They may not reach equilibrium (making D > 0.5), but it's not clear that the discriminator itself is a panacea for identifying fake videos/images.
lambdadmitry 8y ago
The whole GAN thing is useless though if neither of the models have a concept of symmetry or that those pixels do represent actual solid objects in space. Modern neural nets can't represent abstract spatial and semantical concepts or reason, so that videos are full of glaring perspective inconsistencies and large-scale artifacts. So nope, neither AI can help to "identify fake videos", nor do we need AI for that.
jcims 8y ago
The problem is that it's a game of superhuman cat and mouse. We'll have systems arguing with each other and no way to tell which is correct. If, for example, someone is able to get a copy of the forensic AI model, they can train their decoder/generator to work against it until the results pass as legitimate. With no human ability to argue with the results of the forensics AI, we'll just trust it and pass it off as truth. If you can clone the jury and conduct a quadrillion private trials, your chance of success in court is going to increase substantially. Things gonna get creepy.

malmsteen 8y ago

Yeah but it was already the case with CGI. IT's true that it's going to be easier and easier to do fake porn, fake speeches, fake voice recording, fake vides....

colordrops 8y ago

We reached that point of not being able to trust any media a while ago. It's just getting easier and more widely distributed.

jorgemf 8y ago

Probably Nvidia is not interesting in this type of tools and this is more like "hey, look what you can do with a lot of Nvidia cards, buy us a lot of them and you will do this and much more"