11 ms·
Nvidia Vid2vid: High-resolution photorealistic video-to-video translation
- gggggggg 8y agoAm i reading that right? Its making the videos that look real, from the simplistic input? If so, that is amazing. And if so, how do I turn a video I have into a simple/line version, to be able to then put a different 'skin' on it?
- dan-robertson 8y agoWith edge detection. Normally edge detection means looking for local sharp changes in brightness and marking them with a white spot. The edge detection used in this case looks more sophisticated to me. I don’t know how it works
- iamleppert 8y agoLooks just like edge detection + contouring from RGBD data.
- invalidusernam3 8y agoYou could try OpenCV Canny Edge Detection: https://docs.opencv.org/3.1.0/da/d22/tutorial_py_canny.html https://docs.opencv.org/3.1.0/da/d22/tutorial_py_canny.html Example with video here https://www.youtube.com/watch?v=1Ndxtb0q76c https://www.youtube.com/watch?v=1Ndxtb0q76c Obviously would need some tweaking but could be a good starting point
- Erlich_Bachman 8y agoThe level of realism can be gauged from the examples they provide right there on the page. Of course your results may vary basing on the initial bulk of data of realistic source images you use. You have the code right there on Github, just install it on some PC with powerful GPUs (or rent one), tune some parameters, train the network and you can do the same things.
- rasz 8y agoYes, also you might want to look up DLSS - they use pretrained upscaling network on GPU to generate 4K from native 1440p picture, instant ~20-50% performance bump with free AA. https://www.youtube.com/watch?v=MMbgvXde-YA https://www.youtube.com/watch?v=MMbgvXde-YA of course this being Nvidia they didnt implement it universally, you need to sign up for API access to black box gameworx like scam programs in order to implement it in your game.
- iforgotpassword 8y agoI'm probably being captain obvious here, but if this is what's being released for free, I wonder how much better a polished commercial version does, and when we reach the point where we can't trust anything we see anymore. It doesn't even have to be super perfect, even reaching the point where it takes experts about two weeks to determine if something's real or not might already be long enough to do great damage. From a technical standpoint I think this is very impressive, and I'm also interested in creative/artsy use of this. Their "replace trees by houses" example is pretty dull but gives a good glimpse at what can be done.
- TomMarius 8y agoI'm sure the point is already crossed. People on 4chan are making fake porn that is nearly indistinguishable - and they are complete amateurs.
- pbhjpbhj 8y agoAre you sure they're amateurs, not professionals don't it for fun (/pro boner/!?!).
- TomMarius 8y agoI meant something a little different. Any random dude with a random mid-level PC can download the software and produce amazing results without any special hardware or knowledge. I recommend you to try it out for yourself.
- TeMPOraL 8y agoWas there a huge progress made since news about deepfakes broke early this year? Around March/April this year, I actually did download the TensorFlow toolkit for face transfer that was used by /r/deepfakes people and tested it out (there were samples of photos of politicians included); the results were, at best, worse that I could do in 2 minutes in Gimp. Maybe they could get better if I had an expensive GPU farm at my disposal, but I'm pretty doubtful - given that the news died down pretty quickly, and no reasonable-quality faked pictures or videos were reported ever since.
- huling0 8y agoOne of the example translates a full human pose to a video of a dancer. If the network would be trained on the facial pose(?) / features only, would that recreate something like the facial reenactment in http://niessnerlab.org/projects/thies2016face.html http://niessnerlab.org/projects/thies2016face.html (source code for face2face is not public)?
- huling0 8y agoFrom the paper: "we have to use all the GPUs in DGX1 (8 V100 GPUs, each with 16GB memory) for training. We distribute the generator computation task to 4 GPUs and the discriminator computation task to the other 4 GPUs. Training takes ∼10 days for 2K resolution." As I don't have a DGX1 here, training the 2K resolution net for 10 days on a p3.16xlarge instance (also has 8 V100 GPUs) would cost USD 5875 on AWS. (USD24.48 per hour on-demand pricing * 24 hours/day * 10 days)
- marmaduke 8y agoTheir GitHub README says 24GB, and 12/16 GB requires cropping and performance not guaranteed. I’ve only seen P100 with 16 each, and its the big Quadros that have 24
- sp332 8y agoThe DXG-1 costs $129,000 so AWS is cheaper unless you need to do it 22 times. And you can have multiple instances running at once and get all of your results in ten days, instead of waiting ten days again for each run.
- gok 8y agoWell plus electricity. A DGX-1 takes 4 kilowatts or so that 10 day training run will take just about a megawatt hour or about $100 at retail. So the cross over point is more like 23 runs :)
- huling0 8y agoyeah, but does not look like something many hobbyists would try for fun ;-)
- p1esk 8y agoOr you can build your own 8x2080Ti rig which will have 80% of performance for 1/10th of the cost.
- twtw 8y agoDGX 1V has $7500 worth of CPU alone to feed the gpus. Throw in 8 TB of nvme ssd for training data and you're looking at something more like 1/5th the cost. V100 has ~50% higher memory bandwidth than 2080Ti, so you probably wouldn't get 80% of the performance. Also, only two 2080Ti can be connected via nvlink.
- rawoke083600 8y agoholodeck programming step 1.
- jokoon 8y agoI'm curious what would happen if somebody tried to impersonate the US president using this. He would say it's fake, but who would believe him? How exactly can computer scientists explain deepfakes to laymen?
- dan-robertson 8y agoWell I’m this case the results are pretty good locally but have pretty obvious artefacts too. Especially the synthesised road videos, look at the trees or even more at the Lane change in the linked video.
- puranjay 8y agoCan't you easily obfuscate that by making the video intentionally grainy and low-res and passing it off as "caught by CCTV" or "found footage"?
- IanCal 8y agoProbably much simpler to get a lookalike, and that's been possible for a long time.
- sbucky27 8y agoYou say 'pretty obvious artifacts' but that is because you have some clue about the process and know what you are looking for and are interested enough to look. I tried pointing out to some friends some really bad artifacts in a video we watched, and they just could not grasp it. They couldn't see what I was seeing as it didn't look out of place to them. It isn't for lack of intelligence, they just didn't care enough to understand. That pretty much describes vast swathes of the population. You show a video using the above technique to anyone with strongly held political/ideologic beliefs and an inclination to accept 'alternative facts' over actual facts and videos using these techniques will be like a wildfire and almost impossible to refute!
- MikusR 8y agoAn example of Obama. https://www.youtube.com/watch?v=cQ54GDm1eL0 https://www.youtube.com/watch?v=cQ54GDm1eL0
- SeriousM 8y agoI'm scared. I can't trust anything I haven't took myself. The problem is that other people not even know that this technology exists and if you tell them about it, you're a lier.
- mcintyre1994 8y agoI don't think there's going to be a long period where this technology is being used and isn't widely known about. It'll be used extremely quickly to abuse people using their Facebook pictures, and once there's a Facebook angle the media will be able to run with it and everyone will get it.
- posterboy 8y agoIt's already being used for quite a while and you talk as if you aren't aware of the fact. Quite Ironic.
- Findeton 8y agoSome applications for this kind of tech: - Porn, yeah, first application you can think of, there are already some startups doing it. - Doubling actors, and applied to sound, maybe you could translate from one language to another but kind of keeping the accent and tone. - Propaganda and misinformation. Now you can get your enemy to say and do whatever you want, on video. - Photo-realistic games. Create a rough 3D model of an scenario and train the AI for it. Instead of photo-realistic rendering with math, render it with the AI based on a rough render, in real-time.
- magnat 8y ago> Photo-realistic games. Create a rough 3D model of an scenario and train the AI for it. Instead of photo-realistic rendering with math, render it with the AI based on a rough render, in real-time. According to last month's nvidia rtx presentation/launch event [1], they are going to do something similar quite soon. Games will ship with DNN pre-trained offline on extremely high quality renderings. Game itself renders at lower resolution (limited by performance needed for proper raytracing) and uses DNN to upscale it. [1] https://youtu.be/Mrixi27G9yM?t=51m15s https://youtu.be/Mrixi27G9yM?t=51m15s
- antpls 8y agoI wonder, since the NN cores of the GPU are used for real-time raytracing, will they be able to run custom NNs possibly not related to visual stuff in parallel to the ray-tracing stuff? Edit : found the answer on Internet, apparently the RT (raytracing) cores are different and separated from the Tensor (NN) cores on the RTX
- 21 8y agoI think this is about trading storage for computation - you replace terabytes of model/texture data with a compute heavy NN.
- kmlx 8y agoany ideas about porn startups that work on anything related to this? i know pornhub and their network banned deepfakes and there’s also some work being done on detecting deepfakes.
- pjeide 8y agoWhat media would someone collect now to be used in the future to reproduce the likeness of loved ones? Video clips of them moving? Talking? Different poses of pictures? Reading the dictionary out loud to get vocal patterns? Heck with impersonating the POTUS. What about a lost friend, sibling or parent?
- pbhjpbhj 8y agoBlack Mirror did this I think - if we can make video, then why not VR (down the line if processing catches up), Second Life iterated. Heaven on Earth?
- crtasm 8y agoAlso see the film Marjorie Prime. https://en.m.wikipedia.org/wiki/Marjorie_Prime https://en.m.wikipedia.org/wiki/Marjorie_Prime
- atrilumen 8y agoWouldn't that be some kind of horrific emotional uncanny valley?
- sp332 8y agoYeah. https://www.youtube.com/watch?v=fkE6RBlfbXA https://www.youtube.com/watch?v=fkE6RBlfbXA This isn't the worst one, there was a time when PKD's daughter interviewed the head and he went off on a rant about how much he disliked his family. That was really rough.
- Grasshoppeh 8y agoIt would be interesting to see the applications of this technology with the use of thermal cameras. Extracting environments from thermal imaging would be nice.
- ivanb 8y agoThis is why some people believe that Assange had been dead for more than a year.
- dplgk 8y agoIs there a way that a person could create a QR code made from one-way private key? The QR code would contain date and time and other Metadata that is decrypted with a public key that proved the QR code in the video was made by, for example, the person in the video? This would "prove" the video was real? I guess the speech would still be manipulated. So the whole transcript would have to be encrypted in this way...
- lucb1e 8y agoI don't know why you're talking about QR codes and transcripting/encrypting the audio when you can just sign the file? gpg --sign video.mp4
- misnome 8y agoI assume they are talking about signing/watermarking a video in a way that survives video encoding/lossy transmission. A QR code probably wouldn’t work for various reasons but is an easy mental analogy to think of.
- lucb1e 8y agoFor embedding, you should just put it in the metadata. Encoding it in the video itself... I don't really see the point.
- ai_ja_nai 8y agoouch, this hurts
- aethr 8y agoSeeing the example of one facial pose video transcribed to three different looking women, I'm imagining a future where Netflix does a/b testing on its shows, using similar tech to swap out different "actors" to find which one resonates with audiences best. They could even generate a new "cast" for each market, after only shooting the show once.
- iamgopal 8y agoPorn industry will benefit the most.
- Improvotter 8y agoAre there any high resolution examples available? Am I just not finding them in the README?
- lambdadmitry 8y agoI am a bit surprised how shallow the comments on this one are. Look closely, while it does generate videos of a passing similarity, they aren't "photorealistic" in the slightest. They are good locally across time and space domains, but globally they are as far from realism as Doom 2 was. The only explanation for the attitude I see in this submission is that most IT people trained themselves to spot CGI by looking at local artifacts, assuming that global artifacts won't happen because stuff on the scene is reasonable. There is no "stuff on the scene" with those videos, it's just mindless vector manipulation with no underlying world model. Cars wave around, trees grow a feet from each other and behave in a way incompatible with 3D perspective. Relax, it'll require at least another AI/ML revolution (or even several) to achieve photorealism.
- tvdo 8y agoDoes anybody know a) the performance (e.g. introduced latency) and processor requirements for the client/input (e.g. is real time canny edge detection good enough and how fast would that run)? and b) what the latency impact on the NN side to build the images (e.g. how many ms are we talking about?) Thank a lot!
- gok 8y agoKind of surprised this stuff hasn't totally blown away traditional video compression techniques yet.
- make3 8y agoI suspect that a combination of the two is where it's at, ie, store a lossy classical compressed version, then remove the artifacts/dream up details with deep learning
- gukov 8y agoThis got me thinking: in future we will probably stop streaming video and instead just send simple vectors. The data then gets rendered in real time into actual video. The customers will even be able to customize the movie: pick their favorite actors, the environment, etc.
- ksec 8y agoAs someone not into AI / Machine Learning. Can we expect CUDA to be the x86 on PC and Servers? Literally all works are defaulting to CUDA and Nvidia's library. I don't even see a contender trying compete. I don't even see AMD's ROCm being used or even mentioned anywhere.
- dolzenko 8y agoCant wait someone to put Simpsons through this
- jcims 8y agoIf you have any doubts that the face synthesis one is faked (faked fake?), watch the face of the woman in the bottom left as it loops. https://github.com/NVIDIA/vid2vid/blob/master/imgs/face.gif https://github.com/NVIDIA/vid2vid/blob/master/imgs/face.gif
- bronxbomber92 8y agoI'm not following - can you explain?
- jcims 8y agoHer(?) face morphs slightly in the first few frames.