9 ms·
New AI Imaging Technique Reconstructs Photos with Realistic Results
- subcosmos 9y agoThey're at it a-GAN!
- merpnderp 9y agoGAN is just a brilliant concept. I look forward to all the new tools to come out of this.
- p1esk 9y agoThey didn't use GAN for this.
- gmiller123456 9y agoWhile I didn't read the paper in detail, there are a lot of papers in the references section about GANs. And referring back to the text that cites them makes it pretty obvious they did use GANs.
- deleted 9y ago[deleted]
- SlowRobotAhead 9y agoThe example with the eyes being removed works nicely on the attractive girl, it adds sexy eyes back in. Works less nicely on the old man, where it adds the same sexy eyes.
- subcosmos 9y agoIt's trained on celebrities, hence the bias :p
- azinman2 9y agoThe left eye (her right) doesn’t look realistic to me
- apendleton 9y agoIt looks weird when they only do one, but when they do both it becomes symmetrical again and looks less weird.
- glandium 9y agoNote it's not any old man, it's Ernest Borgnine, who played Dominic Santini in Airwolf.
- exhilaration 9y agoIn case you only read the HN comments, don't miss the amazing video from the article: https://www.youtube.com/watch?v=gg0F5JjKmhA https://www.youtube.com/watch?v=gg0F5JjKmhA
- acmecorps 9y agoI thought I was the only one. Came for the news, stayed for the comments.
- khuss 9y agoNice breakthrough. What could be some potential applications?
- jackweirdy 9y agoThe first thing that comes to mind is restoring the images that were recovered (damaged) at the end of this process (follow through to part 3 for full examples of images): http://www.bbc.co.uk/rd/blog/2017-12-morecambe-wise-video-film-archive-restoration http://www.bbc.co.uk/rd/blog/2017-12-morecambe-wise-video-fi... https://www.bbc.co.uk/rd/blog/2017-12-morecambe-wise-video-xray-microtomography https://www.bbc.co.uk/rd/blog/2017-12-morecambe-wise-video-x... http://www.bbc.co.uk/rd/blog/2017-12-morecambe-wise-film-recovery-processing-algorithm http://www.bbc.co.uk/rd/blog/2017-12-morecambe-wise-film-rec...
- deleted 9y ago[deleted]
- coolspot 9y agoDeep fakes
- halfjoking 9y agoHate watermarks? Remove them with this service and no one will know you got your image from 9gag...
- khuss 9y agoClever idea
- gabblegabble 9y agoDemosaic JAV!
- JoeCoder_ 9y agoHow long until I can use this as a Photoshop plugin?
- ttoinou 9y agoYears
- gefh 9y agoThere's been content aware fill for a long time: https://youtu.be/Ge9jsJZ3lA0?t=245 https://youtu.be/Ge9jsJZ3lA0?t=245 It's not the same backing tech but for practical purposes it's as good.
- ttoinou 9y agoCurrent inpainting algorithm is good for removing wires or adding textures. The new nVidia technique should be able to synthesize new information not in the original image
- JoeCoder_ 9y agoI doubt Adobe's content aware fill could replace someone's eye or a few of the other examples in the video.
- Jach 9y agoAbout -10 years ago (at least that's about when a gimp script doing a version of this was put out, Smart Remove / Heal Selection -- https://www.youtube.com/watch?v=3h1gZJsjKxs https://www.youtube.com/watch?v=3h1gZJsjKxs -- I recall Adobe demoing similar stuff at various points though I suppose this is the first time someone's used a generative neural net to fill things in rather than just nearby pixels)
- dnautics 9y agoThey keyframe in the YouTube link is creepy because when they obliterate the eye section there's maybe weak linkage? They don't talk to each other and fill in different eyes on each side.
- daveguy 9y agoKeyframe is only half creepy. The left eye was the one filled in by the algorithm. The right eye is filled in similarly a few seconds later. For human faces it seems to substitute very generic replacements of features (and take no more cue from the surrounding photo than: eye shaped thing should go here, chin shaped thing should go here, brown hair should go here, etc. The video is definitely worth watching. For human faces it seems to take less cue from surroundings than inanimate scenes. Although it probably just seems that way because we are so sensitive to peculiarities in images of the face. Edit: it's somewhere between a surrounding texture fill and a semantic / context based reconstruction. Texture fill would produce blank skin for an eye. Ideal reconstruction would take into account appropriate wrinkles, symmetry, expected bone structure. It works better for still life / scenes than for faces.
- potlee 9y agoCan someone who understands this tell us what is the catch here?
- taeric 9y agoThe catch is it is more like an artists interpretation. Anyone expecting that it will truly fix old damaged pictures to be like they were is going to be disappointed. Anyone just wanting to fill in some gaps will be excited. (Same for just procedural generation of some images, I suppose. Would be neat to see what it can do from a very low fidelity outline of a house/forest/etc.)
- taneq 9y agoAs opposed to it being truly magical, I guess?
- taeric 9y agoI think the expectation has to be set that this is great for creative use cases. Not so much for forensic style ones. To that end, "reconstruct" is not so much reconstructing what was lost, but more "fills in holes" of photos. If you read the title expecting something to literally be reconstructed, you are likely to feel lied to.
- johndough 9y agoYou'll probably hit memory limits if you go much beyond 512x512-sized holes. Additionally, computation times grow quickly with higher resolution and you already need a high end GPU for this resolution to get a reasonably interactive response time. You'll also need a favorably licensed pretrained model or a few 10000 training images and masks. So all in all, I can't see any deal breakers, but I'd probably still use PatchMatch instead.
- rhcom2 9y agoFor reference the GPU they're using for this paper is the NVIDIA V100 GPU, a datacenter GPU costing $8,000.
- devit 9y agoAny actual software? Seems amazing for face shot retouching.
- lepouet 9y agoAdobe often show things like that, maybe a little less advanced... This is nice with a small picture in a video, but for now you always end up disapointed when you try on high res pictures where quality is important...
- fudged71 9y agoHaving used photoshop content-aware-fill quite a bit, what was very interesting in the video with the eyes and the hairline was that it wasn't just filling in with what was already in the image, but also what it thought should be in the image. It's not just locally aware but also... world aware (if given the proper training data).
- state_less 9y agoThis is pretty cool. I'm looking forward to the day that not only can we reconstruct what was removed, but reconstruct what's not in the photo (image sensor), but could be inferred. Can we reconstruct the 3D scene and fill in the parts the camera can't see? For example, if it's a front view of a person, make a 3d model of a person and texture the backside clothing. It seems like if we put together all the advancements these deep inference engines are producing, we may be able to reconstruct a reality, and allow people to walk around in it using VR glasses, or less advanced 3D shooter style controls. It could be like a live earth view, any photo with a timestamp can contribute to this view and we'd create a sweet 4D model of earth across time and space.
- vincnetas 9y agoThis is not reconstruction. It's guessing from what 'i' have seen before. So this would never reconstruct anything unseen. In a broad sense.
- state_less 9y agoI think I follow what you're saying, we aren't going to be able to construct what aliens look like on a distant planet (or some other unknown). We're essentially looking at the information we have, which is why I expect continued interest in exploring for new information. Even as we continue to improve our reconstructions of 4D.
- nartz 9y agoPartially correct, but not necessarily true. Speaking extremely high level, the model is currently trained to fill in the blank space with something like the most probable options per pixels, based on the training set data. However, it is conceivable that it could be trained to also, say, insert an object into a scene, based on other characteristics found in the scene. Inotherwords, it could be trained to maximize a joint goal, where the second goal involved generating an object.
- robotresearcher 9y agoLike so: http://vision.princeton.edu/projects/2016/SSCNet/ http://vision.princeton.edu/projects/2016/SSCNet/ There's similar work at (at least) Berkeley and Stanford.
- dschuetz 9y agoWhenever Nvidia issues a research paper on how well their products work for AI applications like that in their paper I feel like instantly buying some of their more expensive scientific products.
- agumonkey 9y agopersonal opinion: I feel we're reaching the existential uncanny valley. computation can "do" everything and .. I feel dead inside thinking about it.
- ttflee 9y agoI feel that I would be waiting in the queue to buy a GPU to expand my own brain someday.
- agumonkey 9y agoI'm super not sold on this. I've learned so much without any help, even when crippled by illness. I believe the superhuman thing is just a marketing trick to pursue the tech growth.
- 0xffff2 9y agoComputers are clearly different and complementary to our own brains. We most definitely don't know how to replace our brains with better hardware, but having some kind of implant that would allow me to maintain a perfect calendar in my head instead of on my phone or computer would be really helpful, as would being able to do complex arithmetic a the speed of thought instead of the speed of typing equations into a calculator/Mathmatica/etc. It's not about replacing anything we already have with something better, it's adding new tightly integrated features.
- ashraymalhotra 9y agoIs the code for this open source?
- bob_theslob646 9y agoI'm guessing a whole line of business will be created from software like this existing? Some sort of validation engine which would verify if the photo is realistic or shopped. I understand that you can tell in some of the examples in the video, but as it gets better, it maybe really hard to distinguish real from fake.
- acdha 9y agoThis industry already exists (search for “photo forensics”) and I'm certain you're right to suspect it'll be booming soon. ML should also be good at faking some of the characteristics which current tools use so we must be looking at an arms race for years. One of the big challenges I'm expecting will be the categories of attack: it seems plausible that we'd be able to limit abuse in the case where the original image is made by someone who isn't malicious by making it easy for viewers to find the originals (perceptual hashing with some sort of distributed ledger or signature system, and getting major services like Facebook to use it) but I haven't seen a convincing suggestion for how to deal with the case where the original is created by the attacker and thus any validation system would only show what they submitted it with. It seems like that'd fall back on much more failure-prone techniques — e.g. you could rely on public information to convince most people that, for example, Barack Obama didn't pledge allegiance to ISIS at a public rally but most people aren't going to have enough rock-solid documentation to prove a negative. If an attacker said that politician X was having an affair in hotel room it'd probably seem convincing to many people unless they screwed up and left proof of e.g. using stock footage, landmarks from a different city, wrong time of year, etc.
- ashraymalhotra 9y agoHow would it work if the mask was covering the mouth region? Afaik teeth have always been difficult to reproduce...
- Viliam1234 9y agoHere is some prior art: https://en.wikipedia.org/wiki/Censorship_of_images_in_the_Soviet_Union#/media/File:Soviet_censorship_with_Stalin2.jpg https://en.wikipedia.org/wiki/Censorship_of_images_in_the_So...
- hogehoge 9y agoTry it http://blog-imgs-80.fc2.com/o/p/t/opticalillusion48/shinoda_03.png http://blog-imgs-80.fc2.com/o/p/t/opticalillusion48/shinoda_...