7 ms·
3D photographic inpainting from a single source image
- anigbrowl 7y agoOutstanding work. I've been really impressed with the quality of recent colorization, perspective, and texture reconstruction tools and think they do a wonderful job of bringing historical or degraded images 'back to life.' I wonder if we are headed for a future where many of these tools reside client-side and image/video transmission and storage can be reduced to a lightweight stream of vector data.
- frfl 7y agoAnyone with ML development experience know what changes are needed to make this work on a CPU without a CUDA GPU? Seems heavily coupled to CUDA.
- rayuela 7y agoYou could translate this to a non-CUDA GPU, such as a mobile GPU, but even that would require a bit of effort to be able to condense it such that is wasn't a total lag fest. Executing this on CPU seems damn near impossible from a usability standpoint given the large matrix multiplication involved. You really need the parallel capabilities of a GPU.
- fxtentacle 7y agoPort it from Torch to TensorFlow framework, export the graph to tflite, compile with xla&aot and you end up with a c++ library running on CP.
- nestorD 7y agoIt relies on torch and openCV: - I have never tried running openCV explicitely on CPU but I believe it is doable. - It is trivial to run torch on CPU instead of GPU (just comment the line that sends the code to the GPU).
- johndough 7y agoHere is a quick hack to make it work on CPU: https://github.com/983/3d-photo-inpainting https://github.com/983/3d-photo-inpainting Maxed out at 4 GB RAM for 256x144 images for me. PyTorch CPU installation instructions: https://pytorch.org/get-started/previous-versions/ https://pytorch.org/get-started/previous-versions/ OpenCV should work without CUDA. If not, build from source and consider `WITH_CUDA` flag.
- muglug 7y agoVery cool work, but it's a little jarring to see photos of segregation, war and famine being used to show off the algorithm (https://filebox.ece.vt.edu/~jbhuang/project/3DPhoto/3DPhoto_Legacy.mp4 https://filebox.ece.vt.edu/~jbhuang/project/3DPhoto/3DPhoto_...)
- Robotbeat 7y agoI'm not sure I agree. It seemed moving to me to see these historical pictures come alive. It did not seem in bad taste to me.
- craftinator 7y agoPlease do not allow me to see that which I do not want to see.
- sp332 7y agoIt's weird to use negative content in a promotional context. I agree with Robotbeat but it's still fair to say that it's "jarring".
- jshevek 7y agoAre you trying to paraphrase the parent? I disagree with the person you are responding to, but I feel you severely misrepresent what they actually said.
- robbrown451 7y agoI don't see the problem with it. I think their point is that they are important historical photos, and seeing them this way puts some new life in them and can enhance their emotional resonance. It's not like they come off as "let's remember the good ol' days with the separate water fountains for the colored folks." Not to me anyway.
- freepor 7y agoIf they didn’t do stuff like this then the only purpose of these efforts would be to make your cat pics 0.1% more monetizable for Mark Zuckerberg. It’s the application to historical photos like this that is perhaps the most important result of this work.
- Robotbeat 7y agoTheir example images are still 2D, but potentially very interesting for making 360 degree videos much more immersive in virtual reality. (May be some extra steps to ensure consistent rendering among adjacent frames?).
- peteforde 7y agoI wonder if this can be tweaked to generate the 45-perspective quilt files required to feed this into a Looking Glass. https://lookingglassfactory.com/ https://lookingglassfactory.com/
- mrfusion 7y agoWow how does that work?
- modeless 7y agoIt's the same principle as those plastic toys with printed 3D images that you've surely seen: https://www.youtube.com/watch?v=jIfAi_zJ2F4 https://www.youtube.com/watch?v=jIfAi_zJ2F4
- peteforde 7y agoYeah, just like that. Except that there's 45 distinct planes and it directly interfaces with Unity, Unreal and ThreeJS in minutes. Sorry if I'm reading too deeply into your note. There are just so many haters. Meanwhile, I backed these guys on Kickstarter, have had a unit on my desk for 18 months and think it's one of the most incredible things I've ever had to experiment with... and it cost me under $500.
- tsukurimashou 7y agoI agree his comment is somewhat diminishing for the product but he is not wrong saying it's the same principle.
- peteforde 7y agoI think we can safely agree that the video they linked to was a solid tip-off that snark was dialed to 11.
- 7y ago
- akdor1154 7y agoIs this what is already implemented on Facebook's newsfeed for certain photos?
- viggity 7y agoI can't speak for all photos - but I'm fairly certain those 3d looking photos on FB must be taken with with a stereoscopic camera (2+ lenses). They can calculate depth when they know the distance between the lenses (and differences in focal lengths, etc). That isn't to say they couldn't do it retroactively for single lens photos, but I'm guessing not right now they're not.
- nestorD 7y agoNot at the moment. They even compare with facebook's algorithm (the proposed new algorithm seem to do much better thanks to inpainting).
- wokwokwok 7y agoTake a moment and have a deep breath before you get excited about full 3d reconstruction with a single image. That isn't what this is. Watch the video in the bottom right corner, entitled "Comparison with State of the art". Now go and rewatch the examples and actually look at the edges of the objects as they move the camera 'a bit'. You'll see a tonne of artifacting. Less, clearly, than the existing state of the art, so I tip my hat to the efforts here. ...but all this is doing is generating an 'empty gap' generated by perspective and then basically using the equivalent of photoshop's content aware fill to fill that gap with plausible pixels. Since humans don't really pay much attention to edge details, it's quite plausible. To quote the paper: > In this work we present a new learning-based method that generates a 3D photo from an RGB-D input. Ie. This work is taking a depth image as the input and working on that to generate a 3d photo, rebuilding the full content of each 2d layer in the image. Ie. The output is not a 3d model, it is a series of 2D images at depth intervals, where the occluded content in each layer is in-painted (ie. generated artifically). (NB. The 'from a single source image' work used here is not novel; they're just using existing approaches to estimate a depth image)
- saagarjha 7y ago> all It might be "all" that it's doing, but it does it quite well and in a way that is quite believable, which is significantly better than what came before, which makes it almost realistic. That's what you said, but the way you said it felt like it lessened the achievement.
- wokwokwok 7y ago> which is significantly better than what came before It's a bit better. That's the point I'm making; it's just incremental improvement on existing process. Read the actual paper, eg. under 'Quantitative comparison'. If you think I'm belittling the effort I'm sorry, that's not my intention; ...but, for example, the other comments talking about using it to generate a full 3d model to display on a looking glass surface, or in VR displays a total lack of understanding of what has been achieved here.
- echelon 7y agoJust take a look at the historical photos results! That's one of the coolest things I've seen in well over a year. I've come across most of these photos before in textbooks and on the web, but this demo makes all of those historical figures and moments in time feel as real as if they were happening in the world today. It's much more pronounced than colorizing black and white photos. I feel connected. I'm convinced that the folks that present at SIGGRAPH are capable of nothing short of black magic wizardry. I can understand the technology being used, but my visual cortex only sees the impossible becoming real. This is truly deserving of the word awesome, because I am in awe.
- fxtentacle 7y agoProblem is, this paper won't be able to produce those results from historical photos. I find them quite misleading, to be honest. You'll need to either use a separate 3D depth estimation AI, or (more likely) have someone do a manual stereoscopic 3D conversion of your historical image. Only then (when you have depth data) can the algorithm presented in this paper start its work.
- johndough 7y ago> You'll need to either use a separate 3D depth estimation AI They seem to be using MiDaS https://github.com/intel-isl/MiDaS https://github.com/intel-isl/MiDaS for depth estimation which does a reasonable job on some random image pulled from pixabay https://i.imgur.com/IfbeaqY.jpg https://i.imgur.com/IfbeaqY.jpg
- fxtentacle 7y agoThat looks good, indeed :)
- fxtentacle 7y agoIn the meantime, I tried our Midas on other images and it failed most of the time. Really only works for those stock-is pictures with clear foreground and background and I believe it mainly just detects bokeh blur as opposed to actually understanding the scene.
- felixyz 7y agoThe experience of Mark Twaine's shoes was worth it for me. So dapper.
- ladberg 7y agoIf you like this then you'll love neural radiance fields: http://www.matthewtancik.com/nerf http://www.matthewtancik.com/nerf
- echelon 7y agoThank you for sharing! This is really cool tech.
- IshKebab 7y agoVery cool! Not quite the same because they use 20-50 input images and this uses 1. But it looks like it might be really cool for photogrammetry. The traditional photogrammetry software sucks. Very cool application using Matterport I saw recently is to scan AirB&Bs, e.g. https://breconretreat.co.uk/accommodation/swn-y-nant/floor-plan/#content https://breconretreat.co.uk/accommodation/swn-y-nant/floor-p...
- tmilard 7y agoVery very impressive 3D reconstruction. Apparently It can even reconstruct very well ropes, fine details. That's amazing ! The best system I have ever seen this year on reconstruction... Beavo
- petargyurov 7y agoThis is blowing my mind. Incredible.
- signaru 7y agoNow we just need an automated image stabilizer for those shaky videos... Just kidding. This is awesome! Just imagine the possibilities with enough computing power.
- fxtentacle 7y agoThe main issue that I see with this work is that they require RGBD data, meaning you have to do a lidar scan or something similar to measure the depth map. Alternatively, you could pay someone to draw it by hand, but that takes a long time. So basically, if you have impossible to get input data, this network can do its magic. What it then does is hallucinary inpainting, so something like the Photoshop content aware fill. If there's a tree in your photo, this one will make up a fake background behind it, so that you could move or remove the tree without things looking weird.
- yorwba 7y ago> So basically, if you have impossible to get input data, this network can do its magic. Except they evidently got the input data for the examples in the paper, so it can't be impossible to get. They cite at least two different methods for adding depth information to a single image to generate the necessary RGBD data, different views of which can then be rendered with their inpainting applied: - https://arxiv.org/abs/1907.01341 https://arxiv.org/abs/1907.01341 - https://research.cs.cornell.edu/megadepth/ https://research.cs.cornell.edu/megadepth/
- fxtentacle 7y agoYes, you can always create that data by hand, but it's too expensive and, hence, impossible to scale. As for using other AIs, they tend to not work too well on more complex images. But in any case, getting the data that you need to be able to use the paper here is very challenging. Edit: I should probably say that I have hands-on experience with MegaDepth and Midas and that it was underwhelming. Both of them assume a Gradient from close to far from the bottom to the top and both of them assume that optical variation will be in the foreground. A photo of a dining table from the side is already enough to confuse both of them.
- nestorD 7y agoTrue but their collab demo uses a separate network to infer depth information from a simple RGB picture (its a classical task nowadays) so it is not a problem in practice.
- mopierotti 7y agoIt would be interesting to see this applied as a photo viewing app for VR, or applied to videos.
- ConradKilroy 7y agoThis looks like Lytro https://en.wikipedia.org/wiki/Lytro https://en.wikipedia.org/wiki/Lytro
- eganist 7y agoIt does, but applied in reverse. Check the Legacy Photos video for examples.
- ladberg 7y agoKind of an interesting tangent, but the founder of Lytro (Ren Ng) is the advisor on a really cool paper that came out recently and reminds me a lot of this: http://www.matthewtancik.com/nerf http://www.matthewtancik.com/nerf Basically, it's similar to this but also re-lights reflective parts at the cost of needing more than one source image.
- jsilence 7y agoWould be awesome if this could be integrated into gallery software as a 3D Ken Burns effect. The artificial camera would not have to move that much so that the inevitable artifacts would be much less visible.
- amelius 7y agoThis would be great as a CSS scroll effect /s