12 ms·
NeRF in the Wild: reconstructing 3D scenes from internet photography
- nawgz 6y agoThose are some very cool 3D visualizations generated, but it's a bit difficult to understand what the form of the dataset they generated it from is. They say "in-the-wild" photography, but of course don't really give you a great sense. The light->dark transitions having consistent geometry is clean though.
- toomuchtodo 6y ago> They say "in-the-wild" photography, but of course don't really give you a great sense. Flickr user photos. Citation shows up in the lower right hand corner during the video. This appears to be a substantial improvement on current open photogrammetry/structure from motion work [1]. I hope Google supports this making its way into cultural preservation efforts [2]. [1] https://github.com/mapillary/OpenSfM https://github.com/mapillary/OpenSfM (developed by Mapillary, now part of Facebook) [2] https://www.nytimes.com/2015/12/28/arts/design/using-laser-scanners-to-preserve-antiquities-in-isiss-cross-hairs.html https://www.nytimes.com/2015/12/28/arts/design/using-laser-s... (Using Lasers to Preserve Antiquities Threatened by ISIS)
- nawgz 6y agoYes, I mostly meant that I don't get a great sense of "how many photos there are" in these datasets. I saw in the paper their citation [13] pointed to https://arxiv.org/pdf/2003.01587.pdf https://arxiv.org/pdf/2003.01587.pdf, which in section 3 says the following: We thus build on 25 collections of popular landmarks originally selected in [48,101], each with hundreds to thousands of images. So hundreds to thousands of photos are used, which is a decent quantity, but definitely makes the quality of the result very impressive.
- duckworthd 6y agoWe use images from the Image Matching Challenge 2020 dataset. If you look at the Appendix, we list how many images we use and the process by which they were chosen. Download and have a look! https://vision.uvic.ca/image-matching-challenge/data/ https://vision.uvic.ca/image-matching-challenge/data/
- nawgz 6y agoThanks, that's a clean reference.
- mey 6y agoI recall many years back a website, I think a Microsoft project, that linked together photos in a 3D space of tourist destinations. It created something of a point cloud, but nothing this advanced. You could click through the points/photos to jump into each photos perspective of the space. Anyone remember what that project was called, or if it is even still around? Edit: Found it. http://phototour.cs.washington.edu/ http://phototour.cs.washington.edu/ Later the discontinued. https://en.wikipedia.org/wiki/Photosynth https://en.wikipedia.org/wiki/Photosynth
- deleted 6y ago[deleted]
- ur-whale 6y agoWas it PhotoSynth ? https://www.ted.com/talks/blaise_aguera_y_arcas_how_photosynth_can_connect_the_world_s_images/transcript?language=en https://www.ted.com/talks/blaise_aguera_y_arcas_how_photosyn...
- mey 6y agoWatching this demo makes it very stark how much our single page webapps are regressions in fluid performance.
- fastball 6y agoIs this not a Silverlight app or something? I highly doubt that demo was JS / could run in multiple browsers without some proprietary runtime.
- shakna 6y agoIt was, in point of fact, a Java applet. [0] Older, less secure, but more powerful. [0] https://web.archive.org/web/20191231213153/http://phototour.cs.washington.edu/applet/index.html https://web.archive.org/web/20191231213153/http://phototour....
- 6y ago
- ur-whale 6y agoThis is really cool and IMHO an area where ML truly shines: being able to disentangle the base geometric signal from lighting / crowds / occlusion via learning is truly amazing.
- duckworthd 6y agoOriginal author here. AMA!
- hi41 6y agoVery cool! Congratulations!
- duckworthd 6y agoThank you :)
- pferdone 6y agohow many pictures or angles are needed to produce good results? I get that landmarks have an abundance of source material, but whats a reasonable amount of data to reconstruct scenes?
- duckworthd 6y agoOn the order of hundreds to low-digit thousands worked well for us. These photos contain a lot of occluders like tourists, and we needed to have enough views of the subject in question to build a good 3D scene representation.
- _visgean 6y agoIs the model able to capture the underlining geometry? E.g. If I have a pillar part of which was not visible at any training point is it able to reconstruct that part?
- duckworthd 6y agoThe model is trained to reconstruct what is observed, but not what is obscured. If you look closely at our videos, you'll notice some parts of the scene are blurry -- those parts weren't seen often enough to learn well. If you look at parts of the scene not observed at all, I'm not sure what you'd find.
- Mathnerd314 6y agoI'm still looking for a program that takes a video and turns it into an animated 3D scene. All the stuff I've seen is on static scenery, besides some neural nets that can tweak camera angles.
- duckworthd 6y agoThere is currently no way i'm aware of to accurately reconstruct a moving 3D scene. Sorry! Ask us again in a few years :)
- johanneskopf 6y agoCheck this. Code coming (very) soon :) https://roxanneluo.github.io/Consistent-Video-Depth-Estimation/ https://roxanneluo.github.io/Consistent-Video-Depth-Estimati...
- ThisIsMyPasswrd 6y agoDo you happen to know how intellectual property works when someone wants to use the algorithm/code?
- johanneskopf 6y agoI think we're going to use the MIT license. So, you'll be able to use it in almost any way you like...
- lostmsu 6y agoMIT actually does not give an explicit patent grant. So if "any way you like" is your goal, you should choose something different like Apache License 2.0
- flyingcircus3 6y agoAfter going to one of the early Maker Faires, and seeing so many interesting exhibits and projects, I had this same idea, of course with absolutely no clue about how to implement it. If enough people take pictures of the exhibits from a variety of angles, and make them available online, a virtual Maker Faire could be created. Thanks for sharing this!
- schemescape 6y agoIs the geometry from each of the examples available in some format? It would be fun to look more closely. Apologies if I missed a link somewhere!
- duckworthd 6y agoThe magic of this method is that we don't construct a "geometry" the same way one might think. There are no triangles or textures here. Instead, we train a machine learning model to predict the derivative of the color and opacity at every point in 3D space. We then integrate along rays emitted from the camera to render an image. It's similar to what's used in CT scans!
- heyitsguay 6y agoThat's very cool, but also makes it sound more challenging to integrate into the existing 3D modeling ecosystem vs, say, photogrammetry approaches. Is it possible to generate approximate textured meshes from the color and opacity information? [Edit] After a little Googling I do see this has been done, using marching cubes (https://www.matthewtancik.com/nerf https://www.matthewtancik.com/nerf).
- jayd16 6y agoThis means you don't have an occlusion mesh or any other depth information, correct?
- teraflop 6y agoThere is depth information, just not in the form of a mesh. The model learns to compute a function that takes an XYZ position within a volume as input, and returns color and opacity. You can then render images by tracing rays through this volume. You can pretty easily compute the distance to the first sufficiently-opaque region, or the "average" depth (weighted by each sample's contribution to the final pixel color), at the same time. Another recent Google project figured out a way to approximate these radiance fields with layered, partially transparent images for efficient rendering: https://augmentedperception.github.io/deepviewvideo/ https://augmentedperception.github.io/deepviewvideo/
- mrfusion 6y agoHow does this compare to photogrammetry?
- duckworthd 6y agoAccording to Wikipedia, "Photogrammetry is the science and technology of obtaining reliable information about physical objects and the environment through the process of recording, measuring and interpreting photographic images and patterns of electromagnetic radiant imagery and other phenomena." I'd say that this research is in the field of photogrammetry.
- mrfusion 6y agoI guess I meant how it compares to the commercial photogrammetry software out there.
- duckworthd 6y agoI don't know! I'm not familiar enough with the field to say.
- ur-whale 6y agoAs mentioned in another comment, traditional photogrammetry software typically generates a 3D polygonal mesh. It looks like this generates a light field, which is not something that traditional 3D software handles directly.
- woko 6y agoThe videos are very impressive. I wish you let us move the 3D scene with the mouse in the browser to get a better idea of the result.
- randyrand 6y agoWow that is fantastic work! And so quick since NeRF debuted. This is exactly the kind of work I have been waiting for to reconstruct some old photos I have.
- ur-whale 6y ago> reconstruct some old photos I have. Note that it generates a light field, which is note exactly like a polygonal mesh ... YMMV
- billconan 6y agoso why does it use multi-layer perceptron? is it the same as ANN? why not calling it ANN? Does it have activation? There is another thing I don't understand. Traditional volume data is a map of position to density (or color). There doesn't seem to be the need for a direction as an input. Why does the network need a direction? Why can't we get a density and a color given a position?
- duckworthd 6y ago> so why does it use multi-layer perceptron? is it the same as ANN? why not calling it ANN? Does it have activation? According to Wikipedia, "A multilayer perceptron (MLP) is a class of feedforward artificial neural network (ANN)." We're being more specific about what we use :) > There is another thing I don't understand. Traditional volume data is a map of position to density (or color). There doesn't seem to be the need for a direction as an input. Why does the network need a direction? Why can't we get a density and a color given a position? Volume data of this form is unable to express the idea of view-dependent reflections. I admit, we don't make much use of that here, but it does help! See NeRF for where it makes a big, big difference: https://www.matthewtancik.com/nerf https://www.matthewtancik.com/nerf
- brookman64k 6y agoGreat work! Having tried the code from the original NeRF paper I found the inference time (generation of new views) to be rather slow because the network had to be queried multiple times per ray (pixel). The paper said that there is still potential to speed this up. Did you improve inference speed and do you think that it will be possible to get it to real-time (>30 fps) in the foreseeable future?
- duckworthd 6y agoWe did not aim to speed this part of NeRF up. Check out Neural Sparse Voxel Fields (https://arxiv.org/abs/2007.11571 https://arxiv.org/abs/2007.11571) for some effort in that direction. It's 10x faster, but there's still another 10x to go till you get video frame rates :)
- hardmath123 6y agoA while back I stitched together a "hyperlapse" of Stanford's Hoover Tower using lots of Flickr-scraped images. Everything was aligned using "classical" CV tricks and I was really happy with the results. I wonder how NeRF-w would fare on this data? https://github.com/kach/hootow-hyperlapse https://github.com/kach/hootow-hyperlapse
- PeterCorless 6y agoThis sort of work will both allow for digital forensics (imagine reconstructing a scene from multiple socially shared images or video), as well as to create even better "deep fakes" (putting people in scenes they never actually went to; or at different times of day/night, or with different weather effects).
- ekianjo 6y agoIs there a reason why the skies do not appear to be picked up by their "transient" filter of the scene? You end up with the skies constantly changing when moving in 3D point of view, which looks strange.
- duckworthd 6y agoA good question! And a problem yet to be solved!
- Proven 6y agoI clicked on the link because I misread that as “pornography”. Oh well
- nla 6y agoAmazing work! Reminds me of something I saw at SIGGRAPH back in '95 called 'Tour into Picture' I think the work came out of Japan. When will you be sharing some code?!
- nla 6y agoAlso, can you share any details of the compute requirement for this?