8 ms·
Original author here. AMA!
by duckworthd 6y ago
Original author here. AMA!
- hi41 6y agoVery cool! Congratulations!
- duckworthd 6y agoThank you :)
- pferdone 6y agohow many pictures or angles are needed to produce good results? I get that landmarks have an abundance of source material, but whats a reasonable amount of data to reconstruct scenes?
- duckworthd 6y agoOn the order of hundreds to low-digit thousands worked well for us. These photos contain a lot of occluders like tourists, and we needed to have enough views of the subject in question to build a good 3D scene representation.
- _visgean 6y agoIs the model able to capture the underlining geometry? E.g. If I have a pillar part of which was not visible at any training point is it able to reconstruct that part?
- duckworthd 6y agoThe model is trained to reconstruct what is observed, but not what is obscured. If you look closely at our videos, you'll notice some parts of the scene are blurry -- those parts weren't seen often enough to learn well. If you look at parts of the scene not observed at all, I'm not sure what you'd find.
- narrationbox 6y agoHow do you remove tourists? Is the network trained to segment and ignore humans?
- duckworthd 6y agoThe model does not explicitly learn to segment images. The answer is unfortunately more difficult to explain than a HN comment bears. I encourage you to read the paper for more details. https://arxiv.org/abs/2008.02268 https://arxiv.org/abs/2008.02268
- pferdone 6y agoJust gotta say: amazing! My follow up question would be: are you able to compare your results to actual photogrammetry data to see how good your reconstruction performs?
- duckworthd 6y agoI'm actually quite new to the field, and I'm not even sure what to compare against nor how to compare it. What's typically measured and how?
- baq 6y agowould a sufficiently long video in motion, say from a drone, car or even a walking person, work instead?
- duckworthd 6y agoPictures are pictures, even as video frames :)
- sumtechguy 6y agoDid you consider using movies as a source too?
- duckworthd 6y agoConsider? Yes. Try? Nope!
- sumtechguy 6y agoawww! Figured dolly shots and steady cam shots would fit perfect into something like that. Esp 24 frames per second and usually known locations. Course it would probably drag a lot of the net into being biased towards that time spot I guess?
- rasz 6y agoThere are problems associated with using video: motion blur, rolling shutter.
- sumtechguy 6y agoOh I agree. In my head it seems like it should work. I could be wildly wrong though. I am every day :)
- rasz 6y agoIt definitely can work, and even has some additional benefits (1), but requires special considerations. You can deblur using global motion vectors (2), or additional hardware like accelerometer reading embedded in the video feed (3). 1) cant find the paper now, but by exploiting predictable rolling shutter you get additional temporal resolution 2) http://users.ece.northwestern.edu/~sda690/MfB/Motion_CVPR08.pdf http://users.ece.northwestern.edu/~sda690/MfB/Motion_CVPR08.... 3) http://neelj.com/projects/imudeblurring/imu_deblurring.pdf http://neelj.com/projects/imudeblurring/imu_deblurring.pdf
- panabee 6y agocan you elaborate on the key variables for the data? for instance, is it safe to assume 360 photos from the same angle would yield a worse model than 1 photo from 360 different angles? what does the ideal minimal data set look like (eg, 5 photos from each 15-degree offset)? thanks for being so active on this thread.
- duckworthd 6y agoNeRF's (and all of photogrammetry's) bread and butter is 3D consistency -- that is, seeing the same object from multiple angles. A 360 degree photo from a fixed position just won't do. As to how to select the best camera angles...I'm not sure. I believe there is research in this area for classical photogrammetry techniques, but I'm not familiar enough to point you to a body of work.
- akavel 6y agoLooks awsome! I'm not a ML guy and haven't read the paper, just watched the video - one thing isn't clear to me from it: is this fully automatic/unattended, you just throw images into it and out come magic rainbows of 3d structures? or do you need to somehow help it, e.g. to disentangle the structure from the "transient" elements? In other words, I don't really understand what does the "Appearance Embedding" even mean... Or is the "input" that you mention in the video fed into a model that is already trained on a set of photos of a particular scene? I.e. the "input" + "appearance embedding" basically encodes just a choice of a framing & "atmosphere/lighting"?
- duckworthd 6y agoIt's a little hard to describe from scratch, but let me do my best. The method is unattended, in the sense that it's photos + camera parameters in and scene representation out. The photos should all be of the same scene (e.g. the Trevi Fountain). Once you have a scene representation, you can ask what the scene would look like from new camera angles with your choice of lighting. Choosing camera angles is straightforward. You tell me where and what direction the camera is facing. The question then becomes, how do you specify your choice of lighting? The answer is, you can't do so directly. Instead, you provide a picture with the lighting you want, and with a little magic, we can find a way to imitate that lighting. The way we do is by finding a corresponding "appearance embedding" via numerical optimization.
- Ut_Pwnsim 6y agoWhat is the precision required (or used in your datasets) for camera position and angles? Is the geotagging in the images from common cellphones and smart cameras enough? Were they back-calculated using some other method from non- or poorly-georeferenced images?
- duckworthd 6y agoIt's hard for me to say how precise camera position and direction needs to be. We use COLMAP to estimate both via multi-view stereo.
- mey 6y agoWould it possible to integrate video into the model?
- duckworthd 6y agoI'm not sure I understand your question. If the question is, "can you reconstruct a (static) scene from the frames in a video?", the answer is yes! If the question is, "can you reconstruct a scene with people and other moving objects, and model them moving around too?", the answer is not yet.
- mey 6y agoThe former. Once the scene is synthesized, I figure that is where any dynamic output would occur. Although that raises an interesting thought of using the NeRF modeling to paint out certain things in potentially live video.
- airstrike 6y agoI had this vision that one day we'll be able to reconstruct memories from our past by taking old photos and having a ML model collate everything together to form a 3D rendering of that point in time. It seems like you have gotten most of the way there. The next step would be to have the user grab a VR headset and immerse themselves in their favorite childhood moment. One could even add avatars for loved ones using again ML-generated audio based on recordings of their voices. Your project made me think that it wouldn't be that interesting for me to view your memories, so perhaps the best initial step for a proof-of-concept that would allow the technology to mature would be to recreate historical moments so everyone people could relive them – and they could do so entirely virtually, from the comfort of their own couch. Side note: it feels like this technology can disrupt traditional museums with the added bonus of being pandemic-proof. Anyway, I don't really have a question... Just wanted to compliment you on this amazing work and throw this idea out there in case others want to think about it, as I'm in an entirely different field and don't have the skills and resources to make it real, and I do strongly feel this will inevitably come to life.
- duckworthd 6y agoThat's a really cool idea! This technology does a fantastic job at reconstructing static scenes. The moving objects -- people, cars, even flora -- are out of scope here. Why? It's really hard to build a 3D model of something you only see from one direction.
- airstrike 6y agoBut if we know what a car ought to look like in 3D, can't we take the one photo we have from one direction and just fill in the blanks with that a priori 3D knowledge?
- withjive 6y agoSimilar to how GPT-3 can be applied not only to create Text, but also fill in missing pieces of Images (ie. complete the missing half of a face). Would the logical next step, use GPT-3 to create a 3D world? :)
- philipkglass 6y agoCan this technique reconstruct good geometry for the visible parts if only part of the structure is ever imaged? That's a precursor to: could this technique be used to enhance Street View? There are times when I would really like to be able to "walk around" outdoor scenes in finer steps. Current Street View smears between photos taken some distance apart. (I don't know if that is a limitation of the public interface or if the original data capture is really that coarse.) It would be nice to have a real 3D space to explore, but I certainly don't expect the un-imaged parts to be defined correctly. Finally, does this also work for reconstructing interior spaces seen from the inside? Like the geometry of a cave, from pictures of the cave interior?
- duckworthd 6y agoAt this point, this method is only good at reconstructing parts of the scene that are well-photographed. You'll notice that our video for Sacre Coeur has some blurry bits, particularly the staircase in front of the Basilica. That's because we learn to reconstruct what was seen, but aren't yet able to imagine what wasn't! Does this work in reconstructing indoor spaces? Give it a shot and find out!
- idontevengohere 6y agoLove how you explained this! And I thought I was the only one using exclamation marks for everything
- philipkglass 6y agoDoes this work in reconstructing indoor spaces? Give it a shot and find out! Have you released the code? (Or did you mean I could try re-implementing the work you published from the paper? That's a reasonable response too.) I didn't see a link to source in the github.io page or in the arXiv paper. The only source code link I saw was to https://github.com/bmild/nerf https://github.com/bmild/nerf which I thought was earlier work than this paper.
- duckworthd 6y ago
- sramam 6y agothis looks really cool. I'm not am ML chap, but always wondered: Can these kinds of algorithms also give you dimensional data? For example, can I 3D-print one of these models with any accuracy?
- duckworthd 6y agoFor that, you'll need to convert the representation we have (volumetric radiance field) to on your 3D printer can understand (a mesh?). The NeRF authors use the marching cubes algorithm to do just that. Check out their website: https://www.matthewtancik.com/nerf https://www.matthewtancik.com/nerf.
- panabee 6y agoThis looks amazing! Congratulations. What was the most challenging aspect of this? I'm curious to see how you performed edge detection on the transient objects and were able to isolate them so cleanly. For some reason, the paper isn't loading, so feel free to say it's explained in detail there. Do you plan on releasing code? Congrats again. This is very cool research.
- duckworthd 6y ago> This looks amazing! Congratulations. Thank you! > What was the most challenging aspect of this? Wow, that's hard to say! Our work truly stands on the shoulders of giants (Mildenhall et al, 2020). I can list off a few challenges: figuring out if an idea "kinda works" or "definitely works" or has a bug, figuring out how to measure progress, coordinating a group of 6 researchers living 9 hours apart, and assembling everything together for a simultaneous paper-website-video release! > I'm curious to see how you performed edge detection on the transient objects and were able to isolate them so cleanly. We don't! All of this comes by the magic of machine learning :). We train our model to attentuate the importance of "difficult" pixels that aren't 3D-consistent in the training set. We also partition the scene into "static" and "transient" volumetric radiance fields without explicit supervision. We do so by regularizing the latter to be empty unless necessary, and providing it with access to a learned, image-specific latent embedding. We discard the transient radiance field when rendering these videos, thus removing tourists and other moving objects. > For some reason, the paper isn't loading, so feel free to say it's explained in detail there. Well that won't do. Download it here: https://arxiv.org/abs/2008.02268 https://arxiv.org/abs/2008.02268. It's 40 MB, so your download bar may indicate it's almost done, but it actually has a good bit left to go. > Do you plan on releasing code? I hope so! As with most code, what ran on our machines may not run on yours. Migrating the code to open source will be a big effort. I hope what we describe in the paper is sufficient to build something like you see here.
- alacombe 6y ago> Migrating the code to open source will be a big effort. No, it is not. It's 5 shell command at the most. $ git init $ git add . $ git commit -m 'Initial import' $ git remote add origin git://... $ git push origin master
- TaylorAlexander 6y agoGreat work! I’m an ex googler currently working on a farming robot we hope to make open source. I’m particularly interested in neural reconstruction of plants in a field. I want to capture the 3D structure of the plants as well as semantics like plant species. I’ve found that normal photogrammetry produces poor reconstructions due to movement of the plants in the wind. We’re potentially about to start a non profit and formally kick off our whole robot as open source. I’m interested in finding research partners who would like to help produce a research paper on 3D reconstruction of plants. I can produce a high quality geo located dataset with 2cm accurate GPS tags, but I have no experience with neural rendering. This is work I want to do over the course of the next year. Do you know anyone interested in helping with thank kind of work? Thanks!
- duckworthd 6y agoI'm a bit new to the field myself, so I'm afraid I can't provide any contacts. Ask me again in a couple of years. The method presented here wouldn't do well with your problem either. 3D reconstruction of moving objects is an unsolved problem!
- TaylorAlexander 6y agoIndeed. I am seeing some generative approaches that know what the object should look like in 3D and use that knowledge to imagine a model that matches a photo. I think such a technique would be useful for good approximation of plant models. Such a project would require some new datasets I would think, but seems like a good approach. Just came across this which is neat: https://github.com/AljazBozic/DeepDeform https://github.com/AljazBozic/DeepDeform Also this might be useful: https://github.com/paschalidoud/hierarchical_primitives https://github.com/paschalidoud/hierarchical_primitives For now I’m just beginning to collect data but I hope to contribute more to the field in time!
- riotman 6y agoWhy did you use neural networks? There are faster techniques in analytical geometry that can extract surface contours from color gradients from images, and they do this faster and directly.
- duckworthd 6y ago1) My bread and butter for the last 10 years has been machine learning. When all you have is a hammer... 2) We don't extract surface contours, we learn a volumetric radiance field! To oversimplify, we learn a (smooth) function that, given a position in space, produces the differential opacity and color at that space. To render an image from a camera viewpoint, we approximately integrate along rays emitted from each pixel of the camera. Check out NeRF and our paper to learn more about this representation!
- rmbrualla 6y agoNeural networks are better compared to classical methods. One of the best non-classical methods is this one (https://grail.cs.washington.edu/projects/sq_rome_g1/ https://grail.cs.washington.edu/projects/sq_rome_g1/), and our method is significantly improves upon it. We do not compare directly with it, but Neural Rerendering in the Wild does, and we improve upon it.
- randyrand 6y agothese nerf models are like 5MB large are have a ton of directional lighting support. speculars, caustics, refraction, mirrors, you name it! also they're way higher quality than traditional techniques
- billconan 6y agoso why does it use multi-layer perceptron? is it the same as ANN? why not calling it ANN? Does it have activation? There is another thing I don't understand. Traditional volume data is a map of position to density (or color). There doesn't seem to be the need for a direction as an input. Why does the network need a direction? Why can't we get a density (opacity) and a color given a position? and what are z(t) r(t) in equation 5,6?
- duckworthd 6y agoAnswered most your other questions below in another comment. > and what are z(t) r(t) in equation 5,6? r(t) is a position in 3D space along a camera ray of the form, r(t) = origin + t * direction. z(t) is the output of our first MLP. Think of it as a 256-dimensional vector of uninterpretable numbers that represent the input position r(t) in a useful way.
- ThisIsMyPasswrd 6y agoDoes it take reflections into consideration?
- duckworthd 6y agoThe work we build off of, NeRF, does. While there's nothing preventing NeRF-W from also representing reflections, we find it captures a more matte picture of the object.
- nialv7 6y agoNot too familiar with this area. IIUC, training a model would require photos with corresponding information about where and at what angle the photo is taken. Is this information just available from the dataset?
- duckworthd 6y agoIt is provided by the dataset we use, but given a new dataset, you can use off-the-shelf tools to obtain it yourself! Check out COLMAP, it's super duper cool: https://colmap.github.io/ https://colmap.github.io/.
- ctdonath 6y agoRandom thought: Someone made a “camera” which tracks location & direction, and “takes” a picture by selecting the closest picture found taken from that spot & angle. Use this new tech for a next generation of that “camera”, generating the Platonic frame which should occur from that location.
- oillio 6y agoThen instead of a camera, display it in VR goggles. Allows you to walk around and see a landmark without all those pesky people ruining your view. You can use the PhotoSynth tech described above.
- dougabug 6y agoDo you need posed images?
- duckworthd 6y agoYes, the images need to be posed. We use COLMAP to obtain camera pose.
- nullsmack 6y agoIs there a code release somewhere?
- alacombe 6y agoAre you planning to share the code ? GPT-3 has some pretty interesting demo which unfortunately are unfortunately rather disappointing once outside a carefully crafted environment. Said otherwise, a paper is nothing it it cannot be reproduced.
- AtomicOrbital 6y agocool project ... now spin up a public server so we can feed up our own set of images and get back the 3D synth object scene