9 ms·
PixelNeRF Neural Radiance Fields from One or Few Images
- nl 6y agoThis looks to be nice work. The failure modes of these models are always interesting. I noted the dresser/cabinet thing(?) with the Union Jack on the front, and the model isn't quite sure how to handle the colors on the back. You see the same with the monitor and display cabinet below it. Some color bleeds though in both cases. I think the funnest is the failure of DVT and SRN on the CRT monitor, where both turns it into an arm chairs
- faeyanpiraat 6y agoIf you look closely at that set, DVR turns everything into sittingware :D
- subroutine 6y agoThat's a feature. Want a chair to match your lamp? Just input your lamp.
- dash2 6y agoI guess this is coming soon to a smartphone near me: take a few photos, generate a 3D scene.
- tmabraham 6y agoFacebook already has this: https://ai.facebook.com/blog/powered-by-ai-turning-any-2d-photo-into-3d-using-convolutional-neural-nets/ https://ai.facebook.com/blog/powered-by-ai-turning-any-2d-ph...
- calaphos 6y agoNot relay the same. This predicts an additional depth channel using CNNs which you can then use to create some sense of 3D by translating the image elements differently based on depth when rotating the camera a limited amount. However you won't be able to e.g. inspect the backside of the object or model things like cavities using this approach. It's more like a heightmap or relief. With nerfs you get an actual volume representation with a mapping from every point in 3d space to whether it's inside a volume or not. It's an actual 3d model instead of a 2d image with a depth channel.
- aimor 6y agoI tried out the app "3D Creator" years ago and it worked really well. That's traditional structure from motion though.
- 01100011 6y agoShower thought: Is anyone looking at using neural networks to fill in the pillarbox area of 4:3 content? It seems like you'd be able to turn old 4:3 video into 16:9 by learning what was outside the active region from adjacent frames and filling it in.
- goodside 6y agoYes, this is an established task known as image outpainting. What you’re describing is actually video outpainting, because you would want to use surrounding frames for context.
- hantusk 6y ago.. and your phone already does this: https://www.google.com/amp/s/www.macworld.com/article/3444637/how-to-use-the-over-capture-feature-in-the-new-iphone-camera-to-crop-and-adjust-images-later.amp.html https://www.google.com/amp/s/www.macworld.com/article/344463... Also, im sure I saw a very nice video on this, with a car driving through the landscape, using previous frames as well. Can't find the link though :-\
- playpause 6y agoThat’s just an adjustable crop. Outpainting means generating extra imagery that wasn’t part of the original capture.
- ackbar03 6y agoThat's actually an interesting thought. I don't know how much actual value that would actually add but would be pretty interesting
- heyitsguay 6y agoIt'd be great if you could use something like this for quick-and-dirty 3D asset generation, but is there any way to convert neural radiance fields to meshes? It'd be cool if you could take a photo or two of an object and get a 3d mesh out quickly.
- faeyanpiraat 6y agoSoftware already exists for this purpose. You have to take photos of an object from all angles (more the better), and a mesh is generated. I haven’t had much success with it though, that technology certainly has room for improvement.
- heyitsguay 6y agoRight, I'm aware of the structure from motion stuff that's already out there and its large view requirement. I've had a similar experience. Hence my question about using something like PixelNeRF instead.
- Fronzie 6y agoApparantly the good old marching cubes can be used for that: https://www.matthewtancik.com/nerf https://www.matthewtancik.com/nerf shows a video about halfway on the page.
- heyitsguay 6y agoNice find! Time to put together a pipeline... (1) take a few photos of a thing you want to model, (2) use a segmentation algorithm to mask out the background, (3) run a NeRF algorithm, (4) use marching cubes to generate a mesh.
- harperlee 6y ago(5) use the volume as a prior for a probabilistically programmed model of known objects (a la picture [1]); (6) render a model scene in blender; (7) use the difference between the modeled scene and reality to guide attention to the robot moving the camera; (8) make the robot arm poke the thing, move it, rotate it, crash it; (...); (?) Terminator. [1] https://github.com/mrkulk/MIT-Picture https://github.com/mrkulk/MIT-Picture
- faeyanpiraat 6y agoI believe this is a key component to true self driving cars. Being able to imagine correctly what is behind stuff is a required skill for safe driving.
- majidmir 6y agoThere's no understanding happening here. As I understand it only provides a way to render images from novel views without any 3d/2d segmentation.
- faeyanpiraat 6y agoNo understanding yet, but it's a step in that general direction.
- visarga 6y agoThis neural net could create a realistic 3D environment where AI agents could train and explore. The lack of embodiment is the main thing missing from reinforcement learning agents to close the gap to humans. The environment itself can be controllable by embeddings, spanning any conceivable situation. Of course, ideally, if this architecture could be made much more efficient.
- Uberphallus 6y agoBasing life-or-death decisions on the computer vision interpretation of a synthetically produced image of what may or may not be behind an object sounds like a recipe for disaster, as if the current state of the field wasn't disaster enough. Just thinking of the amount of adversarial "speedrunner" type of scenarios gives me a headache.
- drmacak 6y agoOn the other hand thats what people(brain) do all the time.
- purplecats 6y ago
- numlock86 6y agoFinally ML assisted photogrammetry, requiring a lot less input data, doesn't seem so far fetched anymore.
- sabujp 6y agothis will be featured in 2 min papers if it hasn't been already
- ilaksh 6y agoImagine if you could somehow scale this up to analyze frames in a video and create a 3d reconstruction of each frame. That could be an amazing starting point for learning about the world. Say you then combined it with captioning and fed it a significant portion of YouTube. Imagine what funny cat videos you could synthesize.
- gruturo 6y agoHow long until you can feed it a few minutes of video of a random person (e.g. a politician) and have a convincing model you can animate and put in a different context? How do we develop technology to detect such forgeries?
- ilaksh 6y agoI think there may already be a few rudimentary systems like that. Maybe not totally automated or convincing quite yet though. So I would assume sometime between now and five years from now depending on what your standards are. I also believe there is already an enthusiastic community and fledgling industry dedicated to deep fake detection.
- Uberphallus 6y agoI want to feed that model with Escher drawings and see what happens.
- keyle 6y agoNice. What's the use case for this? Render a few frames and let it fill up to save on rendering time? Isn't it just simpler to render all the frames?
- vld 6y agoMy first thought was for better 3D Google maps, ie rendering of the buildings.
- gruturo 6y agoIf you have a source model to render, just do that, indeed. This is for doing the opposite - i.e. you input actual pictures, and it tries to understand the shape of things based on how they look different between pictures. Now you can create more pictures, from new angles. (Oversimplified, but that's the idea)
- stfwn 6y agoIn June I worked on a comparison of the original NeRF [1] to a state of the art proprietary photogrammetry method. The photogrammetry method could process ~80GB worth of 24MP photos into a micrometer-level accurate 3D model in about 8 hours, while the fastest NeRF implementations available took the same time to train a model on just 46 pictures at 0.2MP. A funny extrapolation from a handful of datapoints was that it would have taken 1406 hours or about two months to train a NeRF at a resolution of 24MP, assuming it would converge at all. PixelNeRF improves an aspect that was already great (the number of photos required) but does not seem to tackle this complexity problem. Another problem is this: the representation of the learned scene is entirely abstract, contained within the weights of the neural networks that make up the NeRF. The space itself cannot be meaningfully inspected -- it must be probed and examined by its input/output pairs. The NeRF takes as its input a 3D location plus a viewing direction, and the output is a color radiance in that direction and a density (which depends only on the 3D location). So to generate a 2D image you emit camera rays into the NeRF from a specific hypothetical camera position, direction, focal length and sensor resolution, get the NeRF's output for many different points along all the camera rays and compute an image based on it (volume rendering). This is fine as long as the NeRF is available and there are no time constraints, but does not seem workable for real-time graphics rendering like in gaming/VR. So the NeRF should probably be rendered into a traditional 3D model ahead of time. Afaik this is an open problem that I've only seen solved by using a combination of marching cubes to extract the scene geometry and then rendering colors from normal vectors. In this process, continuity, spatial density and directional color radiance, three of the most important contributions of the NeRF design, are entirely lost. I would be very interested to see papers that tackle higher resolution spaces at feasible training times and faster novel view rendering times. It would be amazing to have NeRF-based graphics engines that can make up spaces out of layers of NeRFs, all probed in real-time. [1]: https://www.matthewtancik.com/nerf https://www.matthewtancik.com/nerf
- fxtentacle 6y agoIn a similar vein to what you described, I recently evaluated multiple deep learning based image synthesis techniques for their viability for filling in parts of real-time games. Turns out, it's just not useful. GPUs are just fine with rendering mind-boggling amounts of geometry, as long as overdraw is low. In other words, using deep learning is using up exactly the resource that's rare already. Also kind of related, the new UE5 game engine is introducing a novel in-GPU compression method which will allow them to handle more geometry. Not only for games, but also for photogrammetry, memory tends to be one of the scarcest resources. In summary, unless the AI uses relatively few memory and unless it's relatively few layers, it won't stand a chance against traditional ways of handling geometry. That said, the promise that I see with NeRF and related methods is their ability to make up plausible things. For everyday objects, these techniques can learn to predict how an apple would look like if you would rotate it reasonably well. That is valuable for robotics, where you need to make sure that you still recognize your environment even after you drive around a corner.