3 ms·

Nowadays, it is possible to create a 3D model of a space by taking hundreds of photographs of a space and using computers to analyze those photos through photog

by nkurz 9y ago

Nowadays, it is possible to create a 3D model of a space by taking hundreds of photographs of a space and using computers to analyze those photos through photogrammetry and computer vision methods. This is a laborious process, and is especially time-consuming and resource-intensive for large scenes. I was recently trying to understand what the state of the art was here, and was surprised to learn that this is true: most 3D reconstruction fares better with a smaller number of high-resolution still photos than with a larger number of lower resolution "stills" extracted from video. I'm still somewhat confused why this is the case. My limited understanding is that main advantage of working with stills is that they more commonly contain location tagging, while frames from video do not. Compared to the base level of difficulty of building a accurate 3D model from lossy 2D sources, figuring out the trajectory of camera doesn't seem too hard. Once one has this, wouldn't the video be just as easier to work with? Trying to figure this out this discrepancy, I got the impression that many researchers may actually be trying to solve the harder problem of trying to compute the camera trajectory in real time, as might be needed for a self-driving vehicle: http://webdiis.unizar.es/~raulmur/orbslam/ http://webdiis.unizar.es/~raulmur/orbslam/. This indeed does seem harder, but still leaves me wondering why a multipass approach wouldn't be feasible. What am I missing? Why can't one do a first pass to calculate camera trajectory, possibly a second pass to combine temporally adjacent frames for greater resolution, then create a better model from the resulting wealth of data? Alternatively stated, why does the quality of the 3D model seem to depend more on the resolution and quality of the input 2D images rather than on the number of these images?

ndh2 9y ago
I've been wondering about this as well. Why are they using so few images? To me that is like reverse blinking: Instead of keeping your eyes open, and blinking for a short amount of time every now and then, you're walking around with your eyes closed, and open them only for a short time. I believe the main problem is knowing which data to trust and which data to discard. Reflections (water, shiny surfaces), moving objects (leaves in the wind), over-exposure, and lens flares are already pretty hard to deal with. But with low resolution data, you make it even more difficult because there's more data to discard for being too inaccurate.
- namlem 9y ago
  Actually, that's not how our eyes work at all. In fact, the "reverse blinking" you describe is much closer to the truth. The retina has a very small region of high acuity that darts around your visual field, essentially taking snapshots that your brain stitches together. A lot of what we perceive as movement is actually our brains predicting trajectories.
  - ghusbands 9y ago
    No, a lot of what we perceive as movement is our eyes seeing movement. Motion is perceived across the whole of our field of vision, not just by the fovea (the "very small region of high acuity" you mentioned). Try not to spread misinformation.
Keyframe 9y ago
Maybe motion blur has something to do with it? I've been using photogrammetry techniques occasionally, when I needed it, in film/3d work. One thing I didn't try out, and what I wanted and you've reminded me of it now is to try going through a space with RED camera, crank the shutter so to eliminate motion blur and maybe even go with the HDRx option (for textures, later on). I'll see what Photoscan comes up with.
angry_octet 9y ago
What you are talking about is called Monocular SLAM. The process of calculating the trajectory inherently maps the world at the same time by estimating feature positions by inter-frame distances. It really isn't a problem to use video frames, but they have different types of error/noise to high res stills. For example, the video camera is moving, and it has rolling shutter, so every pixel (or pixel row) is from a different orientation. If this isn't accounted for it introduces noise, but doing so takes more/better processing. But video frames have consistent intre-frame delay and good coherence, so it is easier to match features frame-by-frame than from a heap of individual photos. If you have good (hot GPS) positions then that makes both situations easier. Model visual quality is improved by better resolution pictures in a fairly straightforward way... But you can get the same data by physically moving a lower res camera closer to all surfaces. It is just harder to scan quickly and maintain track if you have to do that. Some people even put lenses in front of their depth camera to zoom in tighter, for capturing finer detail.
rawnlq 9y ago
> combine temporally adjacent frames for greater resolution Even just combining multiple still photos to produce a higher resolution photo is itself a hard problem in the field of super-resolution imaging. Google has a consumer product for it (mostly for removing glare): https://www.youtube.com/watch?v=MEyDt0DNjWU&feature=youtu.be&t=62 https://www.youtube.com/watch?v=MEyDt0DNjWU&feature=youtu.be....
rasz 9y ago
We had pretty solid location from motion for almost 10 years now. PTAM, realtime on a laptop in 2008 http://www.robots.ox.ac.uk/~gk/PTAM/ http://www.robots.ox.ac.uk/~gk/PTAM/

by nkurz 9y ago

ndh2 9y ago

I've been wondering about this as well. Why are they using so few images? To me that is like reverse blinking: Instead of keeping your eyes open, and blinking for a short amount of time every now and then, you're walking around with your eyes closed, and open them only for a short time. I believe the main problem is knowing which data to trust and which data to discard. Reflections (water, shiny surfaces), moving objects (leaves in the wind), over-exposure, and lens flares are already pretty hard to deal with. But with low resolution data, you make it even more difficult because there's more data to discard for being too inaccurate.

namlem 9y ago
Actually, that's not how our eyes work at all. In fact, the "reverse blinking" you describe is much closer to the truth. The retina has a very small region of high acuity that darts around your visual field, essentially taking snapshots that your brain stitches together. A lot of what we perceive as movement is actually our brains predicting trajectories.
- ghusbands 9y ago
  No, a lot of what we perceive as movement is our eyes seeing movement. Motion is perceived across the whole of our field of vision, not just by the fovea (the "very small region of high acuity" you mentioned). Try not to spread misinformation.

Keyframe 9y ago

Maybe motion blur has something to do with it? I've been using photogrammetry techniques occasionally, when I needed it, in film/3d work. One thing I didn't try out, and what I wanted and you've reminded me of it now is to try going through a space with RED camera, crank the shutter so to eliminate motion blur and maybe even go with the HDRx option (for textures, later on). I'll see what Photoscan comes up with.

angry_octet 9y ago

What you are talking about is called Monocular SLAM. The process of calculating the trajectory inherently maps the world at the same time by estimating feature positions by inter-frame distances. It really isn't a problem to use video frames, but they have different types of error/noise to high res stills. For example, the video camera is moving, and it has rolling shutter, so every pixel (or pixel row) is from a different orientation. If this isn't accounted for it introduces noise, but doing so takes more/better processing. But video frames have consistent intre-frame delay and good coherence, so it is easier to match features frame-by-frame than from a heap of individual photos. If you have good (hot GPS) positions then that makes both situations easier. Model visual quality is improved by better resolution pictures in a fairly straightforward way... But you can get the same data by physically moving a lower res camera closer to all surfaces. It is just harder to scan quickly and maintain track if you have to do that. Some people even put lenses in front of their depth camera to zoom in tighter, for capturing finer detail.

rawnlq 9y ago

> combine temporally adjacent frames for greater resolution Even just combining multiple still photos to produce a higher resolution photo is itself a hard problem in the field of super-resolution imaging. Google has a consumer product for it (mostly for removing glare): https://www.youtube.com/watch?v=MEyDt0DNjWU&feature=youtu.be&t=62 https://www.youtube.com/watch?v=MEyDt0DNjWU&feature=youtu.be....

rasz 9y ago

We had pretty solid location from motion for almost 10 years now. PTAM, realtime on a laptop in 2008 http://www.robots.ox.ac.uk/~gk/PTAM/ http://www.robots.ox.ac.uk/~gk/PTAM/